Data Structures

CMPS 303
Dr. Osama Halabi

Algorithm Analysis

Input Algorithm Output

Copyright © Some of the slides are
© 2014 Goodrich, Tamassia, Goldwasser adopted from colleagues’ slides.
© 2011 The McGraw-Hill Companies, Inc. Copyright © 2014, 2008 Pearson Education,
All rights reserved Inc. Publishing as Pearson Addison-Wesley
 How can we measure the efficiency
of an algorithm?

Comparing Two Algorithms

Empirical Analysis Theoretical Analysis

Data Structures
Empirical Analysis
 Empirical Analysis

1. Implement both algorithms as

computer programs.
2. Run them on a suitable range of inputs.
3. Measure the recourses (time) used by
each one of them.
4. The algorithm that uses less resources
is the best.
Data Structures
 Empirical Analysis to Compare Two Algorithm
✤ consider two algorithms for constructing long strings in Java
✤ calling signature such as repeat('*', 40), that produces a string composed of 40 asterisks:
"****************************************".
✤ Two algorithm

Data Structures 5
 Empirical Analysis to Compare Two Algorithm
✤ used System.currentTimeMillis() to measure the efficiency
of both repeat1 and repeat2

Results of timing experiment on the methods

Data Structures 6
 Why Complexity?
✤ The most striking outcome of these experiments is how much faster the
repeat2 algorithm is relative to repeat1
✤ repeat1 is already taking more than 3 days to compose a string of 12.8
million characters
✤ repeat2 is able to do the same in a fraction of a second.

Data Structures 7
 Algorithm Efficiency
✤ The efficiency of an algorithm is usually expressed in terms
of its use of CPU time
✤ The analysis of algorithms involves categorizing an algorithm
in terms of efficiency
✤ An everyday example: washing dishes
✤ Suppose washing a dish by hand takes 30 seconds and drying
a dish takes an additional 30 seconds
✤ Therefore, n dishes require n minutes to wash and dry

Data Structures 8
 Algorithm Efficiency

✤ Written more formally

✤ Suppose that we were careless while washing the dishes and splashed too
much water
✤ Suppose that each time we washed a dish, we had to dry not only that dish
but also all of the dishes we had washed before that one
✤ It would still take 30s to wash each dish, but now it would take 30s to dry the
last dish (once), 2*30 to dry the second-to-last dish (twice)…
Data Structures 9
 Algorithm Efficiency
✤ The computation could be expressed as follows:

✤ The formula for the arithmetic series

✤ The function becomes

Time (n dishes) = 30n + 30n(n+1)/2
= 15n2 + 45n seconds
✤ If there were 30 dishes to wash, the first approach would take 30 minutes, the
second (careless) would take 247.5 minutes.
✤ The more dishes we wash, the worse that discrepancy becomes
✤ 300 dishes: (1) 300 minutes or 5 hours, (2) 908,315m (15,000 hrs)
Data Structures 10
 Problem Size
✤ For every algorithm we want to analyze, we need to
define the size of the problem
✤ The dishwashing problem has a size n – number of
dishes to be washed/dried
✤ For a search algorithm, the size of the problem is the
size of the search pool
✤ For a sorting algorithm, the size of the program is the
number of elements to be sorted
Data Structures 11
 Growth Functions
✤ We must also decide what we are trying to efficiently
optimize
✦ time complexity – CPU time
✦ space complexity – memory space
✤ CPU time is generally the focus
✤ A growth function shows the relationship between the
size of the problem (n) and the value optimized (time)
Data Structures 12
 Asymptotic Complexity
✤ The growth function of the second dishwashing algorithm is

t(n) = 15n2 + 45n

✤ It is not typically necessary to know the exact growth
function for an algorithm
✤ We are mainly interested in the asymptotic complexity of an
algorithm – the general nature of the algorithm as n
increases
Data Structures 13
 Asymptotic Complexity
✤Asymptotic complexity is based on the dominant term of the
growth function – the term that increases most quickly as n
increases
✤The dominant term for the second dishwashing algorithm is n2:

Data Structures 14
Theoretical Analysis
 Theoretical Analysis

✤ Uses a high-level description of the algorithm instead of an

implementation
✤ Characterizes running time as a function of the input size, n
✤ Takes into account all possible inputs
✤ Allows us to evaluate the speed of an algorithm
independent of the hardware/software environment

Data Structures 16
 Counting Primitive Operations
✤ Basic computations performed
by an algorithm
Examples:
✤ Identifiable in pseudocode
❑

■ Evaluating an expression

✤ Largely independent from the ■ Assigning a value to a variable

programming language ■ Indexing into an array

✤ Exact definition not important ■ Calling a method

(we will see why later) ■ Returning from a method

✤ Assumed to take a constant ■ Performing an arithmetic

operation (for example,

amount of time in the RAM adding two numbers)
model ■ Comparing two numbers

Data Structures 17
 Counting Primitive Operations
public static int total(int a[])
{
int s=0 1 operation
for (int i=0;i<a.length;i++) 1 operation
s=s+a[i]; n+1 operations
return m; n operations
} 3n operations
1 operation
The total time is 1+1+n+1+n+3n+1 = 5n+4
Characterizes running time as a function of the input size, f(n)=5n+4
Data Structures
 Estimating Running Time
2 ops
2 ops
2n ops
2n ops
0 to n ops
1 ops

✤ Worst case: 5n + 5 primitive operations.

✤ Best case: 4n + 5 in the best case.
a = Time taken by the fastest primitive operation
b = Time taken by the slowest primitive operation
✤ Let T(n) be worst-case time of arrayMax. Then
a (5n + 5) ≤ T(n) ≤ b(5n + 5)
Data Structures 19
 Exercise
• Calculate the execution time of the following method
public int m1(int a[ ])
{
for (int i=1;i<=a.length;i++)
a[i]=a[i]+5;

int sum=0;
for (int j=1;j<=a.length;i++)
sum+=a[j];
return sum;
}

The overhead of the method call itself is generally ignored

Data Structures 20
 Exercise
public int m1(int n)
{
int c=0; 2 ops
for (int i=1;i<=n;i++) 1+2n ops
for(int j=1;j<=n;j++) n(1+2n) ops
c++; n2 ops
return c; 1 ops
}
• Total: 3+2n+n(1+2n)+n2
=3+3n+3n2
Assume
a = Tim e tak en by the fastest prim itive operation
b = Tim e tak en by the slow est prim itive operation
Let T(n) be the time of m1. Then
a (3+3n+3n2) ≤ T(n) ≤ b(3+3n+3n2)
Data Structures 21
 Exercise
• Calculate the execution time of each of the following code

for (int i=1;i<=n,i=i+2) 1+(n/2+1) +n/2 operations

cout<<“welcome”; n/2 operations

• Total: 1+3n/2+n/2

=2+4n/2
=2+2n
Data Structures 22
 Exercise
• Calculate the execution time of each of the following code
for (int i=1;i<=n,i=i*2) How many times this
loop will be repeated
cout<<“welcome”;
i will change as follows
1 2 4 8 16 32 64 128 …. n

What is the relationship between these numbers?

20 21 22 23 24 25 26 27 …. 2k

So it will repeat k times, but what is the relationship

between k and n?
If 2k = n, means k= log2n , simply we say “log n “
So total time is 1+2log n+ log n=1+3log n
Data Structures 23
Asymptotic Analysis
 Asymptotic Analysis
✤ It is a theoretical analysis where we evaluate the performance of
an algorithm in terms of input size (we don't measure the actual running
time).

A1=10n
A2=2n2

✤ In asymptotic analysis, we are more interested in the algorithm growth

rate as the input size becomes very large.
Data Structures
 Growth Rate of Running Time
✤ The growth rate T(n) is not affected by
✦ constant factors, or
✦ Lower-order terms
✤ The linear growth rate of the running time T(n) is an
intrinsic property of algorithm arrayMax
✤ Examples
✦ 10n + 105 is a linear function
✦ 105n2 + 108n is a quadratic function

The asymptotic analysis of an algorithm determines the

running time in big-Oh notation
Data Structures 26
 Why Growth Rate Matters Slide by Matt Stallmann
included with permission.

if runtime
time for n + 1 time for 2 n time for 4 n
is...

c lg n c lg (n + 1) c (lg n + 1) c(lg n + 2)

cn c (n + 1) 2c n 4c n

~ c n lg n 2c n lg n + 4c n lg n + runtime
c n lg n quadruples
+ cn 2cn 4cn
when
c n2 ~ c n2 + 2c n 4c n2 16c n2 problem
size doubles
c n3 ~ c n3 + 3c n2 8c n3 64c n3

c 2n c 2 n+1 c 2 2n c 2 4n

Data Structures 27
 Comparison of Two Algorithms
insertion sort is Slide by Matt Stallmann
n2 / 4 included with permission.

merge sort is
2 n lg n
sort a million items?
insertion sort takes
roughly 70 hours
while
merge sort takes
roughly 40 seconds
This is a slow machine, but if
100 x as fast then it’s 40 minutes
versus less than 0.5 seconds
Data Structures 28
 Big-Oh Notation
✤ The Big O notation is an asymptotic notation to describe time
complexity which means amount of time to run an algorithm.

✤ The Big O notation defines an upper bound of an

algorithm (worst case)

✤ Given functions f(n) and g(n), we say

that f(n) is O(g(n)) if there are
positive constants c and n0 such that
f(n) ≤ cg(n) for n ≥ n0
Data Structures 29
 Big-Oh Notation
10,000
3n

✤ Example: 2n + 10 is O(n) 1,000

2n+10
n
✦ 2n + 10 ≤ cn 100
✦ (c − 2) n ≥ 10
✦ n ≥ 10/(c − 2)
10

✦ Pick c = 3 and n0 = 10 1
1 10 100 1,000
n
Data Structures 30
 Big-Oh Example
✤Example: the function 1,000,000
n^2
n2 is not O(n) 100,000 100n
✦n2 ≤ cn 10,000
10n
n
✦n ≤ c 1,000
✦The above inequality 100
cannot be satisfied
10
since c must be a
1
constant 1 10
n 100 1,000
Data Structures 31
 More Big-Oh Examples
❑ 7n - 2
7n-2 is O(n)
need c > 0 and n0 ≥ 1 such that 7 n - 2 ≤ c n for n ≥ n0
this is true for c = 7 and n0 = 1

❑ 3 n3 + 20 n2 + 5
3 n3 + 20 n2 + 5 is O(n3)
need c > 0 and n0 ≥ 1 such that 3 n3 + 20 n2 + 5 ≤ c n3 for n ≥ n0
this is true for c = 28 and n0 = 1

❑ 3 log n + 5

3 log n + 5 is O(log n)
need c > 0 and n0 ≥ 1 such that 3 log n + 5 ≤ c log n for n ≥ n0
this is true for c = 8 and n0 = 2 (Note that log n is zero for n = 1)
Data Structures 32
 Big-Oh and Growth Rate
✤ The big-Oh notation gives an upper bound on the growth
rate of a function
✤ We can use the big-Oh notation to rank functions according
to their growth rate
✤ The statement “f(n) is O(g(n))” means that the growth
rate of f(n) is no more than the growth rate of g(n)
f(n) is O(g(n)) g(n) is O(f(n))
g(n) grows more Yes No
f(n) grows more No Yes
Same growth Yes Yes
Data Structures 33
 Big-Oh Rules
✤ If is f(n) a polynomial of degree d, then f(n) is O(nd), i.e.,
1. Drop lower-order terms
2. Drop constant factors
✤ Use the smallest possible class (the closest or tightest) of
functions
✦ Say “2n is O(n)” instead of “2n is O(n2)”
✤ Use the simplest expression of the class
✦ Say “3n + 5 is O(n)” instead of “3n + 5 is O(3n)”
Data Structures 34
 Common Asymptotic Functions
constant Ο(1)

logarithmic Ο(log n)

linear Ο(n)

n log n Ο(n log n)

quadratic Ο(n2)

cubic Ο(n3)

exponential Ο(2n)

Data Structures
 Growth Functions
1- Constant O(1)
• Growth is independent of the size of input data, n.
• e.g., accessing a specific array element using its index.
int firstElement(int a[])
{
return a[0];
}
2- Logarithmic O(log n)
• Growth increases slowly comparing to n.
• e.g., loop that increases doubly, also binary search
for (int i=1;i<=n,i=i*2)
cout<<“welcome”;
Data Structures
 Growth Functions
3- Linear O(n)
• Growth is directly proportional to n.
• e.g., searching for an element in an array (linear search)

boolean linearSearch(int a[], int x)

{
for (int i=0;i<a.length;i++)
cout<<“welcome”;
}

Data Structures
 Growth Functions

4- O(n log n)
• e.g., sorting using divide and conquer

5- Quadratic O(n 2) .
• e.g., typical in two nested loops

for (int i=1;i<=n,i=i++)

for (int j=1;j<=n,j=j++)
cout<<“welcome”;

Data Structures
 Growth Functions
6- Cubic O( n 3) .
• e.g., typical in three nested loops

for (int i=1;i<=n,i++)

for (int j=1;j<=n,j++)
for (int k=1;k<=n,k++)
cout<<“welcome”;

7- Exponential O(2 n) .
• e.g., it growths extremely and possibly impractical

Data Structures
Exercises
 END OF

End of
Algorithm Analysis