Lecture 2

Fundamentals of the Analysis of

Algorithm Efficiency

1
 Lecture Contents

1. Analysis Framework
2. Asymptotic Notations and Basic Efficiency Classes
3. Mathematical Analysis of Nonrecursive Algorithms
4. Mathematical Analysis of Recursive Algorithms
5. Example: Fibonacci Numbers
6. Empirical Analysis of Algorithms
7. Algorithm Visualization

2
 Analysis of Algorithms

• Issues:
- correctness
- time efficiency
- space efficiency
- optimality
• Approaches:
- theoretical analysis
- empirical analysis
3
 Theoretical Analysis of Time Efficiency

• Time efficiency is analyzed by determining the

number of repetitions of the basic operation as
a function T(n) of input size n

• Basic operation: the operation that contributes

most towards the running time of the algorithm

4
Theoretical Analysis of Time Efficiency

input size

T(n) ≈ copC(n)
running time execution time Number of times
for basic operation basic operation is
executed

5
 Input Size and Basic Operation Examples

Problem Input size measure Basic operation

Searching for key in a list of Number of list’s items,

Key comparison
n items i.e. n

Multiplication of two Matrix dimensions or total Multiplication of two

matrices number of elements numbers

Checking primality of a n’s size = number of digits (in

Division
given integer n binary representation)

Visiting a vertex or
Typical graph problem #vertices and/or edges
traversing an edge

6
 Empirical Analysis of Time Efficiency

• Select a specific (typical) sample of inputs

• Use physical unit of time (e.g., milliseconds)

or
Count actual number of basic operation’s
executions

• Analyze the empirical data

7
 Worst-case, Best-case, Average-case Efficiencies

• For some algorithms, efficiency depends on form

of input, so we need to do analysis for

- Worst case Cworst(n): determine the kind of inputs

for which the count C(n) will be the largest
among all possible inputs of size n.
- Best case Cbest(n): determine the kind of inputs
for which the count C(n) will be the smallest
among all possible inputs of size n.
8
 Worst-case, Best-case, Average-case Efficiencies

- Average case: Cavg(n)

◦ NOT the average of worst and best cases
◦ Expected number of basic operations
considered as a random variable under some
assumption about the probability distribution
of all possible inputs

9
 Example: Sequential Search

Algorithm SequentialSearch(A[0..n – 1], K)

//Searches for given value in given array by sequential search
//Input: Array A[0..n – 1] and search key K
//Output: Index of 1st element of A that matches K
// or –1 if there are no matching elements
i←0
while i < n and A[i] ≠ K do
i←i+1
if i < n return i
else return –1

10
 Example: Sequential Search

• Worst case
Cworst(n) = n
The last element is equal to a search key or
unsuccessful
• Best case
Cbest(n) = 1
The first element is equal to a search key

11
 Example: Sequential Search

• Average case
- We have the following assumptions:
a. probability of successful search is p (0 ≤ p ≤ 1)
b. probability of first match occurring in ith
position is the same for every i.

12
 Example: Sequential Search

• Average case
- successful search, probability of first match
occurring in ith position is p/n for every i, and
#comparisons made by the algorithm in such a
situation is obviously i.
- unsuccessful search, #comparisons is n with
probability of such a search being (1 – p).

13
 Example: Sequential Search

• Average case

p = 1 (successful), Cavg(n) = (n + 1)/2

p = 0 (unsuccessful), Cavg(n) = n
14
 Types of Formulas for Basic Operation’s Count

• Exact formula
e.g., C(n) = n(n-1)/2
• Approximate formula with specific multiplicative
constant. e.g., C(n) ≈ 0.5 n2
• Approximate formula with unknown
multiplicative constant. e.g., C(n) ≈ cn2

15
 Order of Growth

• Efficiency analysis ignores a multiplicative

constant and concentrates on the count’s order
of growth for large input sizes (i.e., n → ∞)
i.e., log2 n < n < n log2 n < n2 < n3 < 2n < n! for n
large enough

16
Values of Some Important Functions as n → 

17
 Lecture Contents

1. Analysis Framework
2. Asymptotic Notations and Basic Efficiency
Classes
3. Mathematical Analysis of Nonrecursive Algorithms
4. Mathematical Analysis of Recursive Algorithms
5. Example: Fibonacci Numbers
6. Empirical Analysis of Algorithms
7. Algorithm Visualization

18
 Asymptotic Order of Growth

• A way of comparing functions that ignores

constant factors and small input sizes

• O(g(n)): set of functions t(n) that grow no faster

than g(n)
• Ω(g(n)): set of functions t(n) that grow at least as
fast as g(n)
• Θ(g(n)): set of functions t(n) that grow at same
rate as g(n)
19
 Formal Definition of Big-Oh O

• A function t(n) is said to be in O(g(n)), denoted

as t(n)  O(g(n)), if t(n) is bounded above by
some constant multiple of g(n) for all large n,
i.e., if there exist some positive constant c and
some nonnegative integer n0 such that
t(n) ≤ cg(n) for all n ≥ n0.
// g(n) is upper bound of t(n)
• Examples: n  O(n2) and 100n + 5  O(n2),
n(n - 1)/2  O(n2)
20
 Big-Oh

Figure 2.1 Big-oh notation: t(n)  O(g(n)).

21
 Formal Definition of Big-Omega Ω

• A function t(n) is said to be in Ω(g(n)), denoted

as t(n)  Ω(g(n)), if t(n) is bounded below by
some positive constant multiple of g(n) for all
large n, i.e., if there exist some positive constant
c and some nonnegative integer n0 such that
t(n) ≥ cg(n) for all n ≥ n0.
// g(n) is lower bound of t(n)
• Examples: n3  Ω(n2), n(n – 1)/2  Ω(n2)

22
 Big-Omega

Figure 2.2 Big-omega notation: t(n)  Ω(g(n)).

23
 Formal Definition of Big-Theta Θ

• A function t(n) is said to be in (g(n)), denoted

as t(n)  (g(n)), if t(n) is bounded both above
and below by some positive constant multiples
of g(n) for all large n, i.e., if there exist some
positive constant c1 and c2 and some nonnegative
integer n0 such that
c2g(n) ≤ t(n) ≤ c1g(n) for all n ≥ n0.
• Examples: n(n – 1)/2  (n2)

24
 Big-Theta

Figure 2.3 Big-omega notation: t(n)  (g(n)).

25
 Some Properties of Asymptotic Order of Growth

• f(n)  O(f(n))
• f(n)  O(g(n)) iff g(n)  (f(n))
• If f (n)  O(g (n)) and g(n)  O(h(n)) , then f(n)
 O(h(n))
// Note similarity with a ≤ b
• If t1(n)  O(g1(n)) and t2(n)  O(g2(n)) , then
t1(n) + t2(n)  O(max{g1(n), g2(n)})
(The analogous assertions are true for the Ω and 
notations as well.)
26
Using Limits for Comparing Order of Growth

27
 L’Hôpital’s Rule and Stirling’s Formula

L’Hôpital’s rule

// f ’(a): the derivative of f at a

Stirling’s formula

28
 Example

Compare the orders of growth of n(n - 1)/2 and n2.

Since the limit is equal to a positive constant,

the functions have the same order of growth,
or symbolically, n(n – 1)/2  (n2).
29
 Orders of Growth of Some Important Functions

• All logarithmic functions loga n belong to the same

class (log n) no matter what the logarithm’s base
a > 1 is
• All polynomials of the same degree k belong to
the same class: aknk + ak-1nk-1 + … + a0  (nk)
• Exponential functions an have different orders of
growth for different a’s
• order log n < order n ( > 0) < order an < order
n! < order nn
30
Basic Asymptotic Efficiency Classes

1 constant
log n logarithmic
n linear
n log n n-log-n
n2 quadratic
n3 cubic
2n exponential
n! factorial
31
 Lecture Contents

1. Analysis Framework
2. Asymptotic Notations and Basic Efficiency Classes
3. Mathematical Analysis of Nonrecursive
Algorithms
4. Mathematical Analysis of Recursive Algorithms
5. Example: Fibonacci Numbers
6. Empirical Analysis of Algorithms
7. Algorithm Visualization

32
 Time Efficiency of Nonrecursive Algorithms

• General Plan for Analysis

- Decide on parameter n indicating input size

- Identify algorithm’s basic operation
- Determine worst, average, and best cases for input of
size n
- Set up a sum for the number of times the basic
operation is executed
- Simplify the sum using standard formulas and rules
(see Appendix A)
33
 Useful Summation Formulas and Rules

1in i2 = 12+22+…+n2
= n(n+1)(2n+1)/6  n3/3  (n3)
0in ai = 1 + a +…+ an
= (an+1 – 1)/(a – 1) for any a  1
0in 2i = 20 + 21 +…+ 2n
= 2n+1 – 1  (2n )
(ai ± bi ) = ai ± bi, cai = cai,
liuai = limai + m+1iuai

34
 Example 1: Maximum Element

Algorithm MaxElement(A[0..n – 1])

//Determines value of the largest element in a given array
//Input: An array A[0..n – 1] of real numbers
//Output: The value of the largest element in A
maxval ← A[0]
for i ← 1 to n – 1 do
if A[i] > maxval
maxval ← A[i]
return maxval

35
 Example 2: Element Uniqueness Problem

Algorithm UniqueElements(A[0..n – 1])

//Determines whether all elements in array are distinct
//Input: An array A[0..n – 1]
//Output: Returns “true” if all elements in A are distinct
and “false” otherwise
for i ← 0 to n – 2 do
for j ← i + 1 to n – 1 do
if A[i] = A[j] return false
return true

36
 Example 3: Matrix Multiplication

Algorithm MatrixMultiplication(A[0..n – 1, 0..n – 1],

B[0..n – 1, 0..n – 1])
//Multiplies two n-by-n matrices by the definition-based algorithm
//Input: Two n-by-n matrices A and B
//Output: Matrix C = AB
for i ← 0 to n – 1 do
for j ← 0 to n – 1 do
C[i, j] ← 0.0
for k ← 0 to n – 1 do
C[i, j] ← C[i, j] + A[i, k] × B[k, j]
return C

37
 Example 4: Counting Binary Digits

Algorithm Binary(n)
//Input: A positive decimal integer n
//Output: The number of binary digits in n’s binary
representation
count ← 1
while n > 1 do
count ← count + 1
n ← n/2
return count

Prove log2 n + 1 = log2 (n + 1), where integer n > 0.

38
 Lecture Contents

1. Analysis Framework
2. Asymptotic Notations and Basic Efficiency Classes
3. Mathematical Analysis of Nonrecursive Algorithms
4. Mathematical Analysis of Recursive
Algorithms
5. Example: Fibonacci Numbers
6. Empirical Analysis of Algorithms
7. Algorithm Visualization

39
 Plan for Analysis of Recursive Algorithms

• Decide on a parameter indicating an input’s size.

• Identify the algorithm’s basic operation.
• Check whether the number of times the basic
operation is executed may vary on different
inputs of the same size. (If it may, the worst,
average, and best cases must be investigated
separately.)

40
 Plan for Analysis of Recursive Algorithms

• Set up a recurrence relation with an appropriate

initial condition expressing the number of times
the basic operation is executed.
• Solve the recurrence (or, at the very least,
establish its solution’s order of growth) by
backward substitutions or another method.

41
 Example 1: Recursive Evaluation of n!

Algorithm F(n)
//Computes n! recursively
//Input: A nonnegative integer n
//Output: The value of n!
if n = 0 return 1
else return F(n – 1) × n

42
 Solving the Recurrence for M(n)

• Method of backward substitutions

M(n) = M(n – 1) + 1
Substitute M(n – 1) = M(n – 2) + 1
M(n) = [M(n – 2) + 1] + 1 = M(n – 2) + 2
Substitute M(n – 2) = M(n – 3) + 1
M(n) = [M(n – 3) + 1] + 2 = M(n – 3) + 3
...
M(n) = M(n – i) + i
...
M(n) = M(n – n) + n = n  (n)
(initial condition M(0) = 0 for i = n)
43
Example 2: The Tower of Hanoi Puzzle

44
 Solving Recurrence for Number of Moves

• Recurrence equation

M(n) = 2M(n – 1) + 1 for n > 1,

M(1) = 1. // initial condition for i = n – 1)

• Method of backward substitutions

M(n) = 2M(n – 1) + 1
Substitute M(n – 1) = 2M(n – 2) + 1
M(n) = 2[2M(n – 2) + 1] + 1 = 22M(n – 2) + 2 + 1
Substitute M(n – 2) = 2M(n – 3) + 1
45
 Solving Recurrence for Number of Moves

M(n) = 22[2M(n – 3) + 1] + 2 + 1
= 23M(n – 3) + 22 + 2 + 1.
...
= 2iM(n – i) + 2i-1 + 2i-2 + … + 2 + 1
= 2iM(n – i) + 2i – 1. // Appendix A
...
// M(1) = 1 when i = n – 1
M(n) = 2n-1M(n – (n – 1)) + 2n-1 – 1
M(n) = 2n-1M(1) + 2n-1 – 1 = 2n-1 + 2n-1 – 1 =
M(n) = 2n – 1  (2n). // exponential algorithm
46
 Example 3: Count #bits Recursively

Algorithm BinRec(n)
//Input: A positive decimal integer n
//Output: The number of binary digits in n’s binary
representation
if n = 1 return 1
else return BinRec(n/2) + 1

47
 Example 3: Count #bits Recursively

• Recurrence relation

A(n) = A(n/2) + 1 for n > 1.

A(1) = 0, // initial condition for n = 1

• Let us solve the recurrence for n = 2k

A(2k) = A(2k-1) + 1 for k > 0,

A(20) = 0. // k = 0

48
 Example 3: Count #bits Recursively

• Method of backward substitutions

A(2k) = A(2k-1) + 1
Substitute A(2k-1) = A(2k-2) + 1
A(2k) = [A(2k-2) + 1] + 1 = A(2k-2) + 2
Substitute A(2k-2) = A(2k-3) + 1
A(2k) = [A(2k-3) + 1] + 2 = A(2k-3) + 3
...
A(2k) = A(2k-i) + i
…
A(2k) = A(2k-k) + k.
49
 Example 3: Count #bits Recursively

→ A(2k) = A(1) + k = k

n = 2k → k = log2 n

→ A(n) = log2 n  (log n).

• The exact solution for an arbitrary value of n is

A(n) = log2 n.

50
 Lecture Contents

1. Analysis Framework
2. Asymptotic Notations and Basic Efficiency Classes
3. Mathematical Analysis of Nonrecursive Algorithms
4. Mathematical Analysis of Recursive Algorithms
5. Example: Fibonacci Numbers
6. Empirical Analysis of Algorithms
7. Algorithm Visualization

51
 Fibonacci Numbers

The Fibonacci numbers:

0, 1, 1, 2, 3, 5, 8, 13, 21, …
Mathematical definition of Fibonacci sequence
F(n) = F(n - 1) + F(n - 2)
F(0) = 0, F(1) = 1 // initial conditions
How to compute the nth Fibonacci number?

52
 Computing the nth Fibonacci Number

1. Explicit formula  (n),

2. Recursive definition-based algorithm  (n)
3. Nonrecursive definition-based algorithm  (n)
4. Logarithmic algorithm based on formula:

n
F(n-1) F(n) 0 1
=
F(n) F(n+1) 1 1

for n ≥ 1
 (log n)
53
 Explicit Formula for the nth Fibonacci Number

F(n) = F(n - 1) + F(n - 2) is rewritten as

F(n) - F(n - 1) - F(n - 2) = 0
Its characteristic equation is r2 - r - 1 = 0,
The roots of r2 - r - 1 = 0 are

54
 Explicit Formula for the nth Fibonacci Number

According to Case 1 of Theorem 1 in Appendix B,

we have

With the initial conditions F(0) = 0, F(1) = 1, we

have

55
 Explicit Formula for the nth Fibonacci Number

Substituting  = - into the second equation, we

get  = 1/ 5 and  = -1/ 5 .

56
Explicit Formula for the nth Fibonacci Number

57
 Compute the nth Fibonacci Number Recursively

Algorithm F(n)
//Computes the nth Fibonacci number recursively
by using its definition
//Input: A nonnegative integer n
//Output: The nth Fibonacci number
if n ≤ 1 return n
else return F(n – 1) + F(n – 2)

58
 Compute the nth Fibonacci Number Recursively

• The number of additions is

A(n) = A(n – 1) + A(n – 2) + 1 for n > 1,
A(0) = 0, A(l) = 0. // initial conditions
• Recast A(n) as follows
[A(n) + 1] – [A(n – 1) + 1] – [A(n – 2) + 1] = 0
• Substitue A(n) + 1 = B(n), we get
B(n) – B(n – 1) – B(n – 2) = 0
B(0) = 1, B(1) = 1.
59
 Compute the nth Fibonacci Number Recursively

The recurrence B(n) is the same as F(n) except that

B(n) starts with two ones and thus B(n) runs one
step ahead of F(n). Thus, B(n) = F(n + 1), and

A(n) = B(n) – 1 = F(n + 1) – 1

60
 Compute the nth Fibonacci Number Iteratively

Algorithm Fib(n)
//Computes the nth Fibonacci number iteratively by using
its definition
//Input: A nonnegative integer n
//Output: The nth Fibonacci number
F[0] ← 0; F[1] ← 1
for i ← 2 to n do
F[i] ← F[i – 1] + F[i – 2]
return F[n]

This algorithm clearly makes n – 1 additions. Hence, A(n)

 (n). 61
 (log n) Algorithm to Compute the nth Fibonacci Number

• It is based on the formula

62
 Lecture Contents

1. Analysis Framework
2. Asymptotic Notations and Basic Efficiency Classes
3. Mathematical Analysis of Nonrecursive Algorithms
4. Mathematical Analysis of Recursive Algorithms
5. Example: Fibonacci Numbers
6. Empirical Analysis of Algorithms
7. Algorithm Visualization

63
 Lecture Contents

1. Analysis Framework
2. Asymptotic Notations and Basic Efficiency Classes
3. Mathematical Analysis of Nonrecursive Algorithms
4. Mathematical Analysis of Recursive Algorithms
5. Example: Fibonacci Numbers
6. Empirical Analysis of Algorithms
7. Algorithm Visualization

64