Data Structures – LECTURE 3 Complexity analysis of an algorithm

Two main methods:

Recurrence equations • Direct counting: sum of the individual steps times
• Formulating recurrence equations the number of times executed T(n) = citi
• Solving recurrence equations Best for repeated iterations (loops).
• Recurrence equation: an equality or inequality
• The master theorem (simple and extended versions)
describing the function in terms of its behavior on
• Examples: Merge-Sort and Quick-Sort smaller inputs: T(n) = T(n –1) + c; T(1) = 1.
the solution of the equation is T(n) = O(n2).
Best for recursive functions and structures.
Data Structures, Spring 2006 © L. Joskowicz Data Structures, Spring 2006 © L. Joskowicz

Recurrence equations Formulating recurrence equations

• Simplifying assumptions • Consider
– n is sufficiently large. – in how many sub-problems the problem is split
– T(1) = (1) for sufficiently small n. A value changes – what is the size of each sub-problem
the solution of the equation, but usually only by a – how much work is required to combine the results of
constant factor, so the order of growth is unchanged! each sub-problem
– Choose n according to boundary conditions: n is even • Recursion tree
(n=2k), a power of two (n=2k) where k >0 is an integer. n
• Formulation: n/2 n/2
Be very careful with the constants!
n/4 n/4 n/4 n/4
T(n) is not the same as T(n/2)!
Data Structures, Spring 2006 © L. Joskowicz Data Structures, Spring 2006 © L. Joskowicz

Common recurrence equations (1) Common recurrence equations (2)

• Factorial: multiply n by (n –1)! • Binary search: see if the root of the tree is the one we are
T(n) = T(n – 1) + O(1) O(n) looking for, and if not, recursively call with either the
• Fibonacci: add fibonacci(n – 1) and fibonacci(n – 2) left or right subtree, which has half the elements
T(n) = T(n – 1) + T(n – 2) O(2n) T(n) = T(n/2) + O(1) O(lg n)
• Sequential search: see if the first element is the one • Binary tree traversal: visit all the nodes of a tree by
we are looking for, and if not, recursively call with one recursively visiting the nodes of the left and right tree:
element less:
T(n) = 2T(n/2) + O(1) O(n)
T(n) = T(n – 1) + O(1) O(n)
• Merge Sort: split the list into two equal-sized parts,
• Insertion sort: find the place of the first element in the
recursively sort each, and merge the resulting lists:
sorted list, and recursively call with one element less:
T(n) = T(n – 1) + O(n) O(n2) T(n) = 2T(n/2) + O(n) O(n lg n)

Data Structures, Spring 2006 © L. Joskowicz Data Structures, Spring 2006 © L. Joskowicz
 Solving recurrence equations The substitution method
The solution to the equation T(n) = 2T(n/2) + n is O(n lg n)
Three ways to solve recurrence equations: for n 2; assume T(1) = 1
• Substitution: guess a bound and use mathematical Prove: T(n) c(n lg n) for c 2
induction to prove the guess correct. Base case: T(2) c 2lg2, which holds for c 2 since T(2) = 3
General case:
• Recursion-tree: convert the recurrence into a tree
Assume that it holds for n/2, that is: T(n/2) 2(cn/2 lg (n/2))
whose nodes represent the costs at each level and
Substitute into the recurrence relation and prove for n:
use bounding summations to solve the recurrence.
T(n) 2(cn/2 lg (n/2) + n
• Master method: apply a theorem for recurrences cn lg (n/2) + n
of the form T(n) = aT(n/b) + nc cn lg n – cn lg 2 + n
where a, b, c are constants. cn lg n – cn + n
cn lg n for c 1
Data Structures, Spring 2006 © L. Joskowicz Data Structures, Spring 2006 © L. Joskowicz

Finding patterns in recurrences (1) Finding patterns in recurrences (2)

Write several elements of the recursion, and see if you can
find a pattern. Once you find the pattern, prove it is true
by substitution (induction) T(n) = T(n – k) + nk – ((k – 1)k)/2
T(n) = T(n – 1) + n
At the end of the recursion, k = n –1 and T(1) = 1, so we get:
T(n –1) = T(n – 2) + (n –1) T(n) = 1 + n2 – n + n2/2 – 3n/2 – 1
T(n –2) = T(n – 3) + (n –2) = n2/2 – n/2
T(n –3) = T(n – 4) + (n –3) = O(n2)
Now substitute:
T(n) = T(n – 1) + n
So the guess is that O(n2) is the solution to the recurrence
= [T(n – 2) + (n –1)] + n
T(n) = T(n – 1) + n
= [[T(n – 3) + (n –2)] +(n –1)] + n
= [[[T(n – 4) + (n –3)] + (n –2)] +(n –1)] + n
= T(n – k) + k
i=1(n –i+1) = T(n – k) + nk – ((k – 1)k)/2
Data Structures, Spring 2006 © L. Joskowicz Data Structures, Spring 2006 © L. Joskowicz

The master theorem (simple version) Useful math to know…

Logarithms
Let T(n) be defined on non-negative integers by the • logc(ab) = logca + logcb
recurrence: • logban = nlogba
T(n) = aT(n/b) + nc • logban = nlogba
where a ≥ 1, b > 1, c ≥ 0 are constants • logb(1/a) = – logba
See Appendix A in book
• logba = 1 / logab
Then for many useful tricks!
• a = b log b a
1. T(n) = (nc) when a/bc < 1 (logba < c) • a log b c = c log b a
2. T(n) = (n logb n) when a/bc = 1 (logba = c)
c k
c k +1 − 1
Geometric series: ci =
3. T(n) = Θ( n log b a ) when a/bc > 1 (logba > c) i =0 c −1

Data Structures, Spring 2006 © L. Joskowicz Data Structures, Spring 2006 © L. Joskowicz
 Recurrence equation of master theorem Recursion-tree for the equation
T(n) = aT(n/b) + nc i=0 n a c
n
T(n/b) = aT(n/b2) + (n/b)c c c c c a a
i=1 n n n ... n b
T(n/b2) = aT(n/b3) + (n/b2)c b b b b
a a
T(n/b4) = aT(n/b5) + (n/b4)c c c c c c
i=2 n ... n ... n ... n 2 n
Now substitute: a2 a
b2 b2 b2 b2 b2
T(n) = aT(n/b) + nc ...
= a[aT(n/b2) + (n/b)c] + nc c
... c c
... ...
c
n n n n c
= a[a[aT(n/b3) + (n/b2)c]+ (n/b)c] + nc i=k
bk bk bk
... ... n
ak
bk bk
= akT(n/bk) + nc[1 + a(1/b)c + a2(1/b2)c +…ak–1 (1/bk–1 )c]
k −1 c
n k= logbn, the depth of the recursion
= a k T ( n bk ) + ai k= logbn, is the depth log b n −1
n
c

i =0 bi
of the recursion
n
= Θ(1) Θ( n log b a ) + ai
Data Structures, Spring 2006 © L. Joskowicz Data Structures, Spring 2006 © L. Joskowicz
b log b n i =0 bi

Master theorem proof Master theorem: Case 1

a log b n −1 i

The number of comparisons is: Case 1: <1 nc

a
bc i =0 bc
logb n −1 c
n
Θ( n logb a
)+ a i i
= log b n
b a
i =0 0 log b n −1 i 1−
a a bc 1
logb n −1 i 1= < = < < const.
a bc bc a a
Θ( n logb a
)+n c i =0
1− c 1−
bc b bc
i =0

which depends on the value of a

Therefore, T(n) = (nc)
bc
Data Structures, Spring 2006 © L. Joskowicz Data Structures, Spring 2006 © L. Joskowicz

Master theorem: Case 2 Master theorem: Case 3

a log b n −1 i a log b n −1 i
Case 2: =1 nc
a Case 3: >1 nc
a
bc i =0 bc bc i =0 bc
logb n −1 b i log n
a a
= Θ
logb n −1
a
i
i =0 bc bc
= log b n logb n logb n
bc a c a
i =0 ncΘ = Θ n
bc bc
logb n c
a n nc
nc c = c logb n .a logb n = c .n logb a = n logb a
Therefore, T(n) = (nc logb n) b b n
Therefore, T ( n ) = Θ( n log b a
)
Data Structures, Spring 2006 © L. Joskowicz Data Structures, Spring 2006 © L. Joskowicz
 Example 1: Merge-Sort Example 2
The recurrence equation is: Solve the recurrence
T(n) = 2T(n/2) + n T ( n ) = 9T ( n / 3) + n
Here, a = 2, b = 2, c = 1 Here a = 9, b = 3, c = 1
Case 2 applies: a/bc = 2/21 = 1

T (n ) = Θ( n1 log 2 n ) Case 3 applies: a/bc = 9/31 = 3 > 1

Conclusion: n log b a = nlog 3 9 = n 2

Conclusion:
T (n) = Θ(n lg n)
T ( n) = Θ n 2 ( )
Data Structures, Spring 2006 © L. Joskowicz Data Structures, Spring 2006 © L. Joskowicz

The master theorem (general version) Recursion-tree for the equation

Let a ≥ 1, b > 1 be constants, let f (n) be a function, i=0 n a
n
and let T(n) be defined on non-negative integers by a af
the recurrence: i=1 f
n
f
n ... f
n b
T (n) = aT (n / b) + f (n) a b b b
a
i=2 n ... f n ... n ... f n n
where n/b is either n / b or n / b f f 2
a2 a f b
( ) ( )
b2 b2 b2 b2
1. If f ( n) = O n log b a −ε ε > 0 then T ( n) = Θ n log a b
...
... ...
( )
...
2. If f (n) = Θ n then T (n) = Θ(n log a lg n )
log a b b

... Θ(1)
( )
i=k Θ(1) Θ(1) Θ(1) Θ(1) ak
log b a +ε
3. If f (n) = Ω n ε > 0 and if a f (n / b) ≤ c f ( n)
log b n −1
for some c < 1 and sufficiently large n, then k= logbn, the depth of the recursion n
Θ( n log b a ) + ai f
T ( n ) = Θ( f ( n ) ) i =0 bi
Data Structures, Spring 2006 © L. Joskowicz Data Structures, Spring 2006 © L. Joskowicz

The master theorem – intuition Example 1

Compares two terms: O n and f (n) ( log b a
) Solve the recurrence
1. When O (n log a ) dominates, the complexity is
b T ( n ) = T ( 2 n / 3) + 1
T ( n) = Θ(n log b
a
) Here a = 1, b = 3/2, f (n) = 1 and

2. When f (n) dominates, the complexity is

n log b
a
= n log 3/ 2
1
= n0 = 1
Case 2 applies:
T ( n ) = Θ( f ( n ) ) f (n) = Θ(1)
3. When they are comparable, there is a lg n penalty Conclusion:
T (n) = Θ n ( log b a
)
lg n = Θ( f (n) lg n )
T ( n) = Θ n 0 lg n = Θ(lg n ) ( )
Data Structures, Spring 2006 © L. Joskowicz
see book for formal proof! Data Structures, Spring 2006 © L. Joskowicz
 Recurrence equations to remember
• T(n) = T(n – 1) + O(1) O(n)

• T(n) = T(n – 1) + O(n) O(n2)

• T(n) = 2T(n – 1) + O(1) O(2n)

• T(n) = T(n/2) + O(1) O(lg n)

• T(n) = 2T(n/2) + O(1) O(n)

• T(n) = 2T(n/2) + O(n) O(n lg n)

Data Structures, Spring 2006 © L. Joskowicz