Algorithms

Solving Recurrences Continued

The Master Theorem

David Luebke 1 06/06/24

 Review: Solving Recurrences
● Substitution method
● Iteration method
● Master method

David Luebke 2 06/06/24

 Review: Solving Recurrences
● The substitution method
■ A.k.a. the “making a good guess method”
■ Guess the form of the answer, then use induction to
find the constants and show that solution works
■ Run an example: merge sort
○ T(n) = 2T(n/2) + cn
○ We guess that the answer is O(n lg n)
○ Prove it by induction
■ Can similarly show T(n) = Ω(n lg n), thus Θ(n lg n)

David Luebke 3 06/06/24

● Let’s solve T(n) = 2T(n/2) + n
● Guess T(n) ≤ cn log n for some constant c
(that is, T(n) = O(n log n))
● Proof: ∗ Base case: we need to show that our
guess holds for some base case (not
necessarily n = 1, some small n is ok). Ok,
● since function constant for small constant n.

David Luebke 4 06/06/24

Assume holds for n/2: T(n/2) ≤ c*(n/2) log (n/2 )
Prove that holds for n: T(n) ≤ cn log n
T(n) = 2T(n/2) + n
≤ 2(c n/2 log n/2 ) + n
= cn log n/2 + n
= cn log n − cn log 2 + n using (log a/b=loga-logb)
= cn log n − cn + n
So ok if c ≥ 1 then answer is = n log n
David Luebke 5 06/06/24
 The Iteration Method

T(n) = c + T(n/2)
T(n/2) = c + T(n/4)
T(n) = c + T(n/2) T(n/4) = c + T(n/8)
= c + c + T(n/4)
= c + c + c + T(n/8)
Assume n = 2k
T(n) = c + c + … + c + T(1)
k times
= clgn + T(1)
= Θ(lgn)
 Review: Solving Recurrences
● The “iteration method”
■ Expand the recurrence
■ Work some algebra to express as a summation
■ Evaluate the summation
● We showed several examples, were in the middle of:

 c n 1
 n
T (n)  aT
   cn n  1
  b 

David Luebke 7 06/06/24

  c n 1
 n
T (n)  aT
   cn n  1
  b 
● T(n) =
aT(n/b) + cn
a(aT(n/b/b) + cn/b) + cn
a2T(n/b2) + cna/b + cn
a2T(n/b2) + cn(a/b + 1)
a2(aT(n/b2/b) + cn/b2) + cn(a/b + 1)
a3T(n/b3) + cn(a2/b2) + cn(a/b + 1)
a3T(n/b3) + cn(a2/b2 + a/b + 1)
…
akT(n/bk) + cn(ak-1/bk-1 + ak-2/bk-2 + … + a2/b2 + a/b + 1)

David Luebke 8 06/06/24

  c n 1
 n
T (n)  aT
   cn n  1
  b 
● So we have
■ T(n) = akT(n/bk) + cn(ak-1/bk-1 + ... + a2/b2 + a/b + 1)

● For k = logb n
■ n = bk
■ T(n) = akT(1) + cn(ak-1/bk-1 + ... + a2/b2 + a/b + 1)
= akc + cn(ak-1/bk-1 + ... + a2/b2 + a/b + 1)
= cak + cn(ak-1/bk-1 + ... + a2/b2 + a/b + 1)
= cnak /bk + cn(ak-1/bk-1 + ... + a2/b2 + a/b + 1)
= cn(ak/bk + ... + a2/b2 + a/b + 1)

David Luebke 9 06/06/24

  c n 1
 n
T (n)  aT
   cn n  1
  b 
● So with k = logb n
■ T(n) = cn(ak/bk + ... + a2/b2 + a/b + 1)
● What if a = b?
■ T(n) = cn(k + 1)
= cn(logb n + 1)
= (n log n)

David Luebke 10 06/06/24

  c n 1
 n
T (n)  aT
   cn n  1
  b 
● So with k = logb n
■ T(n) = cn(ak/bk + ... + a2/b2 + a/b + 1)
● What if a < b?

David Luebke 11 06/06/24

  c n 1
 n
T (n)  aT
   cn n  1
  b 
● So with k = logb n
■ T(n) = cn(ak/bk + ... + a2/b2 + a/b + 1)
● What if a < b?
■ Recall that (xk + xk-1 + … + x + 1) = (xk+1 -1)/(x-1)

David Luebke 12 06/06/24

  c n 1
 n
T (n)  aT
   cn n  1
  b 
● So with k = logb n
■ T(n) = cn(ak/bk + ... + a2/b2 + a/b + 1)
● What if a < b?
■ Recall that (xk + xk-1 + … + x + 1) = (xk+1 -1)/(x-1)
■ So:
a k a k 1 a
 k 1     1 
a b k 1  1 
1  a b 
k 1

1
k
b b b a b   1 1  a b  1 a b

David Luebke 13 06/06/24

  c n 1
 n
T (n)  aT
   cn n  1
  b 
● So with k = logb n
■ T(n) = cn(ak/bk + ... + a2/b2 + a/b + 1)
● What if a < b?
■ Recall that (xk + xk-1 + … + x + 1) = (xk+1 -1)/(x-1)
■ So:
a k a k 1 a
 k 1     1 
a b k 1  1 
1  a b 
k 1

1
k
b b b a b   1 1  a b  1 a b

■ T(n) = cn ·(1) = (n)

David Luebke 14 06/06/24

  c n 1
 n
T (n)  aT
   cn n  1
  b 
● So with k = logb n
■ T(n) = cn(ak/bk + ... + a2/b2 + a/b + 1)
● What if a > b?

David Luebke 15 06/06/24

  c n 1
 n
T (n)  aT
   cn n  1
  b 
● So with k = logb n
■ T(n) = cn(ak/bk + ... + a2/b2 + a/b + 1)
● What if a > b?
k k 1
a a a
 k 1     1 
a b k 1  1 
  a b 
k

k
b b b a b   1

David Luebke 16 06/06/24

  c n 1
 n
T (n)  aT
   cn n  1
  b 
● So with k = logb n
■ T(n) = cn(ak/bk + ... + a2/b2 + a/b + 1)
● What if a > b?
k k 1
a a a
 k 1     1 
a b k 1  1 
  a b 
k

k
b b b a b   1
■ T(n) = cn · (ak / bk)

David Luebke 17 06/06/24

  c n 1
 n
T (n)  aT
   cn n  1
  b 
● So with k = logb n
■ T(n) = cn(ak/bk + ... + a2/b2 + a/b + 1)
● What if a > b?
k k 1
a a a
 k 1     1 
a b k 1  1 
  a b 
k

k
b b b a b   1
■ T(n) = cn · (ak / bk)
= cn · (alog n / blog n) = cn · (alog n / n)

David Luebke 18 06/06/24

  c n 1
 n
T (n)  aT
   cn n  1
  b 
● So with k = logb n
■ T(n) = cn(ak/bk + ... + a2/b2 + a/b + 1)
● What if a > b?
k k 1
a a a
 k 1     1 
a b k 1  1 
  a b 
k

k
b b b a b   1
■ T(n) = cn · (ak / bk)
= cn · (alog n / blog n) = cn · (alog n / n)
recall logarithm fact: alog n = nlog a

David Luebke 19 06/06/24

  c n 1
 n
T (n)  aT
   cn n  1
  b 
● So with k = logb n
■ T(n) = cn(ak/bk + ... + a2/b2 + a/b + 1)
● What if a > b?
a k a k 1
   
a
 1 
a b  1
k 1
  ab k

b k b k 1 b a b   1
■ T(n) = cn · (a / b )
k k

= cn · (alog n / blog n) = cn · (alog n / n)

recall logarithm fact: alog n = nlog a
= cn · (nlog a / n) = (cn · nlog a / n)

David Luebke 20 06/06/24

  c n 1
 n
T (n)  aT
   cn n  1
  b 
● So with k = logb n
■ T(n) = cn(ak/bk + ... + a2/b2 + a/b + 1)
● What if a > b?
a k a k 1
   
a
 1 
a b  1
k 1
  ab k

b k b k 1 b a b   1
■ T(n) = cn · (a / b )
k k

= cn · (alog n / blog n) = cn · (alog n / n)

recall logarithm fact: alog n = nlog a
= cn · (nlog a / n) = (cn · nlog a / n)
= (nlog a )

David Luebke 21 06/06/24

  c n 1
 n
T (n)  aT
   cn n  1
  b 
● So…

 n  ab

T (n)  n log b n  ab

  n logb a
  ab

David Luebke 22 06/06/24

 The Master Theorem
● Given: a divide and conquer algorithm
■ An algorithm that divides the problem of size n
into a subproblems, each of size n/b
■ Let the cost of each stage (i.e., the work to divide
the problem + combine solved subproblems) be
described by the function f(n)
● Then, the Master Theorem gives us a
cookbook for the algorithm’s running time:

David Luebke 23 06/06/24

 Master’s method
● “Cookbook” for solving recurrences of the form:
n
T (n)  aT    f (n)
b

where, a ≥ 1, b > 1, and f(n) > 0

Case 1: if f(n) = O(nlogba -) for some  > 0, then: T(n) = (nlogba)

Case 2: if f(n) = (nlogba), then: T(n) = (nlogba lgn)

Case 3: if f(n) = (nlogba +) for some  > 0, and if

af(n/b) ≤ cf(n) for some c < 1 and all sufficiently large n, then:
regularity condition T(n) = (f(n))
24
 Using The Master Method
● T(n) = 9T(n/3) + n
■ a=9, b=3, f(n) = n
■ nlog a = nlog
b
= (n2)
3 9

■ Since f(n) = O(nlog 9 - ), where =1, case 1 applies:

3

  
T (n)   n logb a when f (n)  O n logb a  
■ Thus the solution is T(n) = (n2)

David Luebke 25 06/06/24

 Examples

T(n) = 2T(n/2) + n

a = 2, b = 2, log22 = 1

Compare nlog22 with f(n) = n

 f(n) = (n)  Case 2

 T(n) = (nlgn)
26
 Examples
T(n) = 2T(n/2) + n2
a = 2, b = 2, log22 = 1
Compare n with f(n) = n2
 f(n) = (n1+) Case 3  verify regularity cond.

a f(n/b) ≤ c f(n)
 2 n2/4 ≤ c n2  c = ½ is a solution (c<1)
 T(n) = (n2)
27
 Examples (cont.)

T(n) = 2T(n/2) + n

a = 2, b = 2, log22 = 1

Compare n with f(n) = n1/2

 f(n) = O(n1-) Case 1

 T(n) = (n)
28
 Recursion-tree method
● A recursion tree models the costs (time) of a
recursive execution of an algorithm.
 Example of recursion tree

Solve T(n) = T(n/4) + T(n/2) + n2:

 Example of recursion tree
Solve T(n) = T(n/4) + T(n/2) + n2:

T(n)
 Example of recursion tree
Solve T(n) = T(n/4) + T(n/2) + n2:

n2

T(n/4) T(n/2)
 Example of recursion tree
Solve T(n) = T(n/4) + T(n/2) + n2:

n2

(n/4)2 (n/2)2

T(n/16) T(n/8) T(n/8) T(n/4)

 Example of recursion tree
Solve T(n) = T(n/4) + T(n/2) + n2:

n2

(n/4)2 (n/2)2

(n/16)2 (n/8)2 (n/8)2 (n/4)2

…

(1)
 Example of recursion tree
Solve T(n) = T(n/4) + T(n/2) + n2:

n2
n2
(n/4)2 (n/2)2

(n/16)2 (n/8)2 (n/8)2 (n/4)2

…

(1)
 Example of recursion tree
Solve T(n) = T(n/4) + T(n/2) + n2:

n2
n2
(n/4)2 (n/2)2 5 n2
16
(n/16)2 (n/8)2 (n/8)2 (n/4)2
…

(1)
 Example of recursion tree
Solve T(n) = T(n/4) + T(n/2) + n2:

n2
n2
(n/4)2 (n/2)2 5 n2
16
(n/16)2 (n/8)2 (n/8)2 (n/4)2 25 n 2
256
…

…
(1)
 Example of recursion tree
Solve T(n) = T(n/4) + T(n/2) + n2:

n2
n2
(n/4)2 (n/2)2 5 n2
16
(n/16)2 (n/8)2 (n/8)2 (n/4)2 25 n 2
256
…

…
(1)
Total =
n 2
= (n )

2
5 5
1  16  16
2
    5 3
16
geometric series
 
 Examples

Ex. T(n) = 4T(n/2) + n

a = 4, b = 2  nlogba = n2; f (n) = n.

f (n) = O(n2 – ) for  = 1 => Case 1

 T(n) = (n2).

Ex. T(n) = 4T(n/2) + n2

a = 4, b = 2  nlogba = n2; f (n) = n2.
f (n) = (n2) => Case 2
 T(n) = (n2lg n).
 Examples
Ex. T(n) = 4T(n/2) + n3
a = 4, b = 2  nlogba = n2; f (n) = n3.
f (n) = (n2 + ) for  = 1 => Case 3
and 4(cn/2)3  cn3 (reg. cond.) for c = 1/2.
 T(n) = (n3).