Analysis of Algorithms 3

Input Algorithm Output

1
Topics
 Math you need to review
 Asymptotic notations
 Solving recurrence
 Guess-and-test method

2
Math you need to review
 Summations  Properties of powers:
a(b+c) = abac
 Arithmetic series
abc = (ab)c
 Geometric series ab/ac = a(b-c)
 … b = a logab
 Powers bc = a c*logab
Properties of logarithms:
Logarithms


logb(xy) = logbx + logby
 Proof techniques logb (x/y) = logbx - logby
 Induction logbxa = a logbx
 Basic probability logba = logxa/logxb

3
Big-Oh Notation
(review)
 Given functions f(n) and
g(n), we say that f(n) is
O(g(n)) if there are
positive constants
10,000
c and n0 such that 3n

f(n) ≤ cg(n) for n ≥ n0 1,000 2n+10

 Example: 2n + 10 is O(n) n

 2n + 10 ≤ cn 100

 10 ≤ (c − 2) n
 If c is 3, 10 ≤ n 10

 Pick c = 3 and n0 = 10
1
1 10 100 1,000
n 4
Big-Oh Example
1,000,000
n^2
 Example: the function 100n
100,000
n2 is not O(n) 10n
 n2 ≤ cn 10,000 n

 n≤c
 The above inequality 1,000
cannot be satisfied
since c must be a 100
constant
10

1
1 10 100 1,000
n

5
More Big-Oh Examples
 7n2 - 2
7n-2 is O(n3) (or 7n-2 = O(n3) )
need c > 0 and n0 ≥ 1 such that 7 n2 - 2 ≤ c n3 for n ≥ n0
This is true for c = 7 and n0 = 1

 3 n3 + 20 n2 + 5
3 n3 + 20 n2 + 5 is O(n3)
need c > 0 and n0 ≥ 1 such that 3 n3 + 20 n2 + 5 ≤ c n3 for n ≥ n0
This is true for c = 4 and n0 = 21

 3 log n + 5
3 log n + 5 is O(log n)
need c > 0 and n0 ≥ 1 such that 3 log n + 5 ≤ c log n for n ≥ n0
This is true for c = 8 and n0 = 2
6
Big-Oh and Growth Rate
 The big-Oh notation gives an upper bound on the
growth rate of a function
 The statement “f(n) is O(g(n))” means that the growth
rate of f(n) is no more than the growth rate of g(n)
 We can use the big-Oh notation to rank functions
according to their growth rate
f(n) is O(g(n)) g(n) is O(f(n))
g(n) grows more Yes No
f(n) grows more No Yes
Same growth Yes Yes
7
Set definition of Big-Oh
 O(g(n)) = { f(n) : there exist
constants c > 0, n0 > 0 such that 0 ≤
f(n) ≤ cg(n) for all n ≥ n0}
 EXAMPLE: 2n2 ∈ O(n3)

8
Macro substitution
Convention: A set in a formula represents
an anonymous function in the set.

9
Macro substitution
Convention: A set in a formula represents
an anonymous function in the set.
EXAMPLE: f(n) = n3 + O(n2)
means
f(n) = n3 + h(n)
for some h(n) ∈ O(n2) .

10
Macro substitution
Convention: A set in a formula represents
an anonymous function in the set.
EXAMPLE: n2 + O(n) = O(n2)
means
for any f(n) ∈ O(n):
n2 + f(n) = h(n)
for some h(n) ∈ O(n2) .

11
Ω-notation
Big-Omega (Ω)
 f(n) is Ω(g(n)) if there is a  Ω-notation is a lower-bound
constant c > 0 and notation.
an integer constant n0 ≥ 1  It makes no sense to say f(n)
such that is at most Ω(n2).
f(n) ≥ c g(n) for n ≥ n0

 Set definition of Ω-notation

Ω(g(n)) = { f(n) : there exist
constants c > 0, n0 > 0 such that 0 ≤
cg(n) ≤ f(n) for all n ≥ n0}
Notice 2 solutions
(4, 16) to 𝑛𝑛 = lg 𝑛𝑛
 EXAMPLE:
𝑛𝑛 = Ω( lg 𝑛𝑛) (c = 1, n0 = 16)
12
Θ-notation (Review)
Big-Theta (Θ)
 f(n) is Θ(g(n)) if there are constants c1 > 0 and c2 > 0
and an integer constant n0 ≥ 1 such that
c1g(n) ≤ f(n) ≤ c2g(n) for n ≥ n0
 Θ -notation is a tight-bound notation.
 Θ(g(n)) = O(g(n)) ∩ Ω(g(n))

 EXAMPLE:
1 2
𝑛𝑛 − 2𝑛𝑛 = Θ(𝑛𝑛2 )
2

13
Intuition for Asymptotic
Notation
big-Oh
 f(n) is O(g(n)) if f(n) is asymptotically

less than or equal to g(n)

big-Omega
 f(n) is Ω(g(n)) if f(n) is asymptotically
greater than or equal to g(n)

big-Theta
 f(n) is Θ(g(n)) if f(n) is asymptotically
equal to g(n)

14
Example Uses of the
Relatives of Big-Oh
 5n2 is Ω(n2)
f(n) is Ω(g(n)) if there is a constant c > 0 and an integer constant n0 ≥ 1
such that f(n) ≥ c g(n) for n ≥ n0
let c = 5 and n0 = 1
 5n2 is Ω(n)
f(n) is Ω(g(n)) if there is a constant c > 0 and an integer constant n0 ≥ 1
such that f(n) ≥ c g(n) for n ≥ n0
let c = 1 and n0 = 1
 5n2 is Θ(n2)
f(n) is Θ(g(n)) if it is Ω(n2) and O(n2). We have already seen the former,
for the latter recall that f(n) is O(g(n)) if there is a constant c > 0 and an
integer constant n0 ≥ 1 such that f(n) < c g(n) for n ≥ n0
Let c = 5 and n0 = 1

15
Example Uses of the Relatives
of Big-Oh
10,000
 2n + 10 is O(n)
3n
 2n + 10 ≤ cn
 10 ≤ cn - 2n 1,000 2n+10

 10/(c − 2) ≤ n n

 10 ≤ n (if c is 3) 100
 let c = 3 and n0 = 10
 2n + 10 is Ω(n) 10
 2n + 10 ≥ c’n
 n ≥ -10 (if c’ is 1)
1
 let c’ = 1 and n0 = 1 1 10 100 1,000
n
 2n + 10 is Θ(n)
 c = 3, c’ = 1
 n0 = 10 (larger of the two n0’s)
16
o-notation and ω -notation

O-notation and Ω-notation are like ≤ and ≥.

o-notation and ω-notation are like < and >.

 Set definition of small-Oh

ο(g(n)) = { f(n) : for any constant c > 0, there
exist constants n0 > 0 such that 0 ≤ f(n) < cg(n)
for all n ≥ n0}
 EXAMPLE:
2n2 = o(n3) (n0 = 2/c)
17
o-notation and ω -notation

O-notation and Ω-notation are like ≤ and ≥.

o-notation and ω-notation are like < and >.

 Set definition of small-Omega

ω(g(n)) = { f(n) : for any constant c > 0, there
exist constants n0 > 0 such that 0 ≤ cg(n) < f(n)
for all n ≥ n0}
 EXAMPLE:
𝑛𝑛 = ω( lg 𝑛𝑛) (n0 = 26c)
18
Solving Recurrences
 The analysis of merge sort required us
to solve a recurrence.
 Recurrences are like solving integrals,
differential equations, etc.
 Learn a few tricks.
 We will see some applications of
recurrence to divide-and-conquer
algorithms

19
Guess-and-Test method
 The most general method:
1. Guess the form of the solution.
2. Verify by induction.
3. Solve for constants.

20
Guess-and-Test method
 The most general method:
1. Guess the form of the solution.
2. Verify by induction.
3. Solve for constants.
 EXAMPLE: T(n) = 4T(n/2) + n
 [Assume that T(1) = Θ(1).]
 Guess O(n3). (Prove O and Ω separately.)
 Assume that T(k) ≤ ck3 for k < n.
 Prove T(n) ≤ cn3 by induction.
21
Example of Guess-and-Test
T (n) = 4T (n / 2) + n
≤ 4c(n / 2)3 + n
= (c / 2)n3 + n
= cn3 − ((c / 2)n3 − n) desired – residual
≤ cn3 desired
whenever (c/2)n3 – n ≥ 0, for example,
if c ≥ 2 and n ≥ 1.
residual

22
Example (continued)
• We must also handle the initial conditions,
that is, ground the induction with base
cases.
• Base: T(n) = Θ(1) for all n < n0, where n0
is a suitable constant.
• For 1 ≤ n < n0, we have “Θ(1)” ≤ cn3, if we
pick c big enough.

23
Example (continued)
• We must also handle the initial conditions,
that is, ground the induction with base
cases.
• Base: T(n) = Θ(1) for all n < n0, where n0
is a suitable constant.
• For 1 ≤ n < n0, we have “Θ(1)” ≤ cn3, if we
pick c big enough.

This bound is not tight!

24
A tighter upper bound?
We shall prove that T(n) = O(n2).

25
A tighter upper bound?
We shall prove that T(n) = O(n2).
Assume that T(k) ≤ ck2 for k < n:
T (n) = 4T (n / 2) + n
≤ 4c(n / 2) 2 + n
= cn 2 + n
= O(n 2 )

26
A tighter upper bound?
We shall prove that T(n) = O(n2).
Assume that T(k) ≤ ck2 for k < n:
T (n) = 4T (n / 2) + n
≤ 4c(n / 2) 2 + n
= cn 2 + n
= O(n 2 ) Wrong! We must prove the I.H.

27
A tighter upper bound?
We shall prove that T(n) = O(n2).
Assume that T(k) ≤ ck2 for k < n:
T (n) = 4T (n / 2) + n
≤ 4c(n / 2) 2 + n
= cn 2 + n
= O(n 2 ) Wrong! We must prove the I.H.
= cn 2 − (−n) [ desired – residual ]
≤ cn 2 for no choice of c > 0. Lose!
28
A tighter upper bound!
IDEA: Strengthen the inductive hypothesis.
• Subtract a low-order term.
Inductive hypothesis: T(k) ≤ c1k2 – c2k for k < n.

29
A tighter upper bound!
IDEA: Strengthen the inductive hypothesis.
• Subtract a low-order term.
Inductive hypothesis: T(k) ≤ c1k2 – c2k for k < n.
T(n) = 4T(n/2) + n
≤ 4{c1(n/2)2 – c2(n/2)} + n
= c1n2 – 2c2n + n
= c1n2 – c2n – (c2n – n)
≤ c1n2 – c2n if c2 ≥ 1.

30
A tighter upper bound!
IDEA: Strengthen the inductive hypothesis.
• Subtract a low-order term.
Inductive hypothesis: T(k) ≤ c1k2 – c2k for k < n.
T(n) = 4T(n/2) + n
≤ 4{c1(n/2)2 – c2(n/2)} + n
= c1n2 – 2c2n + n
= c1n2 – c2n – (c2n – n)
≤ c1n2 – c2n if c2 ≥ 1.
Pick c1 big enough to handle the initial conditions.
31
Example 2
 In the guess-and-test method, we guess a closed form solution
and then try to prove it is true by induction:
 Recurrence equation:  b if n < 2
T (n) = 
2T ( n / 2) + bn log n if n ≥ 2
 Guess:

32
Example 2
 In the guess-and-test method, we guess a closed form solution
and then try to prove it is true by induction:
 Recurrence equation:  b if n < 2
T (n) = 
2T ( n / 2) + bn log n if n ≥ 2
 Guess: T(n) ≤ cn log n.
 Inductive hypothesis: T(k) ≤ ck log k, for k < n.

33
Example 2
 In the guess-and-test method, we guess a closed form solution
and then try to prove it is true by induction:
 Recurrence equation:  b if n < 2
T (n) = 
2T ( n / 2) + bn log n if n ≥ 2
 Guess: T(n) ≤ cn log n.
 Inductive hypothesis: T(k) ≤ ck log k, for k < n.
T (n ) = 2T (n / 2) + bn log n
≤ 2(c (n / 2) log(n / 2)) + bn log n
= cn (log n − log 2) + bn log n
= cn log n − cn + bn log n
 Wrong: we cannot make this last line be less than cn log n

34
Example 2
 In the guess-and-test method, we guess a closed form solution
and then try to prove it is true by induction:
 Recurrence equation:  b if n < 2
T (n) = 
2T ( n / 2) + bn log n if n ≥ 2
 Guess: T(n) ≤ cn log2n.
 Inductive hypothesis: T(k) ≤ ck log2k, for k < n.

35
Example 2
 In the guess-and-test method, we guess a closed form solution
and then try to prove it is true by induction:
 Recurrence equation:  b if n < 2
T (n) = 
2T ( n / 2) + bn log n if n ≥ 2
 Guess: T(n) ≤ cn log2n.
 Inductive hypothesis: T(k) ≤ ck log2k, for k < n.
T (n ) = 2T (n / 2) + bn log n
≤ 2(c (n / 2) log2(n / 2)) + bn log n
= cn (log n − log 2)2 + bn log n
= cn log2 n − 2cn log n + cn + bn log n
≤ cn log2 n if c > b.
 So, T(n) is O(n log2 n).
 In general, to use this method, you need to have a good guess
and you need to be good at induction proofs.
36
Thank You

38