Slides adapted from Stanford CS161

Worst-case analysis Sorting a sorted list

should be fast!!

12345678
• In this class, we will focus on worst-case analysis
Here is my algorithm!
Here is an
Algorithm: input!
Do the thing
Do the stuff
Return the answer

Algorithm
designer

• Pros: very strong guarantee

• Cons: very strong guarantee
Big-O notation
• What do we mean when we measure runtime?
• We probably care about wall time: how long does it take
to solve the problem, in seconds or minutes or hours?

• This is heavily dependent on the programming

language, architecture, etc.
• These things are very important, but are not the
point of this class.
• We want a way to talk about the running time of an
algorithm, independent of these considerations.
Main idea:
Focus on how the runtime scales with n (the input size).
Asymptotic Analysis
How does the running time scale as n gets large?

One algorithm is “faster” than another if its

runtime scales better with the size of the input.

Pros: Cons:
• Abstracts away from • Only makes sense if n is
hardware- and language- large (compared to the
specific issues. constant factors).
• Makes algorithm analysis 2100000000000000 n
much more tractable. is “better” than n2 ?!?!
 pronounced “big-oh of …” or sometimes “oh of …”

O(…) means an upper bound

• Let T(n), g(n) be functions of positive integers.
• Think of T(n) as being a runtime: positive and increasing in n.

• We say “T(n) is O(g(n))” if g(n) grows at least as fast as

T(n) as n gets large.

• Formally,
𝑇𝑇 𝑛𝑛 = 𝑂𝑂 𝑔𝑔 𝑛𝑛
⟺
∃𝑐𝑐, 𝑛𝑛0 > 0 𝑠𝑠. 𝑡𝑡. ∀𝑛𝑛 ≥ 𝑛𝑛0 ,
0 ≤ 𝑇𝑇 𝑛𝑛 ≤ 𝑐𝑐 ⋅ 𝑔𝑔(𝑛𝑛)
 𝑇𝑇 𝑛𝑛 = 𝑂𝑂 𝑔𝑔 𝑛𝑛
Example ⟺
∃𝑐𝑐, 𝑛𝑛0 > 0 𝑠𝑠. 𝑡𝑡. ∀𝑛𝑛 ≥ 𝑛𝑛0 ,
2𝑛𝑛2 + 10 = 𝑂𝑂 𝑛𝑛2 0 ≤ 𝑇𝑇 𝑛𝑛 ≤ 𝑐𝑐 ⋅ 𝑔𝑔(𝑛𝑛)

3n2

2n2 + 10

n2
 𝑇𝑇 𝑛𝑛 = 𝑂𝑂 𝑔𝑔 𝑛𝑛
Example ⟺
∃𝑐𝑐, 𝑛𝑛0 > 0 𝑠𝑠. 𝑡𝑡. ∀𝑛𝑛 ≥ 𝑛𝑛0 ,
2𝑛𝑛2 + 10 = 𝑂𝑂 𝑛𝑛2 0 ≤ 𝑇𝑇 𝑛𝑛 ≤ 𝑐𝑐 ⋅ 𝑔𝑔(𝑛𝑛)

Formally:
3n2 • Choose c = 3
• Choose n0 = 4
• Then:
∀𝑛𝑛 ≥ 4,
0 ≤ 2𝑛𝑛2 + 10 ≤ 3 ⋅ 𝑛𝑛2
n2
 𝑇𝑇 𝑛𝑛 = 𝑂𝑂 𝑔𝑔 𝑛𝑛
same Example ⟺
∃𝑐𝑐, 𝑛𝑛0 > 0 𝑠𝑠. 𝑡𝑡. ∀𝑛𝑛 ≥ 𝑛𝑛0 ,
2𝑛𝑛2 + 10 = 𝑂𝑂 𝑛𝑛2 0 ≤ 𝑇𝑇 𝑛𝑛 ≤ 𝑐𝑐 ⋅ 𝑔𝑔(𝑛𝑛)

Formally:
7n2 • Choose c = 7
• Choose n0 = 2
• Then:
∀𝑛𝑛 ≥ 2,
0 ≤ 2𝑛𝑛2 + 10 ≤ 7 ⋅ 𝑛𝑛2
n2
 𝑇𝑇 𝑛𝑛 = 𝑂𝑂 𝑔𝑔 𝑛𝑛

Another example: ⟺
∃𝑐𝑐, 𝑛𝑛0 > 0 𝑠𝑠. 𝑡𝑡. ∀𝑛𝑛 ≥ 𝑛𝑛0 ,

𝑛𝑛 = 𝑂𝑂(𝑛𝑛 )
2 0 ≤ 𝑇𝑇 𝑛𝑛 ≤ 𝑐𝑐 ⋅ 𝑔𝑔(𝑛𝑛)

g(n) = n2 • Choose c = 1
• Choose n0 = 1
• Then

∀𝑛𝑛 ≥ 1,
T(n) = n 0 ≤ 𝑛𝑛 ≤ 𝑛𝑛2
Ω(…) means a lower bound
• We say “T(n) is Ω(g(n))” if g(n) grows at most as fast
as T(n) as n gets large.

• Formally,
𝑇𝑇 𝑛𝑛 = Ω 𝑔𝑔 𝑛𝑛
⟺
∃𝑐𝑐, 𝑛𝑛0 > 0 𝑠𝑠. 𝑡𝑡. ∀𝑛𝑛 ≥ 𝑛𝑛0 ,
0 ≤ 𝑐𝑐 ⋅ 𝑔𝑔 𝑛𝑛 ≤ 𝑇𝑇 𝑛𝑛
Switched these!!
 𝑇𝑇 𝑛𝑛 = Ω 𝑔𝑔 𝑛𝑛

Example ⟺
∃𝑐𝑐, 𝑛𝑛0 > 0 𝑠𝑠. 𝑡𝑡. ∀𝑛𝑛 ≥ 𝑛𝑛0 ,

𝑛𝑛 log 2 𝑛𝑛 = Ω 3𝑛𝑛 0 ≤ 𝑐𝑐 ⋅ 𝑔𝑔 𝑛𝑛 ≤ 𝑇𝑇 𝑛𝑛

• Choose c = 1/3
• Choose n0 = 3
• Then
∀𝑛𝑛 ≥ 3,
3𝑛𝑛
0≤ ≤ 𝑛𝑛 log 2 𝑛𝑛
3
Θ(…) means both!
• We say “T(n) is Θ(g(n))” if:

T(n) = O(g(n))
-AND-
T(n) = Ω(g(n))
Some more examples
• All degree k polynomials* are O(nk)
• For any k ≥ 1, nk is not O(nk-1)

*Need some caveat here…what is it?

Take-away from examples
• To prove T(n) = O(g(n)), you have to come up with c
and n0 so that the definition is satisfied.

• To prove T(n) is NOT O(g(n)), one way is proof by

contradiction:
• Suppose (to get a contradiction) that someone gives you
a c and an n0 so that the definition is satisfied.
• Show that this someone must by lying to you by deriving
a contradiction.
What have we learned?
This is my
happy face!

Asymptotic Notation
• This makes both Plucky and Lucky happy.
• Plucky the Pedantic Penguin is happy because
there is a precise definition.
• Lucky the Lackadaisical Lemur is happy because we
don’t have to pay close attention to all those pesky
constant factors like “11”.

• But we should always be careful not to abuse it.

• In the course, (almost) every algorithm we see

will be actually practical, without needing to
take 𝑛𝑛 ≥ 𝑛𝑛0 = 210000000 .
Recurrence relations and Master
method
• Recurrence Relations!
• How do we measure the runtime a recursive
algorithm?
• Like Integer Multiplication and MergeSort?

• The Master Method

• A useful theorem so we don’t have to answer this
question from scratch each time.
 Running time of MergeSort
• Let’s call this running time T(n).
• when the input has length n.

• We know that T(n) = O(nlog(n)).

• But if we didn’t know that…

𝑛𝑛 MERGESORT(A):
𝑇𝑇 𝑛𝑛 ≤ 2 ⋅ 𝑇𝑇 + 11 ⋅ 𝑛𝑛 n = length(A)
2 if n ≤ 1:
return A
L = MERGESORT(A[:n/2])
From last time R = MERGESORT(A[n/2:])
return MERGE(L,R)
 Recurrence Relations
𝑛𝑛
• 𝑇𝑇 𝑛𝑛 = 2 ⋅ 𝑇𝑇 + 11 ⋅ 𝑛𝑛 is a recurrence relation.
2
• It gives us a formula for T(n) in terms of T(less than n)

• The challenge:
Given a recurrence relation for T(n), find a
closed form expression for T(n).

• For example, T(n) = O(nlog(n))

Technicalities I
Base Cases Plucky the
Pedantic Penguin

• Formally, we should always have base cases with

recurrence relations.
𝑛𝑛
• 𝑇𝑇 𝑛𝑛 = 2 ⋅ 𝑇𝑇 + 11 ⋅ 𝑛𝑛 with 𝑇𝑇 1 = 1
2
is not the same as
𝑛𝑛
• 𝑇𝑇 𝑛𝑛 = 2 ⋅ 𝑇𝑇 + 11 ⋅ 𝑛𝑛 with 𝑇𝑇 1 = 1000000000
2
• However, T(1) = O(1), so sometimes we’ll just omit it.
Why does T(1) = O(1)?

Siggi the Studious Stork

The master theorem
A useful
• A formula that solves formula it is.
recurrences when all of the Know why it
sub-problems are the same works you should.
size.

Jedi master Yoda

 We can also take n/b to
𝑛𝑛 𝑛𝑛
mean either or and
The master theorem
𝑏𝑏 𝑏𝑏
the theorem is still true.

𝑛𝑛
• Suppose 𝑇𝑇 𝑛𝑛 = 𝑎𝑎 ⋅ 𝑇𝑇 + 𝑂𝑂 𝑛𝑛𝑑𝑑 . Then
𝑏𝑏

Three parameters:
a : number of subproblems Many
b : factor by which input size shrinks symbols
d : need to do nd work to create all the those are….
subproblems and combine their solutions.
Technicalities II
Plucky the
Integer division Pedantic Penguin

• If n is odd, I can’t break it up into two problems of

size n/2.
𝑛𝑛 𝑛𝑛
𝑇𝑇 𝑛𝑛 = 𝑇𝑇 + 𝑇𝑇 + 𝑂𝑂(𝑛𝑛)
2 2
• However one can show that the Master theorem
works fine if you pretend that what you have is:

𝑛𝑛
𝑇𝑇 𝑛𝑛 = 2 ⋅ 𝑇𝑇 + 𝑂𝑂(𝑛𝑛)
2
• From now on we’ll mostly ignore floors and ceilings
in recurrence relations.
 𝑛𝑛
𝑇𝑇 𝑛𝑛 = 𝑎𝑎 ⋅ 𝑇𝑇 + 𝑂𝑂 𝑛𝑛𝑑𝑑 .
Examples 𝑏𝑏

✓
• Needlessly recursive integer mult.
a=4
• T(n) = 4 T(n/2) + O(n) b=2 a > bd

✓
• T(n) = O( n2 ) d=1
• Karatsuba integer multiplication
a=3
• T(n) = 3 T(n/2) + O(n) b=2 a > bd

✓
• T(n) = O( nlog_2(3) ≈ n1.6 ) d=1
• MergeSort
a=2
• T(n) = 2T(n/2) + O(n) b=2 a = bd

✓
• T(n) = O( nlog(n) ) d=1
• That other one
a=1
• T(n) = T(n/2) + O(n)
b=2 a < bd
• T(n) = O(n) d=1
Understanding the Master Theorem
𝑛𝑛
• Suppose 𝑇𝑇 𝑛𝑛 = 𝑎𝑎 ⋅ 𝑇𝑇 + 𝑂𝑂 𝑛𝑛𝑑𝑑 . Then
𝑏𝑏

• What do these three cases mean?

 The eternal struggle

Branching causes the number The problems lower in

of problems to explode! the tree are smaller!
The most work is at the The most work is at
bottom of the tree! the top of the tree!
Consider our three warm-ups
𝑛𝑛
1. 𝑇𝑇 𝑛𝑛 = 𝑇𝑇 2
+ 𝑛𝑛
𝑛𝑛
2. 𝑇𝑇 𝑛𝑛 = 2 ⋅ 𝑇𝑇 2
+ 𝑛𝑛
𝑛𝑛
3. 𝑇𝑇 𝑛𝑛 = 4 ⋅ 𝑇𝑇 2
+ 𝑛𝑛
First example: tall and skinny tree
𝑛𝑛
1. 𝑇𝑇 𝑛𝑛 = 𝑇𝑇 + 𝑛𝑛, 𝑎𝑎 < 𝑏𝑏 𝑑𝑑 Size n
2

• The amount of work done at the n/2

top (the biggest problem) swamps
the amount of work done anywhere n/4
else.

• T(n) = O( work at top ) = O(n)

n/2t
WINNER

Most work at the

1
top of the tree!
 Third example: bushy tree
WINNER
𝑛𝑛
3. 𝑇𝑇 𝑛𝑛 = 4 ⋅ 𝑇𝑇 + 𝑛𝑛, 𝑎𝑎 > 𝑏𝑏 𝑑𝑑
2
Size n
Most work at
the bottom
n/2 n/2 of the tree!
n/2 n/2

• There are a HUGE number of leaves, and the total work is

dominated by the time to do work at these leaves.

• T(n) = O( work at bottom ) = O( 4depth of tree ) = O(n2)

1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1
11 1 1 1 1 1 1 1 1
1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1 1 1 1 1
1
Second example: just right
𝑛𝑛
2. 𝑇𝑇 𝑛𝑛 = 2 ⋅ 𝑇𝑇 + 𝑛𝑛, 𝑎𝑎 = 𝑏𝑏 𝑑𝑑
2 Size n

• The branching just balances n/2

n/2
out the amount of work.
• The same amount of work n/4 n/4 n/4 n/4

is done at every level.

• T(n) = (number of levels) * (work per level)
• = log(n) * O(n) = O(nlog(n))

TIE!
1 1 1 1 1 1 1 1 1 1