CS3230 – Design and Analysis of Algorithms

(S1 AY2024/25)
Lecture 4a: Lower Bound for Comparison-Based Sorting
Sorting
• Input: an array 𝐴𝐴 = (𝑎𝑎1 , 𝑎𝑎2 , … , 𝑎𝑎𝑛𝑛 ) of elements. Examples:
• Insertion sort
• Goal: Sort the elements in 𝐴𝐴 in non-decreasing order. • Selection sort
• A permutation (𝑎𝑎1′ , 𝑎𝑎2′ , … , 𝑎𝑎𝑛𝑛′ ) of 𝐴𝐴 such that 𝑎𝑎1′ ≤ 𝑎𝑎2′ ≤ ⋯ ≤ 𝑎𝑎𝑛𝑛′ . • Merge sort
• Heap sort
• Quick sort
Sorting
Worst-case time complexity

• Input: an array 𝐴𝐴 = (𝑎𝑎1 , 𝑎𝑎2 , … , 𝑎𝑎𝑛𝑛 ) of elements. Examples:

• Insertion sort 𝑂𝑂 𝑛𝑛2
• Goal: Sort the elements in 𝐴𝐴 in non-decreasing order. • Selection sort 𝑂𝑂 𝑛𝑛2

• A permutation (𝑎𝑎1′ , 𝑎𝑎2′ , … , 𝑎𝑎𝑛𝑛′ ) of 𝐴𝐴 such that 𝑎𝑎1′ ≤ 𝑎𝑎2′ ≤ ⋯ ≤ 𝑎𝑎𝑛𝑛′ . • Merge sort 𝑂𝑂 𝑛𝑛 log 𝑛𝑛
• Heap sort 𝑂𝑂 𝑛𝑛 log 𝑛𝑛
• Quick sort 𝑂𝑂 𝑛𝑛2

• Intuition: 𝑂𝑂 𝑛𝑛 log 𝑛𝑛 should be the best possible bound attainable.

Sorting
Worst-case time complexity

• Input: an array 𝐴𝐴 = (𝑎𝑎1 , 𝑎𝑎2 , … , 𝑎𝑎𝑛𝑛 ) of elements. Examples:

• Insertion sort 𝑂𝑂 𝑛𝑛2
• Goal: Sort the elements in 𝐴𝐴 in non-decreasing order. • Selection sort 𝑂𝑂 𝑛𝑛2

• A permutation (𝑎𝑎1′ , 𝑎𝑎2′ , … , 𝑎𝑎𝑛𝑛′ ) of 𝐴𝐴 such that 𝑎𝑎1′ ≤ 𝑎𝑎2′ ≤ ⋯ ≤ 𝑎𝑎𝑛𝑛′ . • Merge sort 𝑂𝑂 𝑛𝑛 log 𝑛𝑛
• Heap sort 𝑂𝑂 𝑛𝑛 log 𝑛𝑛
• Quick sort 𝑂𝑂 𝑛𝑛2

• Intuition: 𝑂𝑂 𝑛𝑛 log 𝑛𝑛 should be the best possible bound attainable.

• How to turn this intuition into an actual lower bound proof?
• There are too many ways of designing a sorting algorithm.
Sorting
Worst-case time complexity

• Input: an array 𝐴𝐴 = (𝑎𝑎1 , 𝑎𝑎2 , … , 𝑎𝑎𝑛𝑛 ) of elements. Examples:

• Insertion sort 𝑂𝑂 𝑛𝑛2
• Goal: Sort the elements in 𝐴𝐴 in non-decreasing order. • Selection sort 𝑂𝑂 𝑛𝑛2

• A permutation (𝑎𝑎1′ , 𝑎𝑎2′ , … , 𝑎𝑎𝑛𝑛′ ) of 𝐴𝐴 such that 𝑎𝑎1′ ≤ 𝑎𝑎2′ ≤ ⋯ ≤ 𝑎𝑎𝑛𝑛′ . • Merge sort 𝑂𝑂 𝑛𝑛 log 𝑛𝑛
• Heap sort 𝑂𝑂 𝑛𝑛 log 𝑛𝑛
• Quick sort 𝑂𝑂 𝑛𝑛2

• Intuition: 𝑂𝑂 𝑛𝑛 log 𝑛𝑛 should be the best possible bound attainable.

• How to turn this intuition into an actual lower bound proof?
• There are too many ways of designing a sorting algorithm.

We will restrict our attention to a certain class of algorithms.

Comparison-based sorting All of them are comparison-based.

• Comparison-based algorithms: Examples:

• Elements can only be compared with each other: • Insertion sort
• <, ≤, =, >, ≥ • Selection sort
• No other information of the elements can be used. • Merge sort
• Heap sort
• Quick sort
Comparison-based sorting All of them are comparison-based.

• Comparison-based algorithms: Examples:

• Elements can only be compared with each other: • Insertion sort
• <, ≤, =, >, ≥ • Selection sort
• No other information of the elements can be used. • Merge sort
• Heap sort
• Quick sort

Allowed Not allowed

𝐈𝐈𝐈𝐈 (𝐴𝐴 𝑖𝑖 < 𝐴𝐴 𝑗𝑗 ), 𝐭𝐭𝐭𝐭𝐭𝐭𝐭𝐭 Do some work 𝐈𝐈𝐈𝐈 (𝐴𝐴 𝑖𝑖 + 𝐴𝐴 𝑗𝑗 = 𝐴𝐴 𝑘𝑘 ), 𝐭𝐭𝐭𝐭𝐭𝐭𝐭𝐭 Do some work
𝐈𝐈𝐈𝐈 (𝐴𝐴 𝑖𝑖 = 𝐴𝐴 𝑗𝑗 ), 𝐭𝐭𝐭𝐭𝐭𝐭𝐭𝐭 Do some work 𝐈𝐈𝐈𝐈 (𝐴𝐴 𝑖𝑖 = 𝑘𝑘), 𝐭𝐭𝐭𝐭𝐭𝐭𝐭𝐭 Do some work
𝐈𝐈𝐈𝐈 (𝐴𝐴 𝑖𝑖 is odd), 𝐭𝐭𝐭𝐭𝐭𝐭𝐭𝐭 Do some work
𝐈𝐈𝐈𝐈 (the 𝑗𝑗th bit of 𝐴𝐴 𝑖𝑖 is 1), 𝐭𝐭𝐭𝐭𝐭𝐭𝐭𝐭 Do some work
Comparison-based sorting All of them are comparison-based.

• Comparison-based algorithms: Examples:

• Elements can only be compared with each other: • Insertion sort 𝑂𝑂 𝑛𝑛2
• <, ≤, =, >, ≥ • Selection sort 𝑂𝑂 𝑛𝑛2
• No other information of the elements can be used. • Merge sort 𝑂𝑂 𝑛𝑛 log 𝑛𝑛
• Heap sort 𝑂𝑂 𝑛𝑛 log 𝑛𝑛
• Quick sort 𝑂𝑂 𝑛𝑛2

Theorem: The worst-case time complexity of any

comparison-based sorting algorithm is Ω 𝑛𝑛 log 𝑛𝑛 .
Comparison-based sorting All of them are comparison-based.

• Comparison-based algorithms: Examples:

• Elements can only be compared with each other: • Insertion sort 𝑂𝑂 𝑛𝑛2
• <, ≤, =, >, ≥ • Selection sort 𝑂𝑂 𝑛𝑛2
• No other information of the elements can be used. • Merge sort 𝑂𝑂 𝑛𝑛 log 𝑛𝑛
• Heap sort 𝑂𝑂 𝑛𝑛 log 𝑛𝑛
• Quick sort 𝑂𝑂 𝑛𝑛2

Theorem: The worst-case time complexity of any

comparison-based sorting algorithm is Ω 𝑛𝑛 log 𝑛𝑛 .

Merge sort and heap sort are asymptotically optimal!

 Decision trees
• The proof of the theorem uses decision trees.

High pay?
yes no

Long work hours? Interesting job?

yes no yes no

Don’t take the job Take the job Take the job Don’t take the job
Decision trees
• A decision tree is a rooted tree.
• Start from the root.
• At every vertex, a question is asked.
• Depending on the answer, a child is chosen.
• At a leaf, a decision is taken.
Decision trees
• A decision tree is a rooted tree.
• Start from the root.
• At every vertex, a question is asked.
• Depending on the answer, a child is chosen.
• At a leaf, a decision is taken.
• Any comparison-based algorithm can be modeled using a decision tree:
• A comparison ⟷ A question asked at a node.
• Program state depends on the result of the comparison ⟷ Chosen child depends on the
answer to the question.
• Output of the algorithm ⟷ Decision at a leaf.

A permutation (𝑎𝑎1′ , 𝑎𝑎2′ , … , 𝑎𝑎𝑛𝑛′ ) of 𝐴𝐴

 An example
• A comparison-based algorithm for sorting 𝐴𝐴 = 𝑎𝑎1 , 𝑎𝑎2 , 𝑎𝑎3 .

𝑎𝑎1 ≤ 𝑎𝑎2 ?
yes no

𝑎𝑎2 ≤ 𝑎𝑎3 ? 𝑎𝑎1 ≤ 𝑎𝑎3 ?

yes no yes no

𝑎𝑎1 ≤ 𝑎𝑎3 ? 𝑎𝑎2 ≤ 𝑎𝑎3 ?

yes no yes no

𝑎𝑎1 , 𝑎𝑎2 , 𝑎𝑎3 𝑎𝑎1 , 𝑎𝑎3 , 𝑎𝑎2 𝑎𝑎3 , 𝑎𝑎1 , 𝑎𝑎2 𝑎𝑎2 , 𝑎𝑎1 , 𝑎𝑎3 𝑎𝑎2 , 𝑎𝑎3 , 𝑎𝑎1 𝑎𝑎3 , 𝑎𝑎2 , 𝑎𝑎1
 An example
•Worst-case
A comparison-based
running time ≥algorithm for sorting
worst-case number 𝐴𝐴 = 𝑎𝑎1 , 𝑎𝑎
of comparisons , 𝑎𝑎3 . of the tree
=2height

𝑎𝑎1 ≤ 𝑎𝑎2 ?
yes no

𝑎𝑎2 ≤ 𝑎𝑎3 ? 𝑎𝑎1 ≤ 𝑎𝑎3 ?

yes no yes no

𝑎𝑎1 ≤ 𝑎𝑎3 ? 𝑎𝑎2 ≤ 𝑎𝑎3 ?

yes no yes no

𝑎𝑎1 , 𝑎𝑎2 , 𝑎𝑎3 𝑎𝑎1 , 𝑎𝑎3 , 𝑎𝑎2 𝑎𝑎3 , 𝑎𝑎1 , 𝑎𝑎2 𝑎𝑎2 , 𝑎𝑎1 , 𝑎𝑎3 𝑎𝑎2 , 𝑎𝑎3 , 𝑎𝑎1 𝑎𝑎3 , 𝑎𝑎2 , 𝑎𝑎1
 Proof of the theorem
Theorem: The worst-case time complexity of any
comparison-based sorting algorithm is Ω 𝑛𝑛 log 𝑛𝑛 .

Proof:
• Model the algorithm as a decision tree, which is a binary tree with at least 𝑛𝑛! leaves:
• Each permutation is a possible answer.
• The height of the binary tree is at least log 𝑛𝑛! .
 Proof of the theorem
Theorem: The worst-case time complexity of any
comparison-based sorting algorithm is Ω 𝑛𝑛 log 𝑛𝑛 .

Proof:
• Model the algorithm as a decision tree, which is a binary tree with at least 𝑛𝑛! leaves:
• Each permutation is a possible answer.
• The height of the binary tree is at least log 𝑛𝑛! .

log 𝑛𝑛! ∈ 𝑛𝑛 log 𝑛𝑛 − 𝑛𝑛 log 𝑒𝑒 + 𝑂𝑂 log 𝑛𝑛 ⊆ Ω 𝑛𝑛 log 𝑛𝑛

Stirling's approximation
https://en.wikipedia.org/wiki/Stirling%27s_approximation
Question 1 @ VisuAlgo online quiz
• Is the following claim true or false?

There exists a comparison-based sorting algorithm that can

sort any 5-element array using at most 6 comparisons.
Answer
• Is the following claim true or false?

There exists a comparison-based sorting algorithm that can

sort any 5-element array using at most 6 comparisons.

False:
• There are 5! = 120 permutations of 5 elements.
• In any binary tree of height at most 6, the number of leaves is at most 26 = 64.
• 120 > 64, so 6 comparisons are not enough.
Question 2 @ VisuAlgo online quiz
Input: 𝑘𝑘 sorted arrays 𝐴𝐴1 1. . 𝑛𝑛 , 𝐴𝐴2 1. . 𝑛𝑛 , … , 𝐴𝐴𝑘𝑘 1. . 𝑛𝑛 .

Goal: Merge the 𝑘𝑘 sorted arrays into one sorted array of length 𝑘𝑘𝑘𝑘.
Question: What is a tight lower bound of the worst-case running time
for comparison-based algorithms for this task?
• Ω 𝑘𝑘𝑘𝑘
2 8 16 21 7 11 13 19
• Ω 𝑘𝑘𝑘𝑘 log 𝑘𝑘
• Ω 𝑘𝑘𝑘𝑘 log 𝑛𝑛
2 2 7 8 11 13 16 19 21
• Ω 𝑘𝑘 𝑛𝑛
Question 2 @ VisuAlgo online quiz
Input: 𝑘𝑘 sorted arrays 𝐴𝐴1 1. . 𝑛𝑛 , 𝐴𝐴2 1. . 𝑛𝑛 , … , 𝐴𝐴𝑘𝑘 1. . 𝑛𝑛 .

Goal: Merge the 𝑘𝑘 sorted arrays into one sorted array of length 𝑘𝑘𝑛𝑛.
Question: What is a tight lower bound of the worst-case running time
for comparison-based algorithms for this task?
• Ω 𝑘𝑘𝑛𝑛
2 8 16 21 7 11 13 19
• Ω 𝑘𝑘𝑘𝑘 log 𝑘𝑘
• Ω 𝑘𝑘𝑘𝑘 log 𝑛𝑛
2 2 7 8 11 13 16 19 21
• Ω 𝑘𝑘 𝑛𝑛
Answer
• Ω 𝑘𝑘𝑘𝑘 log 𝑘𝑘 is a tight lower bound.

There is an 𝑂𝑂 𝑘𝑘𝑘𝑘 log 𝑘𝑘 -time algorithm.

 Answer
• Ω 𝑘𝑘𝑘𝑘 log 𝑘𝑘 is a tight lower bound.

There is an 𝑂𝑂 𝑘𝑘𝑘𝑘 log 𝑘𝑘 -time algorithm.

Proof of the lower bound:

(𝑘𝑘𝑛𝑛)!
• As the 𝑘𝑘 arrays are sorted, the number of possible ways to combine them is .
𝑛𝑛! 𝑘𝑘
(𝑘𝑘𝑘𝑘)!
• Therefore, any comparison-based algorithm requires at least log comparisons.
𝑛𝑛! 𝑘𝑘

(𝑘𝑘𝑘𝑘)!
log = log 𝑘𝑘𝑛𝑛 ! − 𝑘𝑘 log 𝑛𝑛!
𝑛𝑛! 𝑘𝑘
∈ 𝑘𝑘𝑛𝑛 log 𝑘𝑘𝑘𝑘 − 𝑘𝑘𝑛𝑛 log 𝑒𝑒 + 𝑂𝑂 log 𝑘𝑘𝑘𝑘 − 𝑘𝑘 𝑛𝑛 log 𝑛𝑛 − 𝑛𝑛 log 𝑒𝑒 + 𝑂𝑂 log 𝑛𝑛
= 𝑘𝑘𝑛𝑛 log 𝑘𝑘 ± 𝑂𝑂 log 𝑛𝑛 ⊆ Θ 𝑘𝑘𝑘𝑘 log 𝑘𝑘
 Answer
• Ω 𝑘𝑘𝑘𝑘 log 𝑘𝑘 is a tight lower bound.

There is an 𝑂𝑂 𝑘𝑘𝑘𝑘 log 𝑘𝑘 -time algorithm.

Proof of the lower bound:

(𝑘𝑘𝑛𝑛)!
• As the 𝑘𝑘 arrays are sorted, the number of possible ways to combine them is .
𝑛𝑛! 𝑘𝑘
(𝑘𝑘𝑘𝑘)!
• Therefore, any comparison-based algorithm requires at least log comparisons.
𝑛𝑛! 𝑘𝑘

(𝑘𝑘𝑘𝑘)!
log = log 𝑘𝑘𝑛𝑛 ! − 𝑘𝑘 log 𝑛𝑛!
𝑛𝑛! 𝑘𝑘
∈ 𝑘𝑘𝑛𝑛 log 𝑘𝑘𝑘𝑘 − 𝑘𝑘𝑛𝑛 log 𝑒𝑒 + 𝑂𝑂 log 𝑘𝑘𝑘𝑘 − 𝑘𝑘 𝑛𝑛 log 𝑛𝑛 − 𝑛𝑛 log 𝑒𝑒 + 𝑂𝑂 log 𝑛𝑛
= 𝑘𝑘𝑛𝑛 log 𝑘𝑘 + 𝑂𝑂 log 𝑘𝑘𝑛𝑛 − 𝑂𝑂 log 𝑛𝑛 ⊆ Θ 𝑘𝑘𝑘𝑘 log 𝑘𝑘

log 𝑛𝑛! ∈ 𝑛𝑛 log 𝑛𝑛 − 𝑛𝑛 log 𝑒𝑒 + 𝑂𝑂 log 𝑛𝑛

Non-comparison sorts
Question: Can we bypass the Ω 𝑛𝑛 log 𝑛𝑛 lower bound
by an algorithm that is not comparison-based?
Non-comparison sorts
Question: Can we bypass the Ω 𝑛𝑛 log 𝑛𝑛 lower bound
by an algorithm that is not comparison-based?

Suppose each element in the array 𝐴𝐴 belongs to the range 1,2, … , 𝑘𝑘 .

𝐂𝐂𝐂𝐂𝐂𝐂𝐂𝐂𝐂𝐂𝐂𝐂𝐂𝐂𝐂𝐂𝐂𝐂𝐂𝐂𝐂𝐂𝐂𝐂(𝐴𝐴)
• For all 𝑖𝑖 ∈ 1,2, … , 𝑘𝑘 , compute 𝐜𝐜𝐜𝐜𝐜𝐜𝐜𝐜𝐭𝐭 𝑖𝑖 = the number of appearances of 𝑖𝑖 in 𝐴𝐴.
• Set the initial 𝐜𝐜𝐜𝐜𝐜𝐜𝐜𝐜𝐭𝐭1 entries of 𝐴𝐴 to be 1.
• Set the next 𝐜𝐜𝐜𝐜𝐜𝐜𝐜𝐜𝐭𝐭 2 entries of 𝐴𝐴 to be 2.
• Set the next 𝐜𝐜𝐜𝐜𝐜𝐜𝐜𝐜𝐭𝐭 3 entries of 𝐴𝐴 to be 3.
• …

Exercise: Show that the algorithm can be implemented to finish in 𝑂𝑂 𝑛𝑛 + 𝑘𝑘 time.

Acknowledgement
• The slides are modified from previous editions of this course and
similar course elsewhere.
• List of credits:
• Diptarka Chakraborty
• Yi-Jun Chang
• Erik Demaine
• Steven Halim
• Sanjay Jain
• Wee Sun Lee
• Charles Leiserson
• Hon Wai Leong
• Wing-Kin Sung