SWEN 5012
ADVANCED ALGORITHMS
AND PROBLEM SOLVING
LECTURE 2 SORTING
Beakal Gizachew
Based on slides of Serdar Taşıran, Shafi Goldwasser, Erik Demaine, and Alptekin Küpçü
SORTING
• Sorting
• Input: a sequence of n numbers <a1,a2,…,an>
• Output: a re-ordering <a’1,a’2,…,a’n> of the sequence such that
a’1 a’2 … a’n
• Example:
• Input: 8 2 4 9 3 6
• Output: 2 3 4 6 8 9
• Called an instance of the problem
• Check these demos:
• http://www.iti.fh-flensburg.de/lang/algorithmen/sortieren/sortcontest/sortcontest.htm
• http://www.cs.oswego.edu/~mohammad/classes/csc241/samples/sort/Sort2-E.html
• http://www.cs.ubc.ca/~harrison/Java/sorting-demo.html
• http://www.cs.pitt.edu/~kirk/cs1501/animations/Sort3.html
2
INSERTION SORT
• Takes array A[1..n] containing a sequence of length n to be sorted
• Sorts the array in place
• Numbers re-arranged inside A with at most a constant number of
them stored outside.
• O(1) extra space
3
INSERTION SORT
INSERTION-SORT (A, n) ⊳ A[1 . . n]
for j ← 2 to n
do key ← A[ j ]
i←j–1
“pseudocode”
while i > 0 and A[ i ] > key
do A[i+1] ← A[ i]
i←i–1
A[i+1] = key
1 i j n
A:
key
sorted
4
EXAMPLE OF INSERTION SORT
8 2 4 9 3 6
2 8 4 9 3 6
2 4 8 9 3 6
2 4 8 9 3 6
2 3 4 8 9 6
2 3 4 6 8 9 done
5
CORRECTNESS OF INSERTION
SORT
INSERTION-SORT (A, n) ⊳ A[1 . . n]
for j ← 2 to n
do key ← A[ j ]
i←j–1
while i > 0 and A[ i ] > key
do A[i+1] ← A[ i]
i←i–1
A[i+1] = key
• Loop invariant:
• At the start of each iteration, the subarray A[1..j-1] containsthe
elements originally in A[1..j-1] but in sorted (increasing) order
• Prove initialization, maintenance, termination
• Invariant must imply interesting property about algorithm
6
RUNNING TIME
• The running time of insertion sort depends on the input: e.g., an
already sorted sequence is easier to sort.
• Major Simplifying Convention: Parameterize the running time by
the size of the input, since short sequences are easier to sort than
long ones.
➢ TA(n) = time of A when run with inputs of length n
➢ n: Number of bits required to encode input
➢ Ignore machine-dependent constants
➢ Look at growth of T(n) as n → ∞
➢ For small inputs, the algorithm will run fast anyway. Important
thing is for how big inputs we can still run the algorithm given a
reasonable amount of time.
7
KINDS OF ANALYSES
• Worst-case: (mostly-used)
• T(n) = maximum time of algorithm on any input of size n.
• Generally, we seek upper bounds on the running time, to have a
guarantee of performance.
• Average-case: (sometimes used)
• T(n) = expected time of algorithm over all inputs of size n.
• Need assumption of statistical distribution of real inputs.
• Best-case: (almost never used)
• T(n) = best possible time of algorithm over any input of size n.
• Cheat with a slow algorithm that works fast on some input.
• May be useful when proving negative results
• e.g., Even the best case of algorithm X is slower than the worst
case of algorithm Y.
8
RUNNING-TIME OF INSERTION
SORT
INSERTION-SORT (A, n) ⊳ A[1 . . n] COST TIMES
for j ← 2 to n c1 n
do key ← A[ j ] c2 n-1
i←j–1 c3 n-1
while i > 0 and A[ i ] > key c4 σ 𝑛𝑗=2 𝑡𝑗
do A[i+1] ← A[ i] c5 σ 𝑛𝑗=2 (𝑡𝑗−1)
i←i–1 c6 σ 𝑛𝑗=2 (𝑡𝑗− 1)
A[i+1] = key c7 n-1
Let T(n) = running time of INSERTION-SORT
T(n) = c1n + c2 (n-1) + c3 (n-1) + c4 σ 𝑛𝑗=2 𝑡+
𝑗 c 5 σ 𝑗=2
𝑛
(𝑡 𝑗 − 1) +c6 σ 𝑗=2
𝑛
(𝑡𝑗− 1)+c8 (n-1)
The running time depends on the values of tj . This vary according to the input.
9
INSERTION SORT ANALYSIS
Worst case: Input reverse sorted.
T(n) = σ 𝒏 Θ(𝒋) = Θ(n2) [arithmetic series]
𝒋=𝟐
Average case: All permutations equally likely.
T(n) = σ 𝒏 Θ(𝒋ൗ 𝟐) = Θ(n2)
𝒋=𝟐
Best case: Input already sorted. O(n)
Is insertion sort a fast sorting algorithm?
• Moderately so, for small n.
• Not at all, for large n.
10
HOW TO DESIGN ALGORITHMS?
• We’ll see many example styles throughout the semester
• Insertion sort was an example of an “incremental algorithm”
• Another common paradigm: Divide and Conquer
• Divide into simpler/smaller sub-problems
• Solve (conquer) sub-problems recursively
• Combine results of sub-problems
11
MERGE SORT
MERGE-SORT A[1 . . n]
If n = 1, return
Recursively sort
A[ 1 . . n/2 ] and A[ n/2+1 . . n ]
Merge the two sorted lists
Key subroutine: MERGE
12
MERGE SORT EXAMPLE
13
MERGING TWO SORTED ARRAYS
20 12
13 11
7 9
2 1
14
MERGING TWO SORTED ARRAYS
20 12
13 11
7 9
2 1
15
MERGING TWO SORTED ARRAYS
20 12 20 12
13 11 13 11
7 9 7 9
2 1 2
16
MERGING TWO SORTED ARRAYS
20 12 20 12
13 11 13 11
7 9 7 9
2 1 2
1 2
17
MERGING TWO SORTED ARRAYS
20 12 20 12 20 12
13 11 13 11 13 11
7 9 7 9 7 9
2 1 2
1 2
18
MERGING TWO SORTED ARRAYS
20 12 20 12 20 12
13 11 13 11 13 11
7 9 7 9 7 9
2 1 2
1 2 7
19
MERGING TWO SORTED ARRAYS
20 12 20 12 20 12 20 12
13 11 13 11 13 11 13 11
7 9 7 9 7 9 9
2 1 2
1 2 7
20
MERGING TWO SORTED ARRAYS
20 12 20 12 20 12 20 12
13 11 13 11 13 11 13 11
7 9 7 9 7 9 9
2 1 2
1 2 7 9
21
MERGING TWO SORTED ARRAYS
20 12 20 12 20 12 20 12 20 12
13 11 13 11 13 11 13 11 13 11
7 9 7 9 7 9 9
2 1 2
1 2 7 9
22
MERGING TWO SORTED ARRAYS
20 12 20 12 20 12 20 12 20 12
13 11 13 11 13 11 13 11 13 11
7 9 7 9 7 9 9
2 1 2
1 2 7 9 11
23
MERGING TWO SORTED ARRAYS
20 12 20 12 20 12 20 12 20 12 20 12
13 11 13 11 13 11 13 11 13 11 13
7 9 7 9 7 9 9
2 1 2
1 2 7 9 11
24
MERGING TWO SORTED ARRAYS
20 12 20 12 20 12 20 12 20 12 20 12
13 11 13 11 13 11 13 11 13 11 13
7 9 7 9 7 9 9
2 1 2
1 2 7 9 11 12
25
MERGING TWO SORTED ARRAYS
20 12 20 12 20 12 20 12 20 12 20 12
13 11 13 11 13 11 13 11 13 11 13
7 9 7 9 7 9 9
2 1 2
1 2 7 9 11 12
Time = O(n) to merge a total of n
elements (linear time).
26
ANALYZING MERGE SORT
T(n) MERGE-SORT A[1 . . n]
O(1) If n = 1, return
2T(n/2) Recursively sort
A[ 1 . . n/2 ] and A[ n/2+1 . . n ]
O(n) Merge the two sorted lists
Sloppiness: Should be T( n/2 ) + T( n/2 ) , but it
turns out this does not matter asymptotically.
27
RECURRENCE FOR MERGE SORT
O(1) if n = 1
T(n) =
2T(n/2) + O(n) if n > 1
Note: Usually the base case T(1) = O(1)
28
RECURSION TREE
Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.
29
RECURSION TREE
Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.
T(n)
30
RECURSION TREE
Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.
cn
T(n/2) T(n/2)
31
RECURSION TREE
Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.
cn
cn/2 cn/2
T(n/4) T(n/4) T(n/4) T(n/4)
32
RECURSION TREE
Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.
cn cn
cn/2 cn/2 cn
h = log n cn/4
cn/4 cn/4 cn/4 cn
…
O(1) #leaves = n O(n)
Total = O(n log n)
33
CONCLUSIONS
• O(n log n) grows more slowly than O(n2).
• Therefore, merge sort asymptotically beats insertion sort in the
worst case.
• In practice, merge sort beats insertion sort for n > 30 or so.
34
MASTER THEOREM
• Let T(n) = a T(n/b) + f(n) where
• a 1, b > 1 are constants
• f(n) is an asymptotically positive function
• Then, T(n) can be bounded asymptotically as follows
• If f(n)=O(𝒏𝐥𝐨𝐠𝒃 𝒂−𝜺) for some > 0, then T(n)=Θ(𝒏𝐥𝐨𝐠𝒃 𝒂 )
• If f(n)=Θ(𝒏𝐥𝐨𝐠𝒃 𝒂 ) then T(n)=Θ(𝒏𝐥𝐨𝐠𝒃 𝒂 log n)
• If f(n)=Ω(𝒏𝐥𝐨𝐠𝒃 𝒂+ɛ) for some > 0, and if, for all sufficiently large n
and a constant c < 1 we have af(n/b) cf(n) , then T(n)=Θ(f(n))
35
