Sorting Algorithms (Ch. 6 - 8)! Sorting Algorithms!

Slightly modified definition of the sorting problem:! •! A sorting algorithm is comparison-based if the only operation
we can perform on keys is to compare two keys.!
input: A collection of n data items <a1,a2,...,an> where ! •! A sorting algorithm is in place if only a constant number of
data item ai has a key, ki, drawn from a totally elements of the input array are ever stored outside the array.!
ordered set (e.g., ints, chars)!

output: A permutation (reordering) <a'1,a'2,...,a'n> of the !

Running Time of Comparison-Based Sorting Algorithms!
input sequence such that k1 ! k2 ! ...! kn!
worst-case! average-case! best-case in place!

•! In practice, one usually sorts ‘objects’ according to their key

Insertion Sort n 2! n2 n! yes!
(the non-key data is called satellite data.)! Merge Sort" nlgn nlgn nlgn " no!
Heap Sort nlgn nlgn nlgn yes!
•! If the records are large, we may sort an array of pointers based Quick Sort n2 nlgn nlgn yes!
on some key associated with each record.!

Heapsort (Ch. 6)! Heapsort (Ch. 6)!

Heap concept!
binary tree: an array implementation!
•! complete binary tree, except may be missing •! root is A[1]!
some rightmost leaves on the bottom level.! •! for element A[i]!
•! each node contains a key! " - left child is in position A[2i]!
" - right child is in position A[2i + 1]!
•! values in the nodes satisfy the heap property.! " - parent is in A["i/2#]!
Informal Max-Heap Property:! heap as an array implementation!
The value stored in each child 11!
•! store heap as a logical binary tree in array!
node is less than or equal to 9! 10! •! heapsize is number of elements in heap!
the value stored in the parent 4! •! length is number of elements in array!
8! 6! 2!
node. !
5! 7! 3! 1!

Max-Heap! Min-Heap!
In the array representation of a max-heap, the root of the tree"
Min-heaps are commonly used for priority queues in event-driven !
is in A[1], and given the index i of a node, "
simulators.!
Parent(i) " " LeftChild(i)" " RightChild(i)" Parent(i) LeftChild(i) RightChild(i)
return ("i/2#) return 2i " return (2i + 1)" return ("i/2#) return (2i) return (2i + 1)
Max-heap property: A[Parent(i)] " A[i]! Min-heap property: A[Parent(i)] ! A[i]!
1 2 3 4 5 6 7 8 9 10 11 index" 1 2 3 4 5 6 7 8 9 10 11 index!
A! 20 18 15 11 10 9 6 2 4 5 3" keys" A! 2 3 5 4 6 9 11 10 15 20 18! keys!
1" 1!
n = 11! 20" 2!
height = 3! 2" 3" 2! 3!
18" 15" 3! 5!
4" 5" 6" 7" 4! 5! 6! 7!
11" 10" 9" 6" 4! 6! 9! 11!
8" 9" 10" 11"
2" 4" 5" 3" 8! 10! 9! 15! 20!10! 18! 11!
 Heap Sort! Heap Sort!
Input: An n-element array A[1…n] (unsorted).! Input: An n-element array A (unsorted).!
Output: An n-element array A in sorted order, smallest to largest.! Output: An n-element array A in sorted order, smallest to largest.!
HeapSort(A)! HeapSort(A)!
1. Build-Max-Heap(A) /* put all elements in heap */! 1. Build-Max-Heap(A) /* put all elements in heap */!
2. for i $ length(A) downto 2! 2. for i $ length(A) downto 2!
3. ! do swap A[1] % A[i] /* puts max in ith array position */! 3. ! do swap A[1] % A[i] /* puts max in ith array position */!
4. ! heap-size[A] $ heap-size[A] - 1! 4. ! heap-size[A] $ heap-size[A] - 1!
5. ! Max-Heapify(A,1) /* restore heap property */! 5. ! Max-Heapify(A,1) /* restore heap property */!
Running time:! We'll see that ! Running time of HeapSort!
Line 1:! Need to know running time of Build-Max-Heap! •! 1 call to Build-Max-Heap()!
•! Build-Max-Heap(A) takes !
Line 2: c2(length(A)) = c2n! ! & O(n) time!
Line 3:! c3(n - 1)! O(|A|) = O(n) time! •! n-1 calls to Max-Heapify()!
Line 4: !c4(n - 1)! •! Max-Heapify(A,1) takes ! each takes O(lgn) time!
Line 5:! Need to know running time of Max-Heapify! ! & O(nlgn) time!
O(lg|A|) = O(lgn) time!

Max-Heapify: Maintaining the Heap Max-Heapify: Maintaining the Heap Property!

Property!
Max-Heapify(A, i) Max-Heapify(A,i)
0. left $ 2i
•! Assumption: subtrees rooted at left and right children of A[i] 1.# right $ 2i + 1
are max-heaps and the roots of these subtrees are A[2i] and !indices of left & right children of A[i]
A[2i + 1]. 2. largest $ i
3. if left ! heapsize(A) and A[left] > A[i]
...but subtree rooted at A[i] might not be a max-heap (that is, 4. then largest $ left
A[i] may be smaller than its left or right child) 5. if right ! heapsize(A) and A[right] > A[largest]
6. then largest $ right
•! Max-Heapify(A, i) will cause the value at A[i] to "float down" in 7. if largest ' i
the heap so that subtree rooted at A[i] becomes a max-heap. 8.# then exchange A[i] % A[largest])
9.# Max-Heapify(A, largest)

Max-Heapify: Running Time! Build-Max-Heap!

Intuition: uses Max-Heapify in a bottom-up manner to
Running Time of Max-Heapify convert unordered array A into a heap
•! every line is ((1) time -- except the recursive call •! Key point is that the leaves are already heaps. Elements
A[("n/2# + 1) ... n] are all leaves.
•! In the worst-case of the recursion, the bottom row in the •! So the work starts at parents of leaves...then, grandparents
heap is half empty, so the sub-tree rooted at left child has of leaves...etc.
size at most (2/3)n

So we get the recurrence T(n) ! T(2n/3) + ((1) Build-Max-Heap(A)!

which, by case 2 of the master theorem, has the solution 1. heapsize(A) $ length(A) !
T(n) = O(lgn) 2. for i $ "length(A)/2# downto 1!
(or, Max-Heapify takes O(h) time when node A[i] has height h
3. ! do Max-Heapify(A, i)!
in the heap)
 Build-Max-Heap! Build-Max-Heap - Tighter bound!
Build-Max-Heap(A)! Build-Max-Heap(A)!
1. heapsize(A) $ length(A) ! 1. heapsize(A) $ length(A) !
2. for i $ "length(A)/2# downto 1! 2. for i $ "length(A)/2# downto 1!
3. ! do Max-Heapify(A, i)! 3. ! do Max-Heapify(A, i)!
Proof of tighter bound (O(n)) relies on following theorem:
Running Time of Build-Max-Heap
•! Appoximately n/2 calls to Max-Heapify (O(n) calls) Theorem 1: The number of nodes at height h in a max-
heap ! )n/2h+1*
•! Simple upper bound: Each call takes O(lgn) time
Height of a node = longest distance from a leaf.
& O(nlgn) time total. Depth of a node = distance from the root.

•! The book uses a more complex analysis to show that Build- Let H be the height of the tree. If the heap is not a complete binary tree
Max-Heap runs in O(n) time. Since the running time of (because the bottom level is not full), then the nodes at a given depth
HeapSort is dominated by n calls to Max-Heapify, we will don’t all have the same height. Eg., although all the nodes with depth H
have height 0, the nodes with depth H-1 may have either height 0 or 1.
skip that analysis.

T!
Lemma 1: The number of internal nodes in a proper
binary tree is equal to the number of leaves in the
tree - 1.
Defn: In a proper binary tree (pbt), each node has exactly 0 or 2 children.
T1! T2!
Let I be the number of internal nodes and let L be the number of leaves in
a proper binary tree. The proof is by induction on the height of the tree.
#Internal nodes in T = #Internal nodes in T1 + #Internal nodes in T2 + 1!
Basis: h=0. I = 0 and L = 1. I = L - 1 = 1 - 1 = 0, so the lemma holds. " " = (#Leaves in T1 - 1) + (#Leaves in T2 -1) + 1 (IHOP)!
" " = (#Leaves in T1 + #Leaves in T2) - 2 + 1!
Inductive Step: Assume true for proper binary trees of height h (IHOP) " " =" #Leaves in T - 1 (by observation that # of!
and show for proper binary trees of height h + 1. " " " " " leaves in T is equal to # !
" " " " " leaves in its subtrees.)!
Consider a proper binary tree T of height h+1. It has left and right
subtrees (L and R) of height at most h.
IT = (IL + IR) + 1 = (LL - 1) + (LR - 1) + 1 (by the IHOP) =
(LL + LR -2) + 1 = LL + LR -1. Since LT=LL + LR we have that IT = LT - 1.

Theorem 1: The number of nodes at level h in a max- Theorem 1: The number of nodes at level h in a max-
heap ! )n/2h+1* heap ! )n/2h+1*
Let H be the height of the heap. Proof is by induction on h, the height of If n is odd, x is even, so all nodes have siblings (all internal nodes have 2
each node. The number of nodes in the heap is n. children.) By Lemma 1, the number of internal nodes = the number of
leaves - 1.
Basis: Show the thm holds for nodes with h = 0. The tree leaves (nodes
at height 0) are at depths H and H-1. So n = # of nodes = #of leaves + #internal nodes = 2(#of leaves) - 1.
Thus, the #of leaves = (n+1)/2 = )n/20+1* because n is odd.
Let x be the number of nodes at depth H, that is, the number of leaves
assuming that n is a complete binary tree, i.e. that n = 2h+1-1 Thus, the number of leaves = )n/20+1* and the thm holds for the base case.
Note that n-x is odd, because a complete binary tree has an odd number of
nodes (1 less than a power of 2).
 Theorem 1: The number of nodes at level h in a max-
heap ! )n/2h+1*
Since the time of Max-Heapify when called on a node of height h is O(h),
the time of B-M-H is
Inductive step: Show that if the thm holds for height h-1, it holds for h. lg n lg n
n h
Let nh be the number of nodes at height h in the n-node tree T.
"2 h +1
O(h) = O(n "
2 h
)
h= 0 h= 0

Consider the tree T’ formed by removing the leaves of T. It has n’ = n - n0

and since the last summation turns out to be a constant, the running time
nodes. We know from the base case that n0 = )n/2* , so n’ = n - )n/2* =
is O(n).
"n/2#. !
Therefore, we can build a max-heap from an unordered array in linear
Note that the nodes at height h in T would be at height h-1 if the leaves of
time.
the tree were removed--i.e., they are at height h-1 in T’. Letting n’h-1
denote the number of nodes at height h-1 in T’, we have nh = n’h-1

nh = n’h-1 ! )n’/2h* (by the IHOP) = )"n/2#/2h* ! )(n/2)/2h* = )n/2h+1*.

Correctness of Build-Max-Heap! Heapsort Time and Space Usage!

! Loop invariant: At the start of each iteration i of the for loop,
each node i + 1, i + 2, ..., n is the root of a max-heap.!
•! An array implementation of a heap uses O(n) space!
•! Initialization: i = !n/2". Each node !n/2" + 1, !n/2" + 2, ... n is a -!one array element for each node in heap!
leaf, trivially satisfying the max-heap property.!
•! The children of node i are numbered higher than i. Therefore, by
the inductive hypothesis, the children of i are the roots of max- •! Heapsort uses O(n) space and is in place!
heaps. This is the condition required for inputs to Max-Heapify to
make node i a max-heap root.! •! Running time is as good as merge sort, O(nlgn) in worst
•! At termination, i = 0. By the loop invariant, nodes 1, 2,...,n are case.!
the roots of a max-heap. Therefore, the entire tree is a max-heap.!
Build-Max-Heap(A)!
1. heapsize(A) $ length(A) !
2. for i $ "length(A)/2# downto 1!
3. ! do Max-Heapify(A, i)!

Priority Queues! Inserting Heap Elements!

Inserting an element into a heap:!
Definition: A priority queue is a data structure for maintaining a set •! increment heapsize and add new element to the highest numbered
S of elements, each with an associated key. A max-priority-queue position of array!
gives priority to keys with larger values and supports the following •! walk up tree from new leaf to root, swapping values. Insert input
operations! key when a parent key larger than the input key is found!
! 1. insert(S, x) inserts the element x into set S.! Max-Heap-Insert(A, key)!
! 2. max(S) returns element of S with largest key.! 1. heapsize(A) # heapsize(A) +1 !
! 3. extract-max(S) removes and returns element of S with largest ! 2. i $ heapsize(A) !
! key. !
3. while i > 1 and A[parent(i)] < key!
! 4. increase-key(S,x,k) increases the value of element x's key to new !
! value k (assuming k is at least as large as current key's value).! 4.# do A[i] $ A[parent(i)]!
5. i # parent(i)!
6. A[i] $ key!
Running time of Max-Heap-Insert: O(lgn)!
•! time to traverse leaf to root path (height = O(lgn))!
 Priority Queues: Application for Heaps!
Heap-Increase-Key!
An application of max-priority queues is to schedule jobs on a shared
processor. Need to be able to Heap-Increase-Key(A, i, key) - If key is larger than
check current job's priority Heap-Maximum(A) current key at A[i], floats inserted key up heap until heap
remove job from the queue Heap-Extract-Max(A) property is restored.
insert new jobs into queue Max-Heap-Insert(A, key)
increase priority of jobs Heap-Increase-Key(A,i,key)!
An application for a min-heap priority queue is an event-
driven simulator, where the key is an integer representing the
Initialize PQ by running Build-Max-Heap on an array A. number of seconds (or other discrete time unit) from time
A[1] holds the maximum value after this step. zero (starting point for simulation).
Heap-Maximum(A) - returns value of A[1].
Heap-Extract-Max(A) - Saves A[1] and then, like Heap-
Sort, puts item in A[heapsize] at A[1], decrements
heapsize, and uses Max-Heapify(A, 1) to restore heap
property.