Priority Queues (Heaps)
Cpt S 223. School of EECS, WSU
Motivation
Queues are a standard mechanism for ordering tasks on a first-come, first-served basis However, some tasks may be more important or timely than others (higher priority) Priority queues
Store tasks using a partial ordering based on priority Ensure highest priority task at head of queue
Heaps are the underlying data structure of priority queues
Cpt S 223. School of EECS, WSU
Priority Queues: Specification
Main operations
insert (i.e., enqueue)
Dynamic insert specification of a priority level (0-high, 1,2.. Low)
deleteMin (i.e., dequeue)
Finds the current minimum element (read: highest priority) in the queue, deletes it from the queue, and returns it
Performance goal is for operations to be fast
Cpt S 223. School of EECS, WSU
Using priority queues
4 5
3 13 11 22
10
19
insert()
deleteMin()
2
Dequeues the next element with the highest priority
Cpt S 223. School of EECS, WSU 4
Can we build a data structure better suited to store and retrieve priorities?
Simple Implementations
Unordered linked list
10
O(1) insert O(n) deleteMin
2 3 5
Ordered linked list
10
O(n) insert O(1) deleteMin
2 3 5 10
Ordered array
O(lg n + n) insert O(n) deleteMin O(log2n) insert and deleteMin
Cpt S 223. School of EECS, WSU 5
Balanced BST
Binary Heap
A priority queue data structure
Cpt S 223. School of EECS, WSU
Binary Heap
A binary heap is a binary tree with two properties
Structure property Heap-order property
Cpt S 223. School of EECS, WSU
Structure Property
A binary heap is a complete binary tree
Each level (except possibly the bottom most level) is completely filled The bottom most level may be partially filled (from left to right)
Height of a complete binary tree with N elements is log 2 N
Cpt S 223. School of EECS, WSU 8
Structure property
Binary Heap Example
N=10 Every level (except last) saturated
Array representation:
Cpt S 223. School of EECS, WSU
Heap-order Property
Heap-order property (for a MinHeap)
For every node X, key(parent(X)) key(X) Except root node, which has no parent Alternatively, for a MaxHeap, always keep the maximum key at the root
Thus, minimum key always at root
Insert and deleteMin must maintain heap-order property
Cpt S 223. School of EECS, WSU 10
Heap Order Property
Minimum element
Duplicates are allowed No order implied for elements which do not share ancestor-descendant relationship
Cpt S 223. School of EECS, WSU 11
Implementing Complete Binary Trees as Arrays
Given element at position i in the array
Left child(i) = at position 2i Right child(i) = at position 2i + 1 Parent(i) = at position i / 2
i 2i 2i + 1
i/2
Cpt S 223. School of EECS, WSU 12
Just finds the Min without deleting it insert deleteMin Note: a general delete() function is not as important for heaps but could be implemented
Stores the heap as a vector
Fix heap after deleteMin
Cpt S 223. School of EECS, WSU 13
Heap Insert
Insert new element into the heap at the next available slot (hole)
According to maintaining a complete binary tree
Then, percolate the element up the heap while heap-order property not satisfied
Cpt S 223. School of EECS, WSU 14
Percolating Up
Heap Insert: Example
Insert 14:
14 hole
Cpt S 223. School of EECS, WSU
15
Percolating Up
Heap Insert: Example
Insert 14: (1) 14 vs. 31
14 14 hole
Cpt S 223. School of EECS, WSU
16
Percolating Up
Heap Insert: Example
Insert 14: (1) 14 vs. 31
14 14 hole
(2) 14 vs. 21
14
Cpt S 223. School of EECS, WSU
17
Percolating Up
Heap Insert: Example
Insert 14: (1) 14 vs. 31
14 hole
(2) 14 vs. 21
14
(3) 14 vs. 13
Heap order prop Structure prop
Path of percolation up
Cpt S 223. School of EECS, WSU
18
Heap Insert: Implementation
// assume array implementation void insert( const Comparable &x) { ? }
Cpt S 223. School of EECS, WSU
19
Heap Insert: Implementation
O(log N) time
Cpt S 223. School of EECS, WSU
20
Heap DeleteMin
Minimum element is always at the root Heap decreases by one in size Move last element into hole at root Percolate down while heap-order property not satisfied
Cpt S 223. School of EECS, WSU
21
Percolating down
Heap DeleteMin: Example
Make this position empty
Cpt S 223. School of EECS, WSU
22
Percolating down
Heap DeleteMin: Example
Copy 31 temporarily here and move it down
Make this position empty
Is 31 > min(14,16)? Yes - swap 31 with min(14,16)
Cpt S 223. School of EECS, WSU
23
Percolating down
Heap DeleteMin: Example
31
Is 31 > min(19,21)? Yes - swap 31 with min(19,21)
Cpt S 223. School of EECS, WSU
24
Percolating down
Heap DeleteMin: Example
31
31
Is 31 > min(19,21)? Yes - swap 31 with min(19,21)
Is 31 > min(65,26)? Yes - swap 31 with min(65,26)
Percolating down
Cpt S 223. School of EECS, WSU
25
Percolating down
Heap DeleteMin: Example
31
Percolating down
Cpt S 223. School of EECS, WSU
26
Percolating down
Heap DeleteMin: Example
31
Heap order prop Structure prop
Cpt S 223. School of EECS, WSU 27
Other Heap Operations
decreaseKey(p,v)
increaseKey(p,v)
Lowers the current value of item p to new priority value v Need to percolate up E.g., promote a job
Increases the current value of item p to new priority value v Need to percolate down E.g., demote a job
Run-times for all three functions?
remove(p)
First, decreaseKey(p,-) Then, deleteMin E.g., abort/cancel a job
Cpt S 223. School of EECS, WSU
O(lg n)
30
Improving Heap Insert Time
What if all N elements are all available upfront? To build a heap with N elements:
Default method takes O(N lg N) time We will now see a new method called buildHeap() that will take O(N) time - i.e., optimal
Cpt S 223. School of EECS, WSU
31
Building a Heap
Construct heap from initial set of N items Solution 1
Perform N inserts O(N log2 N) worst-case Randomly populate initial heap with structure property Perform a percolate-down from each internal node (H[size/2] to H[1])
Solution 2 (use buildHeap())
To take care of heap order property
Cpt S 223. School of EECS, WSU 32
BuildHeap Example
Input: { 150, 80, 40, 10, 70, 110, 30, 120, 140, 60, 50, 130, 100, 20, 90 }
Leaves are all valid heaps (implicitly) Arbitrarily assign elements to heap nodes Structure property satisfied Heap order property violated Leaves are all valid heaps (implicit) So, let us look at each internal node, from bottom to top, and fix if necessary
33
Cpt S 223. School of EECS, WSU
BuildHeap Example
Nothing to do Swap with left child
Randomly initialized heap Structure property satisfied Heap order property violated Cpt (implicit) S 223. School of EECS, WSU Leaves are all valid heaps
34
BuildHeap Example
Nothing to do
Swap with right child
Dotted lines show path of percolating down
Cpt S 223. School of EECS, WSU 35
BuildHeap Example
Nothing to do
Swap with right child & then with 60
Dotted lines show path of percolating down
Cpt S 223. School of EECS, WSU 36
BuildHeap Example
Swap path
Final Heap Dotted lines show path of percolating down
Cpt S 223. School of EECS, WSU 37
BuildHeap Implementation
Start with lowest, rightmost internal node
Cpt S 223. School of EECS, WSU 38
BuildHeap() : Run-time Analysis
Run-time = ?
O(sum of the heights of all the internal nodes) because we may have to percolate all the way down to fix every internal node in the worst-case
HOW?
Theorem 6.1
For a perfect binary tree of height h, the sum of heights of all nodes is 2h+1 1 (h + 1)
Since h=lg N, then sum of heights is O(N) Will be slightly better in practice
Implication: Each insertion costs O(1) amortized time
Cpt S 223. School of EECS, WSU 39
Cpt S 223. School of EECS, WSU
40
Binary Heap Operations Worst-case Analysis
Height of heap is log 2 N insert: O(lg N) for each insert
In practice, expect less
buildHeap insert: O(N) for N inserts deleteMin: O(lg N) decreaseKey: O(lg N) increaseKey: O(lg N) remove: O(lg N)
Cpt S 223. School of EECS, WSU 41
Applications
Operating system scheduling
Process jobs by priority
Graph algorithms
Find shortest path
Instead of checking for events at each time click, look up next event to happen
Cpt S 223. School of EECS, WSU 42
Event simulation
An Application: The Selection Problem
Given a list of n elements, find the kth smallest element Algorithm 1:
Sort the list => O(n log n) Pick the kth element => O(1) Use a binary heap (minheap)
Cpt S 223. School of EECS, WSU 43
A better algorithm:
Selection using a MinHeap
1. 2. 3.
Input: n elements Algorithm:
buildHeap(n) Perform k deleteMin() operations Report the kth deleteMin output ==> O(n) ==> O(k log n) ==> O(1)
Total run-time = O(n + k log n) If k = O(n/log n) then the run-time becomes O(n)
Cpt S 223. School of EECS, WSU
44
Other Types of Heaps
Binomial Heaps d-Heaps
Generalization of binary heaps (ie., 2-Heaps)
Leftist Heaps
Supports merging of two heaps in o(m+n) time (ie., sublinear)
Skew Heaps
O(log n) amortized run-time
Fibonacci Heaps
Cpt S 223. School of EECS, WSU 45
Run-time Per Operation
Insert
Binary heap O(log n) worst-case O(1) amortized for buildHeap
DeleteMin
O(log n)
Merge (=H1+H2)
O(n)
Leftist Heap
Skew Heap Binomial Heap
O(log n)
O(log n) O(log n) worst case O(1) amortized for sequence of n inserts
O(log n)
O(log n) O(log n)
O(log n)
O(log n) O(log n)
Fibonacci Heap
O(1)
O(log n)
Cpt S 223. School of EECS, WSU
O(1)
46
Priority Queues in STL
Uses Binary heap Default is MaxHeap Methods
Push, top, pop, empty, clear
#include <priority_queue> int main () { priority_queue<int> Q; [Link] (10); cout << [Link] (); [Link] (); } Calls DeleteMax()
For MinHeap: declare priority_queue as: priority_queue<int, vector<int>, greater<int>> Q; Refer to Book Chapter 6, Fig 6.57 for an example
Cpt S 223. School of EECS, WSU 47
Binomial Heaps
Cpt S 223. School of EECS, WSU
48
Binomial Heap
A binomial heap is a forest of heap-ordered binomial trees, satisfying:
i) Structure property, and ii) Heap order property
A binomial heap is different from binary heap in that:
Its structure property is totally different Its heap-order property (within each binomial tree) is the same as in a binary heap
Cpt S 223. School of EECS, WSU
49
Note: A binomial tree need not be a binary tree
Definition: A Binomial Tree Bk
A binomial tree of height k is called Bk:
It has 2k nodes k The number of nodes at depth d = ( d )
k (d ) is the form of the co-efficients in binomial theorem
Depth:
B3:
#nodes:
3 (0 ) 3 (1 ) 3 (2 ) 3 (3 )
d=0 d=1 d=2 d=3
Cpt S 223. School of EECS, WSU
50
What will a Binomial Heap with n=31 nodes look like?
We know that:
i) A binomial heap should be a forest of binomial trees ii) Each binomial tree has power of 2 elements
So how many binomial trees do we need?
B4 B3 B2 B1 B0
n = 31 = (1 1 1 1 1)2
Cpt S 223. School of EECS, WSU
51
A Binomial Heap w/ n=31 nodes
B4 B3 B2 B1 B0
n = 31 = (1 1 1 1 1)2
Forest of binomial trees {B0, B1, B2, B3, B4 }
Bi == Bi-1 + Bi-1
B0
B1 B3 B2
Cpt S 223. School of EECS, WSU
52
Binomial Heap Property
Lemma: There exists a binomial heap for every positive value of n Proof:
All values of n can be represented in binary representation
Have one binomial tree for each power of two with co-efficient of 1 Eg., n=10 ==> (1010)2 ==> forest contains {B3, B1}
Cpt S 223. School of EECS, WSU
53
Binomial Heaps: Heap-Order
Property
Each binomial tree should contain the minimum element at the root of every subtree
Just like binary heap, except that the tree here is a binomial tree structure (and not a complete binary tree)
The order of elements across binomial trees is irrelevant
Cpt S 223. School of EECS, WSU 54
Definition: Binomial Heaps
A binomial heap of n nodes is:
(Structure Property) A forest of binomial trees as dictated by the binary representation of n
(Heap-Order Property) Each binomial tree is a min-heap or a max-heap
Cpt S 223. School of EECS, WSU
55
Binomial Heaps: Examples
Two different heaps:
Cpt S 223. School of EECS, WSU
56
Key Properties
Could there be multiple trees of the same height in a binomial heap?
no
What is the upper bound on the number of binomial trees in a binomial heap of n nodes? lg n
Given n, can we tell (for sure) if Bk exists?
Bk exists if and only if: the kth least significant bit is 1 in the binary representation of n
Cpt S 223. School of EECS, WSU 57
An Implementation of a Binomial Heap
Maintain a linked list of tree pointers (for the forest) Example: n=13 == (1101)2
B7
B6
B5
B4
B3
B2
B1
B0
Shown using the left-child, right-sibling pointer method
Analogous to a bit-based representation of a binary number n
Cpt S 223. School of EECS, WSU 58
Binomial Heap: Operations
x <= DeleteMin() Insert(x) Merge(H1, H2)
Cpt S 223. School of EECS, WSU
59
DeleteMin()
Goal: Given a binomial heap, H, find the minimum and delete it Observation: The root of each binomial tree in H contains its minimum element Approach: Therefore, return the minimum of all the roots (minimums) Complexity: O(log n) comparisons (because there are only O(log n) trees)
Cpt S 223. School of EECS, WSU 60
FindMin() & DeleteMin() Example
B0 B2 B3
B0 B1 B2
For DeleteMin(): After delete, how to adjust the heap? New Heap : Merge { B0, B2 } & { B0, B1, B2 }
Cpt S 223. School of EECS, WSU 61
Insert(x) in Binomial Heap
Goal: To insert a new element x into a binomial heap H Observation:
Element x can be viewed as a single element binomial heap
=> Insert (H,x) == Merge(H, {x})
So, if we decide how to do merge we will automatically figure out how to implement both insert() and deleteMin()
Cpt S 223. School of EECS, WSU 62
Merge(H1,H2)
Let n1 be the number of nodes in H1 Let n2 be the number of nodes in H2 Therefore, the new heap is going to have n1 + n2 nodes
Assume n = n1 + n2
Logic:
Merge trees of same height, starting from lowest height trees If only one tree of a given height, then just copy that Otherwise, need to do carryover (just like adding two binary numbers)
Cpt S 223. School of EECS, WSU 63
Idea: merge tree of same heights
Merge: Example
B0
13
B1
B2
Cpt S 223. School of EECS, WSU
64
How to Merge Two Binomial Trees of the Same Height?
B2: B2: B3:
Simply compare the roots
Note: Merge is defined for only binomial trees of with the same height Cpt S 223. School EECS, WSU
65
Merge(H1,H2) example
carryover
13
14 26 16 26
Cpt S 223. School of EECS, WSU
66
How to Merge more than two binomial trees of the same height?
Merging more than 2 binomial trees of the same height could generate carryovers
+
14
+
26
16 26
Merge any two and leave the third as carry-over
Cpt S 223. School of EECS, WSU 67
Input: +
Merge(H1,H2) : Example
Output:
There are two other possible answers Merge cost log(max{n1,n2}) = O(log n) comparisons
Cpt S 223. School of EECS, WSU 68
Run-time Complexities
Merge takes O(log n) comparisons Corollary:
Insert and DeleteMin also take O(log n)
It can be further proved that an uninterrupted sequence of m Insert operations takes only O(m) time per operation, implying O(1) amortize time per insert
Proof Hint:
unaffected affected
10010111 1 -------------10011000
For each insertion, if i is the least significant bit position with a 0, then number of comparisons required to do the next insert is i+1 If you count the #bit flips for each insert, going from insert of the first element to the insert of the last (nth) element, then => amortized run-time of O(1) per insert
Cpt S 223. School of EECS, WSU 69
Binomial Queue Run-time Summary
Insert
O(lg n) worst-case O(1) amortized time if insertion is done in an uninterrupted sequence (i.e., without being intervened by deleteMins) O(lg n) worst-case
DeleteMin, FindMin
Merge
O(lg n) worst-case
Cpt S 223. School of EECS, WSU
70
Run-time Per Operation
Insert
Binary heap O(log n) worst-case O(1) amortized for buildHeap
DeleteMin
O(log n)
Merge (=H1+H2)
O(n)
Leftist Heap
Skew Heap Binomial Heap
O(log n)
O(log n) O(log n) worst case O(1) amortized for sequence of n inserts
O(log n)
O(log n) O(log n)
O(log n)
O(log n) O(log n)
Fibonacci Heap
O(1)
O(log n)
Cpt S 223. School of EECS, WSU
O(1)
71
Summary
Priority queues maintain the minimum or maximum element of a set Support O(log N) operations worst-case
insert, deleteMin, merge
Many applications in support of other algorithms
Cpt S 223. School of EECS, WSU
72
