Sorting
Sorting
Statement Effort
MergeSort(A, left, right) { T(n)
if (left < right) { (1)
mid = floor((left + right) / 2); (1)
MergeSort(A, left, mid); T(n/2)
MergeSort(A, mid+1, right); T(n/2)
Merge(A, left, mid, right); (n)
}
}
● So T(n) = (1) when n = 1, and
2T(n/2) + (n) when n > 1
● This expression is a recurrence
The Master Theorem
● Given: a divide and conquer algorithm
■ An algorithm that divides the problem of size n
into a subproblems, each of size n/b
■ Let the cost of each stage (i.e., the work to divide
the problem + combine solved subproblems) be
described by the function f(n)
● Then, the Master Theorem gives us a
cookbook for the algorithm’s running time:
The Master Theorem
● if T(n) = aT(n/b) + f(n) then
n
logb a
f (n) O n logb a
0
T (n) n logb a
log n f (n) n
logb a
c 1
f (n)
f (n) n logb a AND
af (n / b) cf (n) for large n
Using The Master Method
● T(n) = 9T(n/3) + n
■ a=9, b=3, f(n) = n
■ nlog a = nlog
b
= (n2)
3 9
T (n) n logb a when f (n) O n logb a
■ Thus the solution is T(n) = (n2)
Sorting
● So far we’ve talked about two algorithms to
sort an array of numbers
■ What is the advantage of merge sort?
■ What is the advantage of insertion sort?
● Next: Heapsort
■ Combines advantages of both previous algorithms
Heaps
● A heap can be seen as a complete binary tree:
16
14 10
8 7 9 3
2 4 1
14 10
8 7 9 3
2 4 1 1 1 1 1 1
The CLR book calls them “nearly complete” binary trees; can
think of unfilled slots as null pointers
Heaps
● In practice, heaps are usually implemented as
arrays:
16
14 10
8 7 9 3
A = 16 14 10 8 7 9 3 2 4 1 =
2 4 1
Heaps
● To represent a complete binary tree as an array:
■ The root node is A[1]
■ The parent of node i is A[i/2] (note: integer divide)
■ The left child of node i is A[2i]
■ The right child of node i is A[2i + 1]
16
14 10
8 7 9 3
A = 16 14 10 8 7 9 3 2 4 1 =
2 4 1
Referencing Heap Elements
● So…
Parent(i) { return i/2; }
Left(i) { return 2*i; }
right(i) { return 2*i + 1; }
The Heap Property
● Heaps also satisfy the heap property:
A[Parent(i)] A[i], for all nodes i > 1
■ In other words, the value of a node is at most the
value of its parent
■ Where is the largest element in a heap stored?
Heap Height
● What is the height of an n-element heap?
Why?
● This is nice: basic heap operations take at most
time proportional to the height of the heap
Heap Operations: Heapify()
● Heapify(): maintain the heap property
■ Given: a node i in the heap with children l and r
■ Given: two subtrees rooted at l and r, assumed to be
heaps
■ Problem: The subtree rooted at i may violate the heap
property (How?)
■ Action: let the value of the parent node “float down”
so subtree at i satisfies the heap property
○ What do you suppose will be the basic operation between i,
l, and r?
Heap Operations: Heapify()
Heapify(A, i)
{
l = Left(i); r = Right(i);
if (l <= heap_size(A) && A[l] > A[i])
largest = l;
else
largest = i;
if (r <= heap_size(A) && A[r] > A[largest])
largest = r;
if (largest != i)
Swap(A, i, largest);
Heapify(A, largest);
}
Heapify() Example
16
4 10
14 7 9 3
2 8 1
A = 16 4 10 14 7 9 3 2 8 1
Heapify() Example
16
4 10
14 7 9 3
2 8 1
A = 16 4 10 14 7 9 3 2 8 1
Heapify() Example
16
4 10
14 7 9 3
2 8 1
A = 16 4 10 14 7 9 3 2 8 1
Heapify() Example
16
14 10
4 7 9 3
2 8 1
A = 16 14 10 4 7 9 3 2 8 1
Heapify() Example
16
14 10
4 7 9 3
2 8 1
A = 16 14 10 4 7 9 3 2 8 1
Heapify() Example
16
14 10
4 7 9 3
2 8 1
A = 16 14 10 4 7 9 3 2 8 1
Heapify() Example
16
14 10
8 7 9 3
2 4 1
A = 16 14 10 8 7 9 3 2 4 1
Heapify() Example
16
14 10
8 7 9 3
2 4 1
A = 16 14 10 8 7 9 3 2 4 1
Heapify() Example
16
14 10
8 7 9 3
2 4 1
A = 16 14 10 8 7 9 3 2 4 1
Analyzing Heapify()
● Aside from the recursive call, what is the
running time of Heapify()?
● How many times can Heapify() recursively
call itself?
● What is the worst-case running time of
Heapify() on a heap of size n?
Analyzing Heapify()
● Fixing up relationships between i, l, and r
takes (1) time
● If the heap at i has n elements, how many
elements can the subtrees at l or r have?
1 3
2 16 9 10
14 8 7
Analyzing BuildHeap()
● Each call to Heapify() takes O(lg n) time
● There are O(n) such calls (specifically, n/2)
● Thus the running time is O(n lg n)
■ Is this a correct asymptotic upper bound?
■ Is this an asymptotically tight bound?
Analyzing BuildHeap()
● Each call to Heapify() takes O(lg n) time
● There are O(n) such calls (specifically, n/2)
● Thus the running time is O(n lg n)
■ Is this a correct asymptotic upper bound?
■ Is this an asymptotically tight bound?
● A tighter bound is O(n)
Analyzing BuildHeap()
● Each call to Heapify() takes O(lg n) time
● There are O(n) such calls (specifically, n/2)
● Thus the running time is O(n lg n)
■ Is this a correct asymptotic upper bound?
■ Is this an asymptotically tight bound?
● A tighter bound is O(n)
■ How can this be? Is there a flaw in the above
reasoning?
Analyzing BuildHeap(): Tight
● To Heapify() a subtree takes O(h) time
where h is the height of the subtree
■ h = O(lg m), m = # nodes in subtree
■ The height of most subtrees is small
● Fact: an n-element heap has at most n/2h+1
nodes of height h
● We can use this fact to prove that
BuildHeap() takes O(n) time
Heapsort
Length of prefix =
0
1 3 7 8 2 6 4 5
next pivo
element t
Length of prefix =
1
1 3 7 8 2 6 4 5
next pivo
element t
Length of prefix =
2
1 3 7 8 2 6 4 5
next pivo
element t
Length of prefix =
2
1 3 7 8 2 6 4 5
pivo
next
t
element
Again, next element is larger than
pivot, no change to prefix
In-Place Partition in Action
after 4 steps
Length of prefix =
2
1 3 7 8 2 6 4 5
pivo
next
t
element
Because next element is less than
pivot,
we shall extend the prefix by
swapping
In-Place Partition in Action
after 5 steps
Length of prefix =
3
1 3 2 8 7 6 4 5
pivo
next
t
element
Because next element is larger than
pivot, no change to prefix
In-Place Partition in Action
after 6 steps
Length of prefix =
3
1 3 2 8 7 6 4 5
pivo
t
next
Because next element is less thanele
pivot,
we shall extend the prefix by men
t
swapping
In-Place Partition in Action
after 7 steps
Length of prefix =
4
1 3 2 4 7 6 8 5
pivo
t
next
When next element is the pivot, elewe put
it after the end of the prefix bymen
t
swapping
In-Place Partition in Action
after 8 steps
Length of prefix =
4
1 3 2 4 5 6 8 7
pivo
t
1 n 1
T n T k T n 1 k n
n k 0
Worst-Case Running Time
The worst-case running time of Quicksort
can be expressed by:
method:
1. Guess T(n) ≤ cn2 for some constant c
2. Next, verify our guess by induction
Worst-Case Running Time
Inductive Case:
T(n) = maxq=0 to n-1 (T(q) + T(n-q-1)) +
(n)
≤ max ( 2) +
q=0 to n-1 + (n)
cq2 + c(n-q-1) (n)q =
≤ c(n-1) 2 Maximized when
0 or when q = n-
= cn2 - 2cn + c + 1
≤ (n)
cn2 when c is large
enough
Inductive Case is OK now
Worst-Case Running Time
Conclusion:
1.T(n) = (n2)
2.However, we can also show
T(n) = (n2)
by finding a worst-case input
T(n) = (n2)
Average-Case Running Time
So, Quicksort runs badly for some
input…
Average # comparisons
= E[X]
= E[X12 + X13 + … + Xn-1,n]
= E[X12] + E[X13] + … + E[Xn-1,n]
Average # of Comparisons
The next slides will prove: E[Xij] =
2/(j-i+1) Using
this result,
E[X] = i=1 to n-1 j=i+1 to n 2/(j-i+1)
2/k
= i=1
i=1to n-1
ton-1 k=1 to n-i 2/(k+1)
k=1to
=
n i=1 to n-1 (log n) = (n
log n)
Comparison between ai and aj
Question: # times ai be compared with
aj ?
Answer: At most once, which happens
only if ai or a1j are chosen as
3 2 4 5 6 8 7
pivot
pivot
After that, the
pivot is fixed
and is never
compared
Comparison between ai and aj
Question: Will ai always be compared with
aj ? Answer: No. E.g., after Partition in
Page 14:
1 3 2 4 5 6 8 7
pivot
we will
separately
Quicksort the
first 4
Comparison between ai and aj
Observation:
Consider the elements ai, ai+1, …, aj-1,
aj
(i)If ai or aj is first chosen as a pivot,
then ai is compared with aj
(ii) Else, if any element of ai+1, …, aj-1
is first chosen as a pivot,
then ai is never compared with aj
Comparison between ai and aj
When the n! permutations are equally
likely to be the input,
Pr(ai compared with aj once) = 2/(j-i+1)
Pr(ai not compared with aj) = (j-i-1)/(j-
i+1)
E[Xij] = 1 * 2/(j-i+1) + 0 * (j-i-1)/(j-
i+1)
= Consider ai,2/(j-i+1)
ai+1, …, aj-1, aj. Given a permutation, if
ai is chosen a pivot first, then by exchanging ai
with ai+1 initially, ai+1 will be chosen as a pivot first3
5
Proof: Running time = (n+X)
Observe that in the Quicksort algorithm:
•Each Partition fixes the position of pivot
at most n Partition calls
•After each Partition, we have 2 Quicksort
•Also, all Quicksort (except 1st one:
Quicksort(A,1,n))
are invoked after a Partition
total (n) Quicksort calls
Proof: Running time = (n+X)
So, if we ignore the comparison time in
all Partition calls, the time used =
(n)
1
About this lecture
• What is Probability?
• What is an Event?
• What is a Random Variable?
• What is Expectation or “Average
Value” of a Random Variable?
• Useful Thm: Linearity of
Expectation
Experiment and Sample Space
• An experiment is a process that
produces an outcome
• A random experiment is an
experiment whose outcome is not
known until it is observed
– Exp 1: Throw a die once
– Exp 2: Flip a coin until Head comes up
Experiment and Sample Space
• A sample space of a random
experiment is the set of all
outcomes
– Exp 1: Throw a die once
– Sample space: { 1, 2, 3, 4, 5, 6 }
– Exp 2: Flip a coin until Head comes
up
– Sample space: ??
2.Instead, if we know
Pr(1) = 0.2, Pr(2) = 0.3, Pr(3) = 0.4,
Pr(4) = 0.1, Pr(5) = Pr(6) = 0.
What is Pr({1,2,4})?
Random Variable
Definition: A random variable X on a
sample space is a function that
maps each outcome of into a real
number.
That is, X: R.
Ex: Suppose that we throw two dice
= { (1,1), (1,2), …, (6,5), (6,6) }
E[X] = i i Pr(X=i)
Question:
•X = sum of outcomes of two fair
dice What is the value of E[X] ?
•How about the sum of three dice?
Expectation (Example)
Let X = sum of outcomes of two
dice. The value of X can vary from 2
to 12 So, we calculate:
Pr(X=2) = 1/36, Pr(X=3) = 2/36,
Pr(X=4) = 3/36, … , Pr(X=12) =
2/36,
E[X] = E[X1+X2] =
E[X1] + E[X2]