Design and Analysis of Algorithm

Tanveer Ahmed Siddiqui

Department of Computer Science

COMSATS University, Islamabad
 Recap and Today Covered

Algorithm Design and Analysis Process

Understand the problem

Decide on : algorithm
design techniques etc.

Design an algorithm

Prove correctness

Analyze efficiency etc

Code the algorithm

Department of Computer Science
 Reading Material

Read Chapter 5
Divide and Conquer

Department of Computer Science

 Objectives

 How to Design Algorithm using divide and

conquer Approach
 Designing solution to sum of an array
 Designing solution to maximum element in an
array
 Designing solution for sorting problem
 Designing solution to Matrix Multiplication
 D.Y.S
 Designing solution to Integer Multiplication
 D.Y.S

Department of Computer Science

 Solving Complex problem

 How to design algorithm of a complex

problem? i.e. How to solve a complex
problem?
 Break down the problem into segments
(smaller chunks/parts) and then target to solve
the segments, ultimately the problem will be
solved.
 How we reduce the size of an instance of a
problem?
 Decreasing
 Decrease and conquer
 Dividing
 Divide and conquer(Today
Department of Computer Science Cover)
A General Divide and Conquer Algorithm

 Last time we discussed decrease-and-conquer

algorithms
 A general divide and conquer have following
three steps

Department of Computer Science

A General Divide and Conquer Algorithm

Department of Computer Science

A General Divide and Conquer Algorithm

Department of Computer Science

A General Divide and Conquer Algorithm
Problem of size n

Subproblem 1 Subproblem 2
of size n/2 of size n/2
Don’t assume
always breaks up
into 2, could be > 2
Solution to subproblems Solution to
subproblem 1 subproblem 2

Solution to the
original probelm

Department of Computer Science

Difference between Divide and Conquer Technique and decrease by constant factor .

10

Department of Computer Science

 Designing Algorithm for
ADDITION OF N NUMBERS
problem using
DIVIDE AND CONQUER

Department of Computer Science

 Example-1: Computing
Brute Force Example a n

 Design an algorithm that computes sum of n

numbers using divide and conquer
 Solution: BF
 What is input
 An array A of n numbers
 What should be the output
 Returns the Sum of A
 Which designing technique?
 Divide and conquer

 Logic/Idea

Department of Computer Science 12

 Example 1: Sum of n numbers

 Logic/Idea/ Divide and conquer technique

a0 + a1 + …… + an-1

Is it more efficient than brute force ?

Let’s see with an example

Department of Computer Science

 Div. & Conq. (add n numbers)
0 1 2 3 4 5 6 7 8 9

2 10 3 5 7 1 6 10 1 3

0 1 2 3 4 0 1 2 3 4
2 10 3 5 7 1 6 10 1 3
# of additions
is same as in
2 10 3 5 7 brute force, 1 6 10 1 3
needs stack
for recursion…
12 3 5 7 7 10 1 3
Bad!
12 not all divide 4
and conquer
15 works!! 14

27 21
Could be efficient
48 for parallel processors though….
Department of Computer Science
Department of Computer Science
 Example-2: Maximum Number
8 3 2 9 7 1 5 4

8 3 2 9 7 1 5 4

8 3 2 9 7 1 5 4

8 3 2 9 7 1 5 4

11 11 8 9

22 17

39 Divide:
Merge:
Department of Computer Science
 Designing Algorithm for
maximum from N NUMBERS
problem using
DIVIDE AND CONQUER

Department of Computer Science

 Example-2:
Brute Maximum Number
Force Example

 Design an algorithm that computes sum of n

numbers using divide and conquer
 Solution: BF
 What is input
 An array A of n numbers
 What should be the output
 Returns the maximum value from A
 Which designing technique?
 Divide and conquer Decrease and Conquer

 Logic/Idea

Department of Computer Science 18

 Example-2: Maximum Number
8 3 2 9 7 1 5 4

8 3 2 9 7 1 5 4

8 3 2 9 7 1 5 4

8 3 2 9 7 1 5 4

8 9 7 5

9 7

9 Divide:
Merge:
Department of Computer Science
 Example-2:
Brute Maximum Number
Force Example

 Design an algorithm that computes maximum

of n numbers using divide and conquer
 Solution: BF
Divide and Conquer

Decrease and Conquer

Department of Computer Science 20

 Example-2:
Brute Maximum Number
Force Example

 Design an algorithm that computes sum of n

numbers using divide and conquer
 Solution:

Department of Computer Science 21

 Example 3: Sorting

 Problem: Arrange the elements of the set such that the

values are in a given order (an increasing or a
decreasing order).
 What is input
 An array A of n numbers e.g. 55 99 6 29 1 7 3
 What should be the output
 List of item arranged in specific order
 Which designing technique?
 Divide and Conquer

 How we divide the array?

 Divides its input elements according to their position in the
array
 Divides input elements according to their value.

Department of Computer Science

 Divide according to their position

Divides its input elements according to their position in the array

Divides input elements according to their value.

Department of Computer Science

 Merge Algorithm

 Design a Brute Force algorithm that take two

sorted list of approximately same size and
merge them into a single sorted list.

Department of Computer Science

 Merge Algorithm

a: 3 5 15 28 30 b: 6 10 14 22 43 50

i=0 j=0

tmp:
k=0
Department of Computer Science 25
 Merge Algorithm

a: 3 5 15 28 30 b: 6 10 14 22 43 50

i=0 j=0

tmp: 3

k=0
Department of Computer Science 26
 Merge Algorithm

a: 3 5 15 28 30 b: 6 10 14 22 43 50

i=1 j=0

tmp: 3 5

k=1
Department of Computer Science 27
 Merge Algorithm

a: 3 5 15 28 30 b: 6 10 14 22 43 50

i=2 j=0

tmp: 3 5 6

k=2
Department of Computer Science 28
 Merge Algorithm

a: 3 5 15 28 30 b: 6 10 14 22 43 50

i=2 j=1

tmp: 3 5 6 10

k=3
Department of Computer Science 29
 Merge Algorithm

a: 3 5 15 28 30 b: 6 10 14 22 43 50

i=2 j=2

tmp: 3 5 6 10 14

k=4
Department of Computer Science 30
 Merge Algorithm

a: 3 5 15 28 30 b: 6 10 14 22 43 50

i=2 j=3

tmp: 3 5 6 10 14 15

k=5
Department of Computer Science 31
 Merge Algorithm

a: 3 5 15 28 30 b: 6 10 14 22 43 50

i=3 j=3

tmp: 3 5 6 10 14 15 22

k=6
Department of Computer Science 32
 Merge Algorithm

a: 3 5 15 28 30 b: 6 10 14 22 43 50

i=3 j=4

tmp: 3 5 6 10 14 15 22 28

k=7
Department of Computer Science 33
 Merge Algorithm

a: 3 5 15 28 30 b: 6 10 14 22 43 50

i=4 j=4

tmp: 3 5 6 10 14 15 22 28 30

k=8
Department of Computer Science 34
 Merge Algorithm

a: 3 5 15 28 30 b: 6 10 14 22 43 50

i=5 j=4

Done.

tmp: 3 5 6 10 14 15 22 28 30 43 50

k=9
Department of Computer Science 35
 Analysis of Merge Algorithm

 What is its basic operation?

 Number of comparisons(≤) of array elements
 Setup the summation in order to determine how
many times its the basic operation executed?

 n-1 times, where n is the size of array

 What is the efficiency class of this algorithm
 Linear i.e., O(n) time, where n is the size of the array
Department of Computer Science
Solution of Sorting Problem using DnC:
8 3 2 9 7 1 5 4
Divide:

8 3 2 9 7 1 5 4

8 3 2 9 7 1 5 4

8 3 2 9 7 1 5 4

Step 1: Divides its input elements according to their position in the

array

Department of Computer Science

 Divide according to their position

8 3 2 9 7 1 5 4

8 3 2 9 7 1 5 4

8 3 2 9 7 1 5 4

8 3 2 9 7 1 5 4

Department of Computer Science

 Solution of Sorting Problem using DnC:

 Step 2 :Conquer
 Step 3: Merge
 How can we take advantage of a Merge
algorithm to perform step 2 and step 3
simultaneously?
 That is how we utilize Merge algorithm to solve sorting
problem?

Department of Computer Science

 Merge Sort Algorithm
8 3 2 9 7 1 5 4
Divide:
Merge:
8 3 2 9 7 1 5 4

8 3 2 9 7 1 5 4

8 3 2 9 7 1 5 4

3 8 2 9 1 7 4 5

2 3 8 9 1 4 5 7

1 2 3 4 5 7 8 9

Department of Computer Science

 Conquer and Merge
8 3 2 9 7 1 5 4

8 3 2 9 7 1 5 4

8 3 2 9 7 1 5 4

8 3 2 9 7 1 5 4

3 8 2 9 1 7 4 5

2 3 8 9 1 4 5 7

1 2 3 4 5 7 8 9
Department of Computer Science
 Sorting Problem

 How we utilize Merge algorithm to solve sorting

problem?

Department of Computer Science

 Merge Sort Algorithm

MergeSort (A, p, r)
// This algorithm sort A[p..r] by divide & conquer
// Input : a sequence of n numbers stored in array A
//OUTPUT: an ordered sequence of n numbers
1 if p < r
2 then q  (p+r)/2
3 MergeSort (A, p, q)
4 MergeSort (A, q+1, r)
5 Merge (A, p, q, r) // merges A[p..q] with A[q+1..r]

Initial Call: MergeSort(A, 1, n)

Department of Computer Science

 Example of Merge Sort
Original Sequence Sorted Sequence
18 26 32 6 43 15 9 1 1 6 9 15 18 26 32 43

18 26 32 6 43 15 9 1 6 18 26 32 1 9 15 43
43

18 26 32 6 43 15 9 1 18 26 6 32 15 43 1 9

18 26 32 6 43 15 9 1 18 26 32 6 43 15 9 1

18 26 32 6 43 15 9 1

Department of Computer Science Comp 122

Analysis of Merge-sort Algorithm
 Now, how do we describe the running time of the entire
MergeSort algorithm?
 Let T(n) denote the worst case running time of
MergeSort on an array of length n.
 If we call MergeSort with an array containing a single
item (n = 1) then the running time is constant. We can
just write T(n) = 1, ignoring all constants.
 For n > 1, MergeSort splits into two halves, sorts the two
and then merges them together.
 The left half is of size n / 2 and the right half is
 In conclusion we have

Department of Computer Science

Analysis of Merge-sort Algorithm
 How to solve following recurrence relation?

 Let’s expand the terms.

 T(1) = 1
 T(2) = T(1) + T(1) + 2 = 1 + 1 + 2 = 4
 T(3) = T(2) + T(1) + 3 = 4 + 1 + 3 = 8
 T(4) = T(2) + T(2) + 4 = 8 + 8 + 4 = 12
 T(5) = T(3) + T(2) + 5 = 8 + 4 + 5 = 17
 ...
 T(8) = T(4) + T(4) + 8 = 12 + 12 + 8 = 32
 ...
 T(16) = T(8) + T(8) + 16 = 32 + 32 + 16 = 80
 ...
 T(32) = T(16) + T(16) + 32 = 80 + 80 + 32 = 192

 What is the pattern here?

Department of Computer Science
Analysis of Merge-sort Algorithm
 Let’s consider the ratios T(n)/n for powers of 2:
 T(1)/1 = 1 = 0 + 1 = log2(1) + 1
 T(2)/2 = 2 = 1 + 1 = log2(2) + 1
 T(4)/4 = 3 = 2 + 1 = log2(4) + 1
 T(8)/8 = 4 = 3 + 1 = log2(8) + 1
 T(16)/16 = 5 = 4 + 1 = log2(16) + 1
 This suggests:
 T(n)/n = log2 n + 1
 T(n) = n log2 n + n where is n is in term of power
of 2.

Department of Computer Science

Analysis of Merge-sort Algorithm

 Floor and ceilings are a pain to deal with.

 If n is assumed to be a power of 2 (n =2k), this
will simplify the recurrence to:
T(n)

T(n/2)
T(n/2)
Θ(n)

T(n) = T(n/2) + T(n/2) + Θ(n)

T(n) = 2T (n/2) + Θ(n)

Department of Computer Science

Analysis of Merge-sort Algorithm
T(n) = T(n/2) + T(n/2) + θ(n)
T(n) = 2T (n/2) + θ(n)
 The above recurrence relation is non-homogenous
and can be solved by any of the methods
 Iterative Method
 Recursion Tree Method
 Master Theorem
 Substitution

Department of Computer Science

Analysis: Iterative Method
n
T (n)  2.T ( )  n
2
n n n
T ( )  2.T ( 2 ) 
2 2 2
n n n
T ( 2 )  2.T ( 3 )  2
2 2 2
n n n
T ( 3 )  2.T ( 4 )  3 . . .
2 2 2

n n n
T ( k 1 )  2.T ( k )  k 1
2 2 2
Department of Computer Science
Analysis of Merge-sort Algorithm
n n
T (n)  2.T ( )  (n)  2 .T ( 2 )  n  n
2

2 2
n
T (n)  2 .T ( 2 )  n  n
2

2
n
T (n)  2 .T ( 3 )  n  n  n
3

2
...

n
T (n)  2 .T ( k )  n
k
n  .
 n
. .
2 k times
Department of Computer Science
 Analysis of Merge-sort Algorithm
n
T (n)  2 .T ( k )  n
k
n  .
 n
. .
2 k times
n
T (n)  2 .T ( k )  k .n
k

2
Let us suppose that : n  2 k  log 2 n  k

Hence, T (n)  n.T (1)  n. log 2 n  n  n. log 2 n

T (n)  (n. log 2 n)

Department of Computer Science

Visualizing Recurrence Using the Recursion Tree

 Repeated Iteration is a very powerful technique

for solving recurrences.
 But, it is easy to get lost in all the symbolic
manipulations and lose sight of what is going on.
 Here is a nice way to visualize what is going on
in iteration.
 We can describe any recurrence in terms of a
tree, where each expansion of the recurrence
takes us one level deeper in the tree.

Department of Computer Science

Visualizing Recurrence Using the Recursion Tree
 Recall that the recurrence for MergeSort (which
we simplified by assuming that n is a power of 2,
and hence could drop the floors and ceilings)

 Suppose that we draw the recursion tree for

MergeSort, but each time we merge two lists,
we label that node of the tree with the time it
takes to perform the associated (no recursive)
merge. Recall that to merge two lists of size n/2
to a list of size m takes θ(n) time, which we will
just write as n. On next slide is an illustration of
the ofresulting
Department Computer Science recursion tree.
 Analysis of Merge-sort: Recursion Tree

Department of Computer Science

 Analysis of Merge-sort: Master Theorem
 T(n) = aT(n/b)+g(n), a ≥ 1, b > 1
 Master Theorem:
If g(n) є Θ(nd) where d ≥ 0 then
Θ(nd) if a < bd
T(n) є Θ(ndlgn) if a = bd

T(n) = 2T(n/2)+n

Here, a = ?, b = ?, d = ? a = 2, b = 2, d = 1
Which of the 3 cases holds ? a = bd
2 = 21, case 2
Department of Computer Science
Θ(nlgn)
Analysis of Merge-sort: Substitution Method

T(n) = 2T(n/2) + n
 Guess: T(n) = O(nlgn)
 Induction goal: T(n) ≤ cn lgn, for some c and n ≥ n0
 Induction hypothesis: T(n/2) ≤ cn/2 lg(n/2)

 Proof of induction goal:

T(n) = 2T(n/2) + n ≤ 2c (n/2)lg(n/2) + n
= cn lgn – cn + n ≤ cn lgn
if: - cn + n ≤ 0  c ≥ 1
 Base case? 57
Department of Computer Science
  Worst-case of Mergesort is Θ(nlgn)
 Average-case is also Θ(nlgn)
 It is stable.
 Not in-place, needs linear amount of extra memory
 Though we could make it in-place, adds a bit more
“complexity” to the algorithm.
 Can you suggest some possible improvements in this
algorithm?

Department of Computer Science

 Possible
Improvements in
Merge Sort
Algorithm
Department of Computer Science
  Possible improvements:
 Implement bottom-up. Merge pairs of elements,
merge the sorted pairs, so on… (does not require
recursion-stack anymore)
 Could divide into more than two parts, particularly
useful for sorting large files that cannot be loaded
into main memory at once: this version is called
“multiway mergesort”.
 Use insertion sort for small sub arrays. We can
improve most recursive algorithms by handling
small cases differently. Switching to insertion sort
for small sub arrays will improve the running time
of a typical merge sort implementation by 10 to 15
percent.
Department of Computer Science
 Possible improvements:
 The bottom-up (non-recursive) method merges each
subsequent pair of items into sorted passes of length
two.
 Next, we combine these into more sorted runs of
length four.
 And after that, merge those into sorted passes of
length eight, and so on.
 We merge until it returns a sorted array containing all
input array elements of the sorted array left.

Department of Computer Science

  Possible improvements:
[1] [2] [3] [4] [5] [6] [7] [8]
N
Merge(A[1], A[2]); Merge(A[3], A[4]) Merge(A[5], A[6]) Merge(A[7], A[8])

N/2
Merge(A[1-2], A[3-4]) Merge(A[5-6], A[7-8])
N/4

Merge(A[1-4], A[5-8])

N/N

Department of Computer Science

 Let f1(n) = O(g1(n) ; f2(n) = O(g2(n) , then
(f1* f2)(n) = O((g1(n)* g2(n))

As we perform n merges for logn passes, the time complexity is O(nlogn)

Department of Computer Science
 In this case, the outer while loop is
executed k = logn times, once for
each level in the sorting tree except
the topmost level (see Fig).
k = logn

N=8

N/2=4
N/4= 2
N/8= 1
Department of Computer Science
 In the first iteration, there are n/2
comparisons. In the second
iteration, n/2 sorted sequences of
two elements each are merged in
k = logn
pairs. The number of comparisons
needed to merge each pair is either
2 or 3. In the third iteration, n/4
sorted sequences of four elements
each are merged in pairs. The
number of comparisons needed to
merge each pair is between 4 and
N=8 7. In general, in the jth iteration of
the while loop, there are n/2j
merge operations on two subarrays
N/2=4 of size 2j−1 and it follows, that
N/4= 2 the number of comparisons needed
in the jth iteration is between
N/8= 1 (n/2j)2j−1 and (n/2j)(2j − 1).
Department of Computer Science
 Case 1: when (n/2j)2j−1 comparison needed

 Case 2: when (n/2j)(2j − 1).

Department of Computer Science

Department of Computer Science
 Possible improvements:
 Could divide into more than two parts,
particularly useful for sorting large files that
cannot be loaded into main memory at once:
this version is called “multiway mergesort”

Department of Computer Science

 Possible improvements:

Department of Computer Science

 How we do it?

CONCLUSION

Department of Computer Science

A General Divide and Conquer Algorithm

 Three steps are involved:

 Divide the problem into several subproblems,
perhaps of equal size
 Subproblems are solved, typically recursively
 The solutions to the subproblems are combined to
get a solution to the original problem

Real work is done in 3 different places: in partitioning; at the

very tail end of the recursion, when subproblems are so small
that they are solved directly; and in gluing together of partial
solutions

Department of Computer Science

Solution of Sorting problem using Divide and conquer

 Divides its input elements according to their

position in the array
 Divide before processing.
 all work is in combining the partial solutions.
 Famous algorithm: Merge Sort

Department of Computer Science

 Design and Analysis of Algorithm

Tanveer Ahmed Siddiqui

Department of Computer Science

COMSATS University, Islamabad
 Recap and Today Covered

Algorithm Design and Analysis Process

Understand the problem

Decide on : algorithm
design techniques etc.

Design an algorithm

Prove correctness

Analyze efficiency etc

Code the algorithm

Department of Computer Science
 Reading Material

Read Chapter 5
Divide and Conquer

Department of Computer Science

 Objectives

 How to Design Algorithm using divide and

conquer Approach
 Designing solution for sorting problem
 Designing solution to Matrix Multiplication
 D.Y.S
 Designing solution to Integer Multiplication
 D.Y.S

Department of Computer Science

Recap

Lecture No 20

Department of Computer Science 77

A General Divide and Conquer Algorithm

Department of Computer Science

A General Divide and Conquer Algorithm
Problem of size n

Subproblem 1 Subproblem 2
of size n/2 of size n/2
Don’t assume
always breaks up
into 2, could be > 2
Solution to subproblems Solution to
subproblem 1 subproblem 2

Solution to the
original probelm

Department of Computer Science

 Sorting Problem : Revisit

 Problem: Arrange the elements of the set such that the

values are in a given order (an increasing or a
decreasing order).
 What is input
 An array A of n numbers e.g. 55 99 6 29 1 7 3
 What should be the output
 List of item arranged in specific order
 Which designing technique?
 Divide and Conquer

 How we divide the array?

 Divides its input elements according to their position in the
array(Previous Lecture)
 Divides input elements according to their value(Todays Lecture)

Department of Computer Science

 Example-2

 Problem: Design an algorithm that split the list around a

split value called pivot so that all the elements in the
positions before the pivot are smaller than or equal to
the pivot and those after the pivot are larger than (or
equal) to the pivot
INPUT

OUTPUT

Department of Computer Science

 Example-2

Department of Computer Science

 Example-2

Department of Computer Science

 Example-2

Department of Computer Science

 Example-2

Department of Computer Science

 Example-2

Department of Computer Science

 Example-2

Department of Computer Science

 Example-2

Department of Computer Science

 Example-2

Department of Computer Science

 Example-2

Department of Computer Science

 Analysis of Partition Algorithm

 What is its basic operation?

 Number of comparisons(≤) of array elements
 Setup the summation in order to determine
how many times its the basic operation
executed?

 n-1 times, where n is the size of array

 What is the efficiency class of this algorithm
 Linear i.e., O(n) time, where n is the size of the array
Department of Computer Science
 Sorting Problem using DnC: Revisit
 How can we take advantage of a list partition
algorithm to sort the list?
 Think about: what does it buy us?
 The pivot element ends up in the position it retains in the final
sorted order.
 After a partitioning, no element flops to the other side of the
pivot in the final sorted order.
 Thus, we can sort the elements to the left of
the pivot and the right of the pivot
independently!
 This gives us a recursive sorting algorithm,
since we use the portioning approach to sort
each sub problem.
Department of Computer Science
 Example-3
 How can we take advantage of a list partition algorithm
to sort the list?
 Obviously, after a partition is achieved, A[s] will be in its
final position in the array

 We can continue sorting the two sub arrays to the left

and to the right of A[s] independently (e.g., by the
same method).

Department of Computer Science

 p q r

Department of Computer Science

 Sort Demo

Department of Computer Science

Department of Computer Science
Department of Computer Science
Department of Computer Science
Department of Computer Science
Department of Computer Science
Department of Computer Science
Department of Computer Science
Department of Computer Science
Department of Computer Science
 Quick sort Trace

Department of Computer Science

 Analysis of Quick Sort

 Set up a recurrence relation of the given

algorithm.

 Analysis: The key to quick sort will be finding a

good pivot point. In fact, we can say that
 T(n) = T(nl) + T(nr) + (n)
 where nl and nr are the number of items in the
right and left half, respectively, so nl+nr+1 = n.
 (n) is the time required for the partition.
Department of Computer Science 106
 Analysis of Quick Sort

 Let T(n) be the number of comparisons required

by Quicksort

 T(n) = T(nl) + T(nr) + (n)

 If the pivot ends up at position k, what will be the

value of nl and nr in term of k?
 nl = k 1 nr = nk
T(n) T(nk)  T(k 1)  n

 Partition can be done in O(n) time, where n is the size
of the array
Department of Computer Science 107
Best-, Worst-, and Average-case of Quick Sort

 To determine best-, worst-, and average-case

complexity we need to determine the values of k
that correspond to these cases.
 What is value of k for best case, and worst case?
 When does best case occur?
 nl =? and nr= ?
 When does worst case occur?
 nl=? and nr= ?

Department of Computer Science 108

 Best case complexity of Quick Sort

 The best case is clearly when the pivot always

partitions the array equally.

 In such case nl  nr= n/2

 T(n) T(n/2)  T(n/2)  n  (n lg n)

Department of Computer Science 109

 Best case complexity of Quick Sort

 The best case is clearly when the pivot always

partitions the array equally.
 When does pivot always partitions the array equally.
 If pivot chosen is the median value
 Median Value
 The Median is the "middle" of a sorted list of
numbers.

Department of Computer Science 110

Department of Computer Science 111
Best-, Worst-, and Average-case of Quick Sort

 If n is odd then the median is defined to be

element of rank (n + 1)/2.
 When n is even, there are two choices(which
are):
 n/2 and (n + 1)/2.
 Medians are useful as a measures of the central
tendency of a set especially when the distribution
of values is highly skewed.

Department of Computer Science 112

 Example of Best case scenario

 Can you give an example of a list of n items

(e.g., an array of 10 integers) representing the
best- case scenario.

Department of Computer Science

 Example of Best case scenario

Department of Computer Science

 Example of Best case scenario

Department of Computer Science

 Example of Best case scenario

Department of Computer Science

 Example of Best case scenario

Department of Computer Science

 Example of Best case scenario

Department of Computer Science

 Example of Best case scenario

Department of Computer Science

 Example of Best case scenario

Department of Computer Science

 Example of Best case scenario

Department of Computer Science

 Example of Best case scenario

Department of Computer Science

 Example of Best case scenario

Department of Computer Science

 Example of Best case scenario

Department of Computer Science

 Example of Best case scenario

Department of Computer Science

 Example of Best case scenario

Department of Computer Science

 Worst case complexity of Quick Sort

 The worst-case is when the pivot always ends

up in the first or last element. That is,
partitions the array as unequally as possible.

Department of Computer Science 127

 Worst case complexity of Quick Sort
 The worst-case is when the pivot always ends
up in the first or last element. That is,
partitions the array as unequally as
possible.
 In this case: T(n) = T(nl) + T(nr) + (n)
 nl = n-1 (or 0)
 nr= 0( n-1)
 T(n)  T(n1)  T(11)  n
 T(n1)  n Decrease by one
 n  (n1)  (n2)+ …+2+ 1
 n(n  1)/2  (n2)
FOOD FOR THOUGHT: When does divide and conquer
Department of Computer Science
design technique is not a suitable choice? 128
 Example of Worst case scenario

 Can you give an example of a list of n items

(e.g., an array of 10 integers) representing the
worst- case scenario?

Department of Computer Science

Department of Computer Science
Department of Computer Science
Department of Computer Science
Department of Computer Science
Department of Computer Science
Department of Computer Science
Department of Computer Science
 Best case of Insertion Sort
ALGORITHM InsertionSort(A[0..n-1])
for i <- 1 to n-1 do
v <- A[i] 17 | 29 34 45 68 89 90

j <- i-1 17 29 | 34 45 68 89 90
while j ≥ 0 and A[j] > v do
17 29 34 | 45 68 89 90
A[j+1] <- A[j]
j <- j-1 17 29 34 45 | 68 89 90
A[j+1] <- v
17 29 34 45 68 | 89 90

17 29 34 45 68 89 | 90

17 29 34 45 68 89 90

Department of Computer Science

 Worst case of Quick Sort

17 | 29 34 45 68 89 90
17 | 29 34 45 68 89 90

17 29 | 34 45 68 89 90

If we select first element as pivot 17 29 34 | 45 68 89 90

element
17 29 34 45 | 68 89 90
Decrease by one
17 29 34 45 68 | 89 90

17 29 34 45 68 89 | 90

17 29 34 45 68 89 90

(n2)
Department of Computer Science
 Comparison of Merge sort and Quick Sort

Department of Computer Science 139

Solution of Sorting problem using Divide and conquer

 Divides input elements according to their position

in the array
 Divide before processing.
 all work is in
 combining the partial solutions.
 Divides input elements according to their value.
 Process before dividing
 all work is in
 dividing the problem, Combining does not require any
work.

Department of Computer Science

 Average case complexity of Quick Sort

 Average case is rather complex, but is where

the algorithm earns its name. The bottom line
is:
T (n)  1.386n lg n  (n lg n)

Department of Computer Science 141

 Average case complexity of Quick Sort

 To justify its name Quicksort had better be

good in the average case
 Showing this requires some fairly intricate
analysis.
Average case is rather complex, but is
where the algorithm earns its name.
The bottom line is:

T (n)  1.386n lg n  (n lg n)

Department of Computer Science 142

 Meaningful Learning

How we do
that?

Department of Computer Science

 Meaningful Learning
through Identifying
Differences and
Similarities between
Certain Algorithms and
Problems
Department of Computer Science
 Mapping problems to Related ones

It is an excellent approach to make one to one corresponding

between two problems and then solving one problem based on the
approach already used in the solution of the other problem

Identify different problems with the same solution

Map one problem to another

Department of Computer Science

Department of Computer Science
Department of Computer Science
Department of Computer Science
Department of Computer Science
Department of Computer Science
Department of Computer Science
Department of Computer Science
Department of Computer Science
Department of Computer Science
Department of Computer Science
Department of Computer Science
Department of Computer Science
Department of Computer Science
Department of Computer Science
Department of Computer Science
Department of Computer Science
Department of Computer Science
Department of Computer Science
Department of Computer Science
Department of Computer Science
Department of Computer Science
Department of Computer Science
Department of Computer Science
Department of Computer Science
 TAS Version

Department of Computer Science

 (Relation between Binary
Search Tree, In-oder Traversal
and Quicksort . . .)

Department of Computer Science 171

 How would you create a BST using an array ?
 A[n] = 3 1 8 2 6 7 5, where A is an array and n
is the size of it
 BST insert comes out like this

3 1 8 2 6 7 5

Department of Computer Science

 When you do an in-order traversal of the above
BST, you get the sorted array..
 In-order: 1 2 3 5 6 7 8
 So, this means, we have to do n BST inserts and
then do an in-order traversal to get a sorted
array.
 This itself is a new sorting method named BST
sort. :) BST-Sort( A[1……n])
// Input: An Array of orderable items
//Output: Sorted Array
Tree T
array A
for i = 0 to n do
Tree-Insert(T, A[n])
Department of Computer Science Inorder-Traverse-Tree(T)
 What is the performance of the BST sort?
 There are ‘n’ Tree inserts, so its O(nh) where h is the
height of the tree.
 and then an in-order traversal is done which is O(n) .
So it will be max{O(nh),O(n)}= O(nh).
 But we know that height of the tree can be computed
from number of nodes in it: h = log n.

 Time complexity of BST Sort is = O(n log n)

Department of Computer Science

 But, take an example of an sorted(or reverse sorted)
array.
 A[n] = {1,2,3,4,5,6}
 If you create a BST with these values,
you get a tree like this.

 What is the performance of the Tree-Insert in the above

for loop? In such a case, Height of the tree is equal to
the number of nodes!!

Department of Computer Science

 height of tree, h = log n, holds approximately
only for trees which are partially balanced. So, if
height equals number of nodes, you get O(n2)
even for this BST sort.

Department of Computer Science

 height of tree, h = log n, holds approximately
only for trees which are partially balanced. So, if
height equals number of nodes, you get O(n2)
even for this BST sort.
 How to construct balanced binary search tree if list is
sorted?
1) Get the Middle of the array and make it root.
2) Recursively do same for left half and right half.
a) Get the middle of left half and make it left child of the root
created in step 1.
b) Get the middle of right half and make it right child of the
root created in step 1.

 The "middle" of a sorted list of numbers is called

Median.
Department of Computer Science
Similarity between BST Sort and Quick sort
BST Sort Quick Sort

Quick sort as well creates a BST

internally. Just way comparisons are
made is different.

Department of Computer Science 178

Department of Computer Science
Department of Computer Science
Department of Computer Science
Department of Computer Science
Department of Computer Science
Department of Computer Science
Department of Computer Science
Department of Computer Science
Department of Computer Science
Department of Computer Science
Department of Computer Science
Department of Computer Science
Similarity between BST Sort and Quick sort
BST Sort(Worst case) Quick Sort(Worst case)

Can’t you recollect that, this is matching with

performance of the Quicksort algorithm ?
Yes, it is matching!! :)
Quick sort gives O(n2) performance if the array is already
sorted.

Department of Computer Science 191

Similarity between BST Sort and Quick sort
BST Sort(Best case) Quick Sort(Best case)

Can’t you recollect that, this is matching with

performance of the Quicksort algorithm ?
Yes, it is matching!! :)
Quick sort gives O(nlogn) performance if the array partitioned
balanced(equally)

Department of Computer Science 192

Similarity between BST Sort and Quick sort
BST Sort(Best case) Quick Sort(best case)

Can’t you recollect that, this is matching with

performance of the Quicksort algorithm ?
Yes, it is matching!! :)
Quick sort gives O(nlogn) performance if the array partitioned
balanced(equally)

Department of Computer Science 193

Similarity between BST Sort and Quick sort
Balanced BST Sort Randomized Quick Sort

To avoid worst case in sorted files, randomized quick sort came

into picture.
Randomized quick sort is the basis for all kinds of
self-balancing trees!!!
median of three partitioning avoids worst case in sorted files
If you understand how randomized quick sort work, we can
understand AVL, red-black trees etc..

Department of Computer Science 194

 Analysis of Quick sort
average case

Department of Computer Science

Department of Computer Science
Department of Computer Science
Department of Computer Science
Department of Computer Science
Department of Computer Science
 n Let us assume that T (n) < c n log n for n > 2
where c is a Constant

Department of Computer Science

Department of Computer Science
Department of Computer Science
Department of Computer Science
Department of Computer Science
Department of Computer Science
Department of Computer Science
Department of Computer Science
Department of Computer Science
Department of Computer Science
Department of Computer Science
Department of Computer Science
Department of Computer Science
Department of Computer Science
Department of Computer Science
Department of Computer Science
Department of Computer Science
Department of Computer Science
 Designing Algorithm for
MATRIX MULTIPLICATION
problem using DIVIDE AND
CONQUER

Department of Computer Science

 Matrix Multiplication
How do we multiply two 2×2 matrices ?
q
Apq  Bqr  C pr where cij   aik bkj
k 1

2 5 Basic Operation: scalar multiplications

2 5 1  
4 2 3  1 1 
   3 2
 
2  2  5  1  1  3 4  2  2  1  3  3
 
 2  5  5  1  1  2 4  5  2  1  3  2

Department of Computer Science

A General Brute force Algorithm

 What is Time Complexity of above algorithm?

Department of Computer Science

A General Divide and Conquer Algorithm

Recursive

Department of Computer Science

Analysis of Divide and Conquer Algorithm

Can we improve it further using divide and conquer?

Department of Computer Science

 Div. & Conq. : Strassen’s Matrix
Multiplication
 How do we multiply two 2×2 matrices ?
1 2 3 5 5 13 How many multiplications
=
and additions did we need?
3 4 1 4 13 31
8 mults and 4 adds
V. Strassen in 1969 found out, he can
do the above multiplication in the following way:
m1+m4-m5+m7 m3+m5
a00 a01 b00 b01 c00 c01
= =
c10 c11
m2+m4 m1+m3 - m2+m6
a10 a11 b10 b11

m1 = (a00+a11)*(b00+b11) m2 = (a10+a11)*b00 m3 = a00*(b01-b11)

m4 = a11*(b10-b00) m5 = (a00+a01)*b11
7 mults
m6 = (a10-a00)*(b00+b01) m7 = (a01-a11)*(b10+b11)18 adds/subs

Department of Computer Science

 Div. & Conq. : Strassen’s Matrix
Multiplication
 Let us see how we can apply Strassen’s idea for
multiplying two n×n matrices
Let A and B be two n×n matrices where n is a power of 2

A00 A01 B00 B01 C00 C01

=
A10 A11 B10 B11 C10 C11

Each block is (n/2)×(n/2)

E.g., In Strassen’s
You can treat blocks method,
as if they were numbers M1 = (A00+A11)*(B00+B11)
to get the C = A*B M2 = (A10+A11)*B00
etc.

Department of Computer Science

 Div. & Conq. : Strassen’s Matrix
Multiplication
ALGORITHM Strassen(A, B, n)
//Input: A and B are n×n matrices Recurrence for
//where n is a power of two
//Output: C = A*B
# of multiplications is
M(n) = 7M(n/2) for n > 1
if n = 1
A00 A01 B00 Breturn M(1) = ?
01 C = A*B
else For n = 2m,
A10 A11 B10 B11
M(n) = 7M(n/2) = 72M(n/22)
Partition A = and B =
M(n) = 7m M(n/2m)
where the blocks Aij and Bij are (n/2)-by-(n/2) = 7m = 7lgn
M1 <- Strassen(A00+A11, B00+B11, n/2) = nlg7 ≈ n2.807
M2 <- Strassen(A10+A11, B00, n/2)
M3 <- Strassen(A00, B01-B11, n/2)
M4 <- Strassen(A11, B10-B00, n/2)
For # of adds/subs,
M5 <- Strassen(A00+A01, B11, n/2) A(n) = 7A(n/2)+18(n/2)2 for n > 1
M6 <- Strassen(A10-A00, B00+B01, n/2) A(1) = 0
M7 <- Strassen(A01-A11, B10+B11, n/2)
C00 <- M1+M4-M5+M7 Using Master thm,
C01 <- M3+M5 A(n) є Θ(nlg7) better
C10 <- M2+M4
C11 <- M1+M3-M2+M6 than brute-force’s Θ(n3)
return C =

C00 C01 DONE WITH STRASSEN!

Department of Computer Science C10 C11
 Designing Algorithm for
INTEGER MULTIPLICATION
problem using
DIVIDE AND CONQUER

Department of Computer Science

 Div. & Conq. : Multiplication of
Large Integers
 We want to efficiently multiply two very large
numbers, say each has more than 100 decimal
digits
 How do we usually multiply 23 and 14?
 23 = 2*101 + 3*100, 14 = 1*101 + 4*100
 23*14 = (2*101 + 3*100) * (1*101 + 4*100)
 23*14 = (2*1)102 + (2*4+3*1)101+(3*4)100
 How many multiplications? 4 = n2

Department of Computer Science

 Div. & Conq. : Multiplication of
Large Integers
 23*14 = (2*1)102 + (2*4+3*1)101+(3*4)100
We can rewrite the middle term as:
What has been gained?
(2*4+3*1) = (2+3)*(1+4) - 2*1 - 3*4
We have reused 2*1 and 3*4 and now need
one less multiplication
If we have a pair of 2-digits numbers a and b
a = a1a0 and b = b1b0
we can write c = a*b = c2102+c1101+c0100

c2 = a1*b1 c0 = a0*b0

c1 = (a1+a0)*(b1+b0)-(c2+c0)

Department of Computer Science

 Div. & Conq. : Multiplication of Large
Integers
If we have a pair of 2-digits numbers a and b
a = a1a0 and b = b1b0
we can write c = a*b = c2102+c1101+c0100
c2 = a1*b1 , c0 = a0*b0 a = 1234 = 1*103+2*102+3*101+4*100
c1 = (a1+a0)*(b1+b0)-(c2+c0) = (12)102+(34)
If we have two n-digits numbers, (12)10 2+(34) = (1*101+2*100)102+3*101+4*1

a and b (assume n is a power of 2) Apply the same idea

a: a1 a0 recursively to get c2, c1, c0
n/2 digits until n is so small that you can
b: b1 b0 directly multiply

We can write,
a = a110n/2 + a0
Why?
b = b110n/2 + b0
c2 = a1*b1 c0 = a0*b0
c1 = (a1+a0)*(b1+b0)-(c2+c0)

Department of Computer Science

 Div. & Conq. : Multiplication of Large
Integers

Notice: a1, a0, b1, b0 all are n/2 5 additions

digits numbers 1 subtraction
c2 = a1*b1 c0 = a0*b0
So, computing a*b requires
three n/2-digits multiplications c1 = (a1+a0)*(b1+b0)-(c2+c0)

Recurrence for the number of

Multiplications is Assume n = 2m
M(n) = 3M(n/2) = 3[ 3M(n/22) ] = 32M(n/22)
M(n) = 3M(n/2) for n > 1
M(1) = ? M(n) = 3mM(n/2m) = ?
How many additions
And subtractions?
# of add/sub, M(n) = 3m = 3lgn = nlg3
A(n) = 3A(n/2)+cn for n > 1 Why?
A(1) = 0
Using Master Thm, M(n) ≈ n1.585
A(n) є Θ(nlg3)

Department of Computer Science

 Conclusion

Department of Computer Science

 What is Divide and Conquer?

 3rd algorithm design technique, we studied

today
 Very important: two out of ten most
influential algorithms in the twentieth
century is based directly on “Divide and
Conquer” technique.

Department of Computer Science

A General Divide and Conquer Algorithm
Step 1:
 If the problem size is small, solve this problem
directly
 Otherwise, split the original problem into sub-
problems(2 or more) with almost equal sizes.
Step 2:
 Recursively solve these sub-problems by
applying this algorithm.
Step 3:
 Merge the solutions of the sub- problems into a
solution of the original problem.

Department of Computer Science

A General Divide and Conquer Algorithm

 Three steps are involved:

 Divide the problem into several subproblems,
perhaps of equal size
 Subproblems are solved, typically recursively
 The solutions to the subproblems are combined to
get a solution to the original problem

Real work is done in 3 different places: in partitioning; at the

very tail end of the recursion, when subproblems are so small
that they are solved directly; and in gluing together of partial
solutions

Department of Computer Science

Solution of Sorting problem using Divide and conquer

 Divides its input elements according to their

position in the array
 Divide before processing.
 all work is in combining the partial solutions.
 Famous algorithm: Merge Sort
 Divides input elements according to their value.
 Process before dividing
 all work is in dividing the problem, Combining
does not require any work.
 Famous algorithm: Quick Sort

Department of Computer Science