Outcome (1) and (3): How to analyze recursive algorithms?

Will teach an algortihm design technique.

Recurrence, recursive function, divide-and-conquer approach
Review:
A function is recursive if it is defined in terms of itself.

Let n be positive integers Two components of a recursion

1 n=1 Base case: value of f(n) is known

e.g. f(n)= f(n-1)+n n>1 Recursive part: by following
this recursion, the value of
f(n) can be calculated by
f(4) = f(3) + 4
making progress towards the
= f(2) + 3 +4
= f(1) + 2 + 7
base case
=1+9 (non-recursive formula)
= 10 Q: How to find a “closed form” for f(n)?
(how to solve it?)
1
One simple approach: The Iteration Method
1 n=1
e.g. f(n)= f(n-1)+n n>1
f(n) = f(n-1) + n Remarks:
= f(n-2) + (n-1) + n 1) Then, formally prove it by MI.
= …..
= f(1) + 2 + 3 + … + n 2) Sometimes we only need to
= n(n+1)/2 know the solution in asymptotic
notation.
MI proof (sketch)
Base: when n = 1, f(n) = 1 by definition.
n(n+1)/2 = 1, so f(n) = n(n+1)/2 is true.
Induction: for n = k,
f(k) = f(k – 1)+k by definition
= (k-1)k/2 + k by hypothesis
= k(k+1)/2.

2
 1 n=1
e.g. f(n)= f(n-1)+n n>1

f(n) = f(n-1) + n If we only need to show that

= f(n-2) + (n-1) + n f(n) = O(n2), we only need to
= ….. prove f(n) £ cn2 for some
= n(n+1)/2 constant c for n ³ 1 using MI.
Step 1: try to get the value of c (do it in rough paper)
For the induction to go through,
f(n) = f(n-1) + n £ c(n-1)2 + n = cn2 – (2cn – c – n).
Since we want f(n) £ cn2, so we set 2cn – c – n ³ 0
Þc ³ n/(2n-1) where n/(2n-1) £ 1 for n ³ 1
So, we can set c = 1.
Step 2: formal proof
We will show that f(n) £ n2 for all n ³ 1.
Base:….
Induction: ….
Take-home exercise:
Prove f(n) = O(n): f(n) = 3 for n = 1, otherwise f(n) = f(n-1) + 10.
3
To analyze a recursive algorithm,
(1) we model the running time of the algorithm as a
recursive function, then
(2) solve the recursive function to get the answer in
asymptotic notation.

Alg(n) { Let f(n) be the running time of

Alg(n).
if (n = 1) return 1;
temp = Alg(n-1);
for (i = 1 to n) c1 n=1
e.g. f(n)= f(n-1)+c n n>1
temp = temp + 1; 2

return temp;
} where c1 and c2 are constants

The time complexity of Alg(n) is O(n2).

4
Solving recurrences (obtain the asymptotic bounds on the
solution)

[MIT, ch4.3] substitution method

[MIT, ch4.4] recursion-tree method

[MIT, ch4.5] master method

Or you can get related material from web or any

related reference books
Can you learn it yourself or let me know later if you
want a tutorial [will NOT be a main topic in the exam]?

5
 Divide-and-conquer approach usually gives a recursive
algorithm and the time complexity of the algorithm is
usually formulated as a recursive function. Solving the
recursive function will give the time complexity of the
algorithm.
ØDivide problem into subproblems
ØConquer subproblems recursively
ØCombine subproblem solutions

Simple example: calculate n! Let T(n) be the running time of fact(n)

Let f(n) = n!, c1 if n=1
then f(1) = 1 and f(n) = n*f(n-1) Then, T(n) = T(n-1)+c2 if n>1
fact(n){
if (n = 1) return 1; where c1,c2 are constants
else return n*fact(n-1); T(n) = T(n-1)+c2
} = T(n-2)+c2+c2
= …..
= T(1) + (n-1)c2 = c1 + (n-1)c2
= O(n) 6
Review: Tower of Hanoi
Given 3 pegs and n disks of different sizes placed in an
increasing order of size on one peg, transfer the disks from
the original peg to another peg such that:

- only one disk can be moved in each step

- in each peg, the disks must be in increasing order of size

How?

1 2 3

7
Solve it recursively:
Base case: if there is only one disk.
move disk from peg 1 to peg 3

1 2 3

Assuming that we know how to move n-1 disks,

can we solve it for n disks?

8
 Move top (n-1) disks from peg 1 to peg 2

1 2 3
Move the largest disks from peg 1 to peg 3

1 2 3
Move (n-1) disks from peg 2 to peg 3

1 2 3 9
TofH(A,B,C,n) // source peg: A, destination peg: C
{ To call the program:
if (n=1) TofH(1, 2,3, 3) // 3 disks
move disk from A to C
else {
What is the number of moves?
TofH(A,C,B,n-1);
move disk from A to C; Let f(n) be the number of moves
TofH(B,A,C,n-1);
f(n) = 1 n=1
} 2f(n-1)+1 n>1
}
f(n) = 2f(n-1) + 1
= 2(2f(n-2)+1) + 1 = 22f(n-2) + 2 + 1
= 22(2f(n-3)+1) + 2 + 1 = 23f(n-3) + 22 + 2 + 1
= ….
= 2n-1f(1) + 2n-2 + … + 2 + 1
= 2n-1 + 2n-2 + … + 2 + 1 How about time
= 2n - 1 complexity of
the algorithm?
10
Review: Divide-and-conquer approach
(1) In order to obtain a solution for input size n, we assume
the solution(s) for smaller n (subproblem) is available,
then construct the final solution from these solutions.

E.g. computing n! (assuming (n-1)! is available)

E.g. solving the Tower of Hanoi with n disks
(assuming we know how to do (n-1) disks)
(2) Don’t forget the base case:
E.g. n! = 1 when n = 1
E.g. Solving the Tower of Hanoi problem with 1 disk
(3) Derive a “recursive” algorithm from this idea.

(4) Capture the running time of the algorithm by a

recursive function, then solve the recurrence for the time
complexity of the algorithm.
Pay attention to the subproblem definition =>
11
 A student tries to solve the Tower of Hanoi problem
recursively as follows: Assume that we know how to
move n/2 disks (n: even for simplicity)
TofH2(A,B,C,n) // source peg: A, destination peg: C
{
if (n=1)
move disk from A to C
else {
TofH2(A,C,B,n/2); // move 1st half from A to B
TofH2(A,B,C,n/2); // move 2nd half from A to C
TofH2(B,A,C,n/2); // move 1st half from B to C
} Is it correct?
}
** Definition: TofH(A,B,C,n): move n disks from A to C with B
empty or having disks with the correct order & all larger than
the given n disks.

Q: to solve this problem, is 2n – 1 the minimum

number of moves? 12
Example: Find the largest number from a given set of n
numbers using divide-and-conquer approach
Divide the problem into subproblems
1) Solve base cases for subproblems
2) Assume that solutions of subproblems are
available, combine them to find the solution for
the original problem

There may be more than one way to define the subproblems.

Recall that for the problem of Tower of Hanoi:

problem: moving n disks
subproblem: moving (n-1) disks

13
We can do something similar here:
problem: finding largest in n numbers
subproblem: finding largest in first (n-1) numbers

Base case: if there is only one number, return that number!

Recurrence: if the subproblem is solved, say x is the

solution, how can we find the largest number among all n
numbers?

Check if (x > A[n]) // A stores the numbers

return x;
else
return(A[n]);

14
largest1(A[1..n]) {
if (n = 1) return A[1]; Execution:
else e.g. Ans: 60
x = largest1(A[1..n-1]); largest1([4, 60, 21, 3, 5])
if (x > A[n]) return x;
else return A[n];
largest1([4, 60, 21, 3]) return
}
60
Time complexity: largest1([4, 60, 21])
return
T(n) = c1 if n =1;
60
T(n) = T(n-1) + c2 for n > 1. largest1([4, 60])
return 60
=> T(n) = O(n). largest1([4])
Base case: return 4

15
Example: Find the largest number from a given set of n numbers

Another way to define subproblems:

Divide the numbers into two groups of size n/2 each.

Base case: If the group has only one number,

return it.
Recurrence: Assume that the two
subproblems have been solved and x1, x2 are
returned, what we do to get the final
answer?
Check if (x1 > x2)
return x1;
else
return x2;

16
largest2(A[1..n]) // for simplicity, assume n is a power of 2
{
If (n = 1)
return A[1];
else {
x1 = largest2(A[1..n/2]);
x2 = largest2(A[n/2+1..n]);
if (x1 > x2)
return x1;
else
return x2;
}
}
Let T(n) be the time complexity of largest2.
T(1) = c1
T(n) = 2T(n/2) + c2 where c1, c2: positive constants
Þ T(n) = O(n)
17
largest2(A[1..n])
{ e.g. 14 65 34 33 7 56 100 20
If (n = 1)
return A[1]; 14 65 34 33 7 56 100 20
else {
x1 = largest2(A[1..n/2]); 14 65 34 33
x2 = largest2(A[n/2+1..n]);
if (x1 > x2)
return x1; 14 65 34 33
else Base case
return x2;
}
}
Remark: This problem, of course,
can be solved using a more
straightforward approach, so it is
only for illustration purpose

18
Note: the following straightforward approach will give
an algorithm with the same time complexity.

largest(A[1..n])
{
maximum = A[1];
for i = 2 to n do
if (A[i] > maximum)
maximum = A[i];
return maximum;
}

19
Remarks:
Theorem :
If T(n) = a T(n/b) + f(n) and f(n) = Q(np), then
Q( n p ) if a < b p
T(n) = Q( n p log n ) if a = b p where a ³1, b ³ 2, p ³ 0
Q( n logb a ) if a > b p

How about n ¹ bk (e.g. b = 2) for any integer k?

Usually the time complexity remains the

same, but the algorithm will be
complicated to handle extra cases.
Interested students can explore more on
this aspect.

20
 Note: recursion may not always be a good solution
The Fibonacci numbers
F1 = 1
Definition: F2 = 1
Fi = Fi-1 + Fi-2 for i > 2
1, 1, 2, 3, 5, 8, 13, 21, 34, 55, …..
Problem: Generate the nth Fibonacci number
Algorithm 1: Direct implementation of recursive definition

FIB1(n) { Time complexity?

if (n <=2) then c1 if n£2
T(n) =
return 1 T(n-1)+T(n-2) + c2 if n>2
else
return (FIB1(n-1) + FIB1(n-2)) where c1,c2 are constants
}

21
Claim: T(n) > 2(n-2)/2
Proof: T(n) > T(n-1) + T(n-2)
> T(n-2) + T(n-3) + T(n-2)
= 2T(n-2) + T(n-3)
> 2T(n-2)
> 2(2T(n-4)) = 4T(n-4)
> 4(2T(n-6)) = 8T(n-6)
> ....
> 2k T(n-2k)

If n is even, n-2k = 2. So, T(n) > 2(n-2)/2

If n is odd, n-2k = 1. So, T(n) > 2(n-1)/2
Therefore, T(n) is exponential in n and FIB1 runs in
exponential time.

1
Note: Fn can be shown to be @ 1.618n T(n) = Q(1.618n)
5
22
 In the recursive version, we need to
Can we do better?
recompute FIB1(n) values for many n’s.

Algorithm 2: Calculate Fn by a loop

FIB2(n) {
if (n <=2) then
return 1
else
p=1 Time Complexity?
q=1
Q(n)
for i = 3 to n do
r=p+q
p=q
q=r Q: Do you think we can
return r still do better?
}
23
Revisit: Computing Fibonacci Numbers

We first consider the following problem.

Given a 2x2 matrix A, design an algorithm to compute An
for any given n ³ 1.
1st Attempt: A straight-forward implementation

Power1(A, n) { // n ³ 1; A: 2 ´ 2 matrix
if (n = 1)
return A;
else
return 2-Multiply(A, Power1(A, n-1)); // details of 2-
} // Multiply omitted

Complexity:
2-Multiple takes constant time.
Power1(A, n) takes O(n) time.
24
2nd Attempt:
[Hint: if n is always a power of 2, can you design a better
(faster) algorithm?]

Power2(A, n) { // n = 2k for a non-neg int k; A: 2 ´ 2 matrix

if (n = 1)
return A;
else
B = Power2(A, n/2);
return 2-Multiply(B, B);
}
Let T(n) be the running time for Power2(A, n).
Complexity:
c1 if n = 1
T(n) =
T(n/2) + c2 if n > 1
where c1,c2 are constants

T(n) = O(log n). 25

How about for general n?
e.g. n = 10 If n is odd,
A10 = A5 x A5 An = A x (An-1/2)2
To compute A5: A5 = A x A4 else
To compute A4: A4 = A2 x A2 An = An/2 x An/2
And A2 = A x A.

Power3(A, n) { // n ³ 1; A: 2 ´ 2 matrix
if (n = 1)
return A;
else Complexity: O(log n)
if (n is odd)
B = Power3(A, (n-1)/2);
return(2-Multiply(A, 2-Multiply(B, B));
else
B = Power3(A, n/2);
return(2-Multiply(B, B));
}
26
How this relates to our problem (computing the Fibonacci #)?
[MIT, Ex31-3, p.981]

Hint: Consider the following matrix and its powers.

0 1
1 1

See if you can design a faster algorithm for computing

the n-th Fibonacci #.

27
One more example: The maximum subarray problem
Given an array A[1..n] of integers, find a nonempty,
contiguous subarray of A whose values have the largest sum.

e.g. -4 6 -3 -1 6 1 -2

A[2..6] gives the largest sum (9)

28
1st attempt (brute-force approach)
Idea: check all possible subarrays

Max_subarray1(A, n){
max = -¥
for i = 1 to n
for j = i to n
sum = 0;
for k = i to j
sum = sum + A[k];
if sum > max
max = sum;
ans = (i, j, max);
return ans;
Time complexity: O(n3)
Can we do better?

29
This brute-force algorithm can be further improved due to
many redundant addition

i/j 1 2 3 4 …
1 1
2 2 1
3 3 2 1
4 4 3 2 1 ….
….. ….. ….. ….. ….. ….
redundant
When i = 1; j = 1, we compute sum = A[1]
When i = 1; j = 2, we compute sum = A[1] + A[2]
When i = 1; j = 3, we compute sum = A[1] + A[2] + A[3]
When i = 1; j = 4, we compute sum = A[1] + A[2] + A[3] + A[4]

Do you know how to improve it to O(n2) time?

30
2nd attempt (brute-force approach with improvement)
Idea: check all possible subarrays, faster computation of sum

Time complexity:
O(n2)

Can we do better?
31
 3rd attempt: How about divide-and-conquer approach?

middle
You know how to find the maximum subarray in the 1st half
You know how to find the maximum subarray in the 2nd half
What else you need to find?

e.g. 3 -1 3 -5 4 5 -8 2 0 4

Recursion for 1st half returns (A[1..3], max = 5)

Recursion for 2nd half returns (A[8..10], max = 6)
But the correct answer is (A[5..6], max = 9)

We need to find also the maximum subarray across the mid point!

32
 Finding the max subarray across the middle:

Idea:

Search for A[i, n/2] for 1 £ i £ n/2 middle Search for A[,
with maximum sum. n/2+1, j] for
n/2+1 £ j £ n
max = -¥; sum = 0;
with maximum
for i from n/2 downto 1
sum.
sum = sum + A[i];
if sum > max
max = -¥; sum = 0;
max = sum; ans = i
for j from n/2+1 to n
return (ans, max)
sum = sum + A[i];
if sum > max
max = sum; ans = j
Combine these two results return (ans, max)
It takes O(n) time.
33
Max_subarray3(A, p, q) {
(i1, j1, max1) = Max_subarray3(A, p, p+q/2);
(i2, j2, max2) = Max_subarray3(A, p+q/2+1, q);
(i3, j3, max3) = Max_subarray_middle(A, p, q);
if max 1 > max 2
if max 1 > max 3
return (i1, j1, max1) O(n) time
else
return(i3, j3, max3)
else
if max2 > max3
return(i2, j2, max2)
else
return(i3, j3, max3)
}
Time complexity: T(n) = 2T(n/2) + cn for n > 1 (n = 2k)
=> T(n) = O(n log n)
Can we do better?
34
Exercises

1) Given an array of numbers, design a recursive

algorithm to find the average of all numbers in the array
and analyze the time complexity of your algorithm.

2) Design a recursive algorithm for reversing the entries

in an array and analyze the time complexity of your
algorithm. You are only allowed to use constant amount
of extra storage.

35
 Generating Combinatorial Objects

Design a recursive algorithm to generate all n-bit strings.

e.g. If n = 3, the algorithm should output all the followings:

000, 001, 010, 011, 100, 101, 110, 111

Hint: Assume the bits are stored in an array A[1..n]. Try

to generate the strings recursively by fixing the last bit
A[n].

Q1: Given a set of n objects, do you know how to

generate all its subsets?

Q2: Given a set of n objects, do you know how to

generate all its subsets with size <= x, where x <= n?
36