Algorithms

Week 1
AIUB
 Course Objective

• Analyze the asymptotic performance of algorithms.

• Demonstrate a familiarity with major algorithms and data structures.
• Apply important algorithmic design paradigms and methods of analysis.
• Synthesize efficient algorithms in common engineering design
situations.
 Prerequisite
• Good skill in C/C++ programming and Data structure
• List
• Array
• Link list
• String
• Stack
• Queue
• Graph
• Tree
 Pre-Requisite in C++
• 1. Array / Dynamic Array
• 2. Function
• 3. Pointer
• 4. Dynamic Array - 2D/ Matrix
• 5. Random number
• 6. Standard Template Library(STL)
 Warm up problem
• 1. Fibonacci
• 2. Factorial
• 3. Prime number
• 4. Euclid’s algorithm for GCD
• 6. Find second largest number in a List
• 7. Linear Search
• 8. Bubble sort
 WHAT IS AN ALGORITHM?

An algorithm is a sequence of unambiguous instructions for solving a

problem, i.e., for obtaining a required output for any legitimate input in
a finite amount of time.
NOTION OF ALGORITHM
 Properties of an Algorithm
• Input: A number of quantities are provided to an algorithm initially before the
algorithm begins. These quantities are inputs which are processed by the algorithm.
• Definiteness: Each step must be clear and unambiguous.
• Effectiveness: Each step must be carried out in finite time.
• Finiteness: Algorithms must terminate after finite time or step
• Output: An algorithm must have output.
• Correctness: Correct set of output values must be produced from the each set of
inputs.
Example of an Algorithm
 WHY STUDY ALGORITHMS

• Theoretical importance
• The core of computer science
•Practical importance
• A practitioner’s toolkit of known algorithms
• Framework for designing and analyzing algorithms for new
problems
 ANALYSIS OF ALGORITHMS

• How good is the algorithm?

• Correctness
• Time efficiency
• Space efficiency
• Does there exist a better algorithm?
• Lower bounds
• Optimality
SOME WELL-KNOWN COMPUTATIONAL PROBLEMS

•Sorting
•Searching
•Shortest paths in a graph
•Minimum spanning tree
•Traveling salesman problem
•Knapsack problem
•Chess
•Towers of Hanoi
•Program termination
 ALGORITHM DESIGN STRATEGIES

•Brute force
•Divide and conquer
•Decrease and conquer
•Transform and conquer
•Greedy approach
•Dynamic programming
•Backtracking and Branch and bound
•Space and time tradeoffs
 ANALYSIS OF ALGORITHMS

• How good is the algorithm?

• Correctness
• Time efficiency
• Space efficiency
• Does there exist a better algorithm?
• Lower bounds
• Optimality
 Function
• Following is the source code for a function called max(). This
function takes two parameters num1 and num2 and returns the
maximum between the two:
 Function Declarations
• A function declaration(function prototype) tells the compiler about a function name
and how to call the function. The actual body of the function can be defined separately.
• A function declaration has the following parts:

• For the above defined function max(), following is the function declaration:

• Parameter names are not important in function declaration only their type is required,
so following is also valid declaration:

• Function declaration is required when you define a function in one source file and you call
that function in another file. In such case you should declare the function at the top of the file
calling the function.
Calling a Function
Code Example: Calling a Function
 RECURSIVELY DEFINED FUNCTIONS
o A function is said to be recursively defined if the function
refers to itself such that
1. There are certain arguments, called base values, for which
the function does not refer to itself.
2. Each time the function does refer to itself, the argument
of the function must be closer to a base value.
 Recursion

• RECURSION
The process of defining an object in terms of smaller versions of
itself is called recursion.
• A recursive definition has two parts:
1. BASE:
An initial simple definition which cannot be expressed in terms of
smaller versions of itself.
2. RECURSION:
The part of definition which can be expressed in terms of smaller
versions of itself.
THE FACTORIAL OF A POSITIVE INTEGER: Example

o For each positive integer n, the factorial of n denoted as n! is defined

to be the product of all the integers from 1 to n:
n! = n.(n - 1).(n - 2) . . .3 . 2 . 1
• Zero factorial is defined to be 1
0! = 1
EXAMPLE:0! = 1 1! = 1
2! = 2.1 = 2 3! = 3.2.1 = 6
4! = 4.3.2.1 = 24 5! = 5.4.3.2.1 = 120
! = 6.5.4.3.2.1= 720 7! = 7.6.5.4.3.2.1= 5040
REMARK:
5! = 5 .4.3 . 2 . 1 = 5 .(4 . 3 . 2 .1) = 5 . 4!
THE FACTORIAL OF A POSITIVE INTEGER

Thus, the recursive definition of

factorial function F(n) is:
1. F(0) = 1
2. F(n) = n * F(n-1)
THE FACTORIAL OF A POSITIVE INTEGER
 The Fibonacci numbers
o f(0) = 0, f(1) = 1
o f(n) = f(n – 1) + f(n - 2)
f(0) = 0
f(1) = 1
f(2) = f(1) + f(0) = 1 + 0 = 1
f(3) = f(2) + f(1) = 1 + 1 = 2
f(4) = f(3) + f(2) = 2 + 1 = 3
f(5) = f(4) + f(3) = 3 + 2 = 5
f(6) = f(5) + f(4) = 5 + 3 = 8
 The Fibonacci numbers
• Following is an example, which calculates Fibonacci for a given number
using a recursive function:
RECURSIVELY DEFINED FUNCTIONS
procedure fibo(n: nonnegative integer)
if n = 0 then fibo(0) := 0
else if n = 1 then fibo(1) := 1
else fibo(n) := fibo(n – 1) + fibo(n – 2) f(4)
end
f(3) f(2)

f(2)
f(1) f(1) f(0)

f(1) f(0)
 USE OF RECURSION
o At first recursion may seem hard or impossible, may be
magical at best. However, recursion often provides elegant,
short algorithmic solutions to many problems in computer
science and mathematics.
– Examples where recursion is often used
• math functions
• number sequences
• data structure definitions
• data structure manipulations
• language definitions
 USE OF RECURSION
o For every recursive algorithm, there is an
equivalent iterative algorithm.
o However, iterative algorithms are usually more
efficient in their use of space and time.
 GOOD ALGORITHMS?

Run in less time

Consume less memory

But computational resources (time complexity) is usually

more important
 MEASURING EFFICIENCY
The efficiency of an algorithm is a measure of the amount of
resources consumed in solving a problem of size n.
The resource we are most interested in is time
We can use the same techniques to analyze the consumption of other
resources, such as memory space.

It would seem that the most obvious way to measure the

efficiency of an algorithm is to run it and measure how much
processor time is needed

Is it correct ?
 FACTORS
Hardware

Operating System

Compiler

Size of input

Nature of Input

Algorithm

Which should be improved?

 RUNNING TIME OF AN ALGORITHM

Depends upon
Input Size
Nature of Input

Generally time grows with size of input, so running time of an

algorithm is usually measured as function of input size.

Running time is measured in terms of number of

steps/primitive operations performed

Independent from machine, OS

FINDING RUNNING TIME OF AN
ALGORITHM / ANALYZING AN
ALGORITHM
Running time is measured by number of steps/primitive
operations performed

Steps means elementary operation like

,+, *,<, =, A[i] etc

We will measure number of steps taken in term of size of

input
Asymptotic Notation
Complexity

Introduction34
Analysis of Algorithms
• Efficiency:
• Running time
• Space used
• Efficiency as a function of the input size:
• Number of data elements (numbers, points).
• The number of bits of an input number .

Introduction35
The RAM Model
• RAM model represents a “generic” implementation of the
algorithm
• Each “simple” operation (+, -, =, if, call) takes exactly 1 step.
• Loops and subroutine calls are not simple operations, but
depend upon the size of the data and the contents of a
subroutine. We do not want “sort” to be a single step
operation.
• Each memory access takes exactly 1 step.
The RAM model (cntd..)
• It is important to choose the level of detail.
• The RAM model:
• Instructions (each taking constant time), we usually choose one type of
instruction as a characteristic operation that is counted:
• Arithmetic (add, subtract, multiply, etc.)
• Data movement (assign)
• Control flow (branch, subroutine call, return)
• Comparison (logical ops)
• Data types – integers, characters, and floats

Introduction37
Example
Algorithm arrayMax(A, n)
# operations
currentMax  A[0] 2
for (i =1; i<n; i++) 2n
(i=1 once, i<n n times, i++ (n-1) times)
if A[i]  currentMax then 2(n  1)
currentMax  A[i] 2(n  1)
return currentMax 1
Total 6n 1

Introduction38
 Example: N-by-N matrix, N-by-1 vector, multiply
(3 accesses, 1 add, 1 multiply)

Y = zeros(N,1); initialize space, c1N

for i=1:N initialize “for” loop, c2N
Scalar assignment, c3
Y(i) = 0.0;
initialize “for” loop, c2N
for j=1:N

N times
c4

N times
Y(i) = Y(i) + A(i,j)*x(j);
end End of loop, return/exit, c5
end End of loop, return/exit, c5

Total = c1N+c2N+N(c3+c2N+N(c4+c5)+c5)
= (c2+c4+c5)N2 + (c1+c2+c3+c5)N
= c6N2 + c7N
Introduction39
 Time complexity familiar tasks
Task Growth rate
Matrix/vector multiply O(N2)
Getting a specific element from a list O(1)
Dividing a list in half, dividing one halve in half, etc O(log2N)
Binary Search O(log2N)
Scanning (brute force search) a list
O(N)
Nested for loops (k levels)
O(Nk)
MergeSort
O(N log2N)
BubbleSort
O(N2)
Generate all subsets of a set of data
O(2N)
Generate all permutations of a set of data
O(N!)

Introduction40
Best/ Worst/ Average Case
• Best case: works fast on some input.
• Worst case: (usually) maximum time of algorithm on any input of size.
• Average case: (sometimes) expected time of algorithm over all inputs of size. Need assumption
of statistical distribution of inputs.

• Analyzing insertion sort’s

• Best case: elements already sorted, tj=1, running time » (n-1), i.e., linear time.
• Worst case: elements are sorted in inverse order, tj = j-1, running time » (n2-n)/2 , i.e., quadratic time.
• Average case: tj = j / 2, running time » (n2+n-2)/4 , i.e., quadratic time.

Introduction41
 …Best/ Worst/
• For inputs of all sizes:
Average Case

worst-case

6n average-case
Running time

5n
best-case
4n

3n

2n

1n

1 2 3 4 5 6 7 8 9 10 11 12 …..
Input instance size

Introduction42
 …Best/ Worst/ Average Case
• Worst case is usually used:
• It is an upper-bound.
• In certain application domains (e.g., air traffic control, surgery) knowing the worst-case
time complexity is of crucial importance.
• For some algorithms worst case occurs fairly often
• The average case is often as bad as the worst case.
• Finding the average case can be very difficult.

Introduction43
 Asymptotic Notation
• Asymptotic Notations are languages that allow us to analyze an algorithm's
running time by identifying its behavior as the input size for the algorithm
increases.
• This is also known as an algorithm's growth rate.(Wiki)

Introduction44
 Asymptotic Notation
• The “big-Oh” O-Notation
• asymptotic upper bound
• f(n) = O(g(n)), if there exists constants c>0 and n0>0, s.t. f(n) £ c * g(n) for n ³ n0
• f(n) and g(n) are functions
over non- negative integers c g ( n )
f (n )
• Used for worst-case analysis

Running Time
• F(n)=2n^2+n+3
• G(n)=n^2 n0 Input Size

Introduction45
Asymptotic Notation

Introduction46
 ...Asymptotic Notation
• The “big-Omega” W-Notation
• asymptotic lower bound

• f(n) = W(g(n)) if there exists constants c>0 and n0>0, s.t. c g(n) £ f(n) for n ³ n0

• Used to describe best-case running times or lower bounds of algorithmic problems.

E.g., lower-bound of searching in an unsorted array is W(n).

f (n )

Running Time
c g ( n )

n0 Input Size

Introduction47
 ...Asymptotic Notation
• The “big-Theta” Q-Notation
• asymptoticly tight bound

• f(n) = Q(g(n)) if there exists constants c1>0, c2>0, and n0>0, s.t. for n ³ n0 c1 g(n) £ f(n) £ c2 g(n)

• f(n) = Q(g(n)) if and only if f(n) = O(g(n)), f(n) = W(g(n))

c 2 g (n )
• O(f(n)) is often abused instead of Q(f(n)) f (n )

Running Time
c1 g (n )

n0 Input Size

Introduction48
Asymptotic Analysis
• Goal: to simplify the analysis of the running time by getting rid of
details, which are affected by specific implementation and hardware
• rounding of numbers: 1,000,001 » 1,000,000
• rounding of functions: 3n2 » n2

• Capturing the essence: how the running time of an algorithm

increases with the size of the input in the limit.
• Asymptotically more efficient algorithms are best for all but small inputs

Introduction49
...Asymptotic Analysis
• Simple Rule: Drop lower order terms and constant factors.
• 50 n log n is O(n log n)
• 7n - 3 is O(n)
• 8n2 log n + 5n2 + n is O(n2 log n)

• Note: Although (50 n log n) is O(n5), it is expected that an

approximation is of the smallest possible order.

Introduction50
 Correctness of Algorithms
• An algorithm is correct if for any legal input it terminates and produces the
desired output.
• Automatic proof of correctness is not possible (so far).

• There are practical techniques and rigorous formalisms that help to reason about
the correctness of (parts of) algorithms.

Introduction51
Pre/post-conditions

• Example:
• Write a pseudocode algorithm to find the two smallest numbers in a sequence of
numbers (given as an array).

• INPUT: an array of integers A[1..n], n > 0

• OUTPUT: (m1, m2) such that

• m1<m2 and for each iÎ[1..n]: m1 £ A[i] and, if A[i] ¹ m1, then m2 £ A[i].

• m2 = m1 = A[1] if "j,iÎ[1..n]: A[i]=A[j]

Introduction52
 Loop Invariants
• Invariants: assertions that are valid any time they are reached (many times during the
execution of an algorithm, e.g., in loops)

• We must show three things about loop invariants:

• Initialization: it is true prior to the first iteration.

• Maintenance: if it is true before an iteration, then it is true after the iteration.

• Termination: when a loop terminates the invariant gives a useful property to show the correctness of the
algorithm

Introduction53
 SIMPLE EXAMPLE (1)

// Input: int A[N], array of N integers

// Output: Sum of all numbers in array A

int Sum(int A[], int N)

{
int s=0;
for (int i=0; i< N; i++)
s = s + A[i];
return s;
}

How should we analyse this?

 SIMPLE EXAMPLE (2)

// Input: int A[N], array of N integers

// Output: Sum of all numbers in array A

int Sum(int A[], int N){

int s=0; 1
for (int i=0; i< N; i++)
2 3 4
s = s + A[i];
5
6 7
return s;
} 1,2,8: Once
8
3,4,5,6,7: Once per each iteration
of for loop, N iteration
Total: 5N + 3
The complexity function of the 9
algorithm is : f(N) = 5N +3
 SIMPLE EXAMPLE (3) GROWTH
OF 5N+3
Estimated running time for different values of N:

N = 10 => 53 steps
N = 100 => 503 steps
N = 1,000 => 5003 steps
N = 1,000,000 => 5,000,003 steps

As N grows, the number of steps grow in linear proportion to N for

this function “Sum”
 WHAT DOMINATES IN PREVIOUS
EXAMPLE?

What about the +3 and 5 in 5N+3?

As N gets large, the +3 becomes insignificant
5 is inaccurate, as different operations require varying amounts of time and
also does not have any significant importance

What is fundamental is that the time is linear in N.

Asymptotic Complexity: As N gets large, concentrate on the

highest order term:
 Drop lower order terms such as +3
 Drop the constant coefficient of the highest order term i.e. N
WHICH NOTATION DO WE USE?
To express the efficiency of our algorithms which of the
three notations should we use?

As computer scientist we generally like to express our

algorithms as big O since we would like to know the
upper bounds of our algorithms.

Why?

If we know the worse case then we can aim to improve it

and/or avoid it.
 COMPLEXITY CLASSES

t s( e mi T
)spe

26
wth Rates and Dominance Relations

Introduction60
Complexity Analysis
Time complexity
Time complexity
Time complexity
Time complexity
Time complexity
Time complexity
 STANDARD ANALYSIS
TECHNIQUES

Constant time statements

Analyzing Loops

Analyzing Nested Loops

Analyzing Sequence of Statements

Analyzing Conditional Statements

CONSTANT TIME STATEMENTS

Simplest case: O(1) time statements

Assignment statements of simple data types

int x = y;

Arithmetic operations:
x = 5 * y + 4 - z;

Array referencing:
A[j] = 5;

Array assignment:
∀ j, A[j] = 5;

Most conditional tests:

if (x < 12) ...
 ANALYZING LOOPS[1]
Any loop has two parts:
How many iterations are performed?
How many steps per iteration?

int sum = 0,j;

for (j=0; j < N; j++)
sum = sum +j;

Loop executes N times (0..N-1)

4 = O(1) steps per iteration

Total time is N * O(1) = O(N*1) = O(N)

 ANALYZING LOOPS[2]

What about this for loop?

int sum =0, j;
for (j=0; j < 100; j++)
sum = sum +j;

Loop executes 100 times

4 = O(1) steps per iteration

Total time is 100 * O(1) = O(100 * 1) = O(100) = O(1)

 ANALYZING LOOPS – LINEAR
LOOPS

Example (have a look at this code segment):

Efficiency is proportional to the number of iterations.

Efficiency time function is :
f(n) = 1 + (n-1) + c*(n-1) +( n-1)
= (c+2)*(n-1) + 1
= (c+2)n – (c+2) +1
Asymptotically, efficiency is : O(n)
 ANALYZING NESTED LOOPS[1]
Treat just like a single loop and evaluate each level of nesting as
needed:

int j,k;
for (j=0; j<N; j++)
for (k=N; k>0; k--)
sum += k+j;

Start with outer loop:

How many iterations? N
How much time per iteration? Need to evaluate inner loop

Inner loop uses O(N) time

Total time is N * O(N) = O(N*N) = O(N )

 ANALYZING NESTED LOOPS[2]
What if the number of iterations of one loop depends on the
counter of the other?

int j,k;
for (j=0; j < N; j++)
for (k=0; k < j; k++)
sum += k+j;

Analyze inner and outer loop together:

Number of iterations of the inner loop is:

0 + 1 + 2 + ... + (N-1) = O(N )
 HOW DID WE GET THIS
ANSWER?
When doing Big-O analysis, we sometimes have to compute a
series like: 1 + 2 + 3 + ... + (n-1) + n

i.e. Sum of first n numbers. What is the complexity of this?

Gauss figured out that the sum of the first n numbers is always:
 ANALYZING NESTED LOOPS[3]

int K=0;
for(int i=0; i<N; i++)
{
cout <<”Hello”;
for(int j=0; j<K; j--)
Sum++;
}
 SEQUENCE OF STATEMENTS
 For a sequence of statements, compute their complexity
functions individually and add them up

 Total cost is O(n 2) + O(n) +O(1) = O(n 2)

37
 CONDITIONAL STATEMENTS
What about conditional statements such as

if (condition)
statement1;
else
statement2;

where statement1 runs in O(n) time and statement2 runs in O(n2 )

time?

We use "worst case" complexity: among all inputs of size n, what is

the maximum running time?

The analysis for the example above is O(n2 )

 DERIVING A RECURRENCE
EQUATION
 So far, all algorithms that we have been analyzing have been non
recursive

 Example : Recursive power method

 If N = 1, then running time T(N) is 2

 However if N ≥ 2, then running time T(N) is the cost of each step taken plus time
required to compute power(x,n-1). (i.e. T(N) = 2+T(N-1) for N ≥ 2)
39

 How do we solve this? One way is to use the iteration method.

 ITERATION METHOD
This is sometimes known as “Back Substituting”.

Involves expanding the recurrence in order to see a pattern.

Solving formula from previous example using the iteration method

Solution : Expand and apply to itself :

Let T(1) = n0 = 2
T(N) = 2 + T(N-1)
= 2 + 2 + T(N-2)
= 2 + 2 + 2 + T(N-3)
= 2 + 2 + 2 + ……+ 2 + T(1)
= 2N + 2 remember that T(1) = n0 = 2 for N = 1

So T(N) = 2N+2 is O(N) for last example.

 SUMMARY
Algorithms can be classified according to their
complexity => O-Notation
only relevant for large input sizes

"Measurements" are machine independent

worst-, average-, best-case analysis
 Thank you
&
Any question?