Design and Analysis of Algorithm

Tanveer Ahmed Siddiqui

Department of Computer Science

COMSATS University, Islamabad
Recap

Lecture No 7

Department of Computer Science 2

 Asymptotic Notation

O(g(n)) = {f(n) :  positive constants c and n0,

such that n  n0, we have 0  f(n)  cg(n) }

(g(n)) = {f(n) :  positive constants c and n0,

such that n  n0, we have 0  cg(n)  f(n)}

(g(n)) = {f(n) :  positive constants c1, c2, and

n0, such that n  n0, we have 0  c1g(n)  f(n) 
c2g(n) }

Department of Computer Science

 Asymptotic Notation

Department of Computer Science

 Analysis of Algorithm

Lecture No 8

Analysis Framework

Department of Computer Science

 Today Covered

After completing this lecture you should be able

to understand
Some Mathematical Background used in
Algorithm Analysis
Analysis Framework
 Measuring an Input’s size
 Identifying basic operation(s)
 Cases to be considered(Worst-case, Best Case, and
Average case efficiencies)
 Orders of Growth
Examples
Basic Efficiency classes
Department of Computer Science
 What to Analyze?

 Does Algorithm efficiency depend on only

input size?
 No, not always
 Choosing what input to consider when analyzing
an algorithm can have a significant impact on
how an algorithm will perform.
 For some algorithms efficiency depends on
form of input.

Department of Computer Science 7

 What to Analyze? Cases to Consider

 Consider the following algorithm for searching

problem

3,6,2,5,1,7,4 K =4
3,6,2,5,1,7,8 K =4
4,2,8,5,3,1,9 K=4
5,2,8,4,3,1,9 K=4
 Does Algorithm efficiency depend on only input
size?
Department of Computer Science 8
 What to Analyze?

 So, for those algorithm whose efficiency depend

on input form, we consider following cases.
 Best-case Complexity
 Worst-case Complexity
 Average-case Complexity

Department of Computer Science 9

 Best-case Time complexity
 As its name indicates, the best case for an
algorithm is the input that requires the algorithm
to take the minimum time.
 If we are looking at a searching algorithm, the
best case would be if the value we are searching
for was the value stored in the first location that
the search algorithm would check.
 This would then require only one comparison no
matter how complex the algorithm is.
 Since the best case for an algorithm will usually
be a very small and frequently constant value, we
will not do a best-case analysis very frequently.
Department of Computer Science
 Worst-case Time complexity
 Worst-case analysis requires that we identify the
input values that cause an algorithm to do the
most work.
 For searching algorithms, the worst case is one
where the value is in the last place we check or
is not in the list.
 This could involve comparing the key to each list
value for a total of N comparisons.
 The worst case gives us an upper bound on how
slowly parts of our programs may work based
on our algorithm choices.

Department of Computer Science

 Average-case Time complexity
 Average of the running times of all possible
inputs.
 NOT the average of worst and best case
 Average-case analysis is the toughest(Why)
 Demands a definition of probability of each input, which
is usually difficult to provide and to analyze.
 How to find the average case efficiency?

Department of Computer Science

 Average-case Time complexity
 How to find the average case efficiency?
 The basic process begins by determining the
number of different groups into which all
possible input sets can be divided.
 The second step is to determine the probability
that the input will come from each of these
groups.
 The third step is to determine how long the
algorithm will run for each of these groups.
 All of the input in each group should take the
same amount of time, and if they do not, the
group must be split into two separate groups.
Department of Computer Science
 Average-case Time complexity
 When all of this has been done, the average
case time is given by the following formula:

where
 n is the size of the input,
 m is the number of groups,

 pi is the probability that the input will be from

group i,
 ti is the time that the algorithm takes for input
from group i.
Department of Computer Science
 Average case of sequential Search

 The standard assumptions are:

 The probability of a successful search is equal to p (0
≤ p ≤ 1)
 The probability of the first match occurring in the ith
position of the list is the same for every i
Department of Computer Science 15
 Average case of sequential Search

 We can find the average number of key

comparisons Cavg(n) as follows.
 Case of a successful search:
 The probability of the first match occurring in the ith
position of the list is p/n for every i,
 The number of comparisons made by the algorithm
in such a situation is obviously i.

Department of Computer Science 16

 Average case of sequential Search

 Case of an unsuccessful search:

 The number of comparisons will be n with the
probability of such a search being (1− p).

 Therefore

Department of Computer Science 17

 Mathematical Background
 Floor and Ceiling Formulas

 Logarithms
 Since logarithms will play an important role in
our analysis, there are a few properties that
must be discussed.
 The logarithm base y of a number x is the
power of y that will produce the number x.

Department of Computer Science

 Mathematical Background
 Logarithms

Department of Computer Science

 Some useful ssummation formulas

 Sum Manipulation Rules

How to map the loop to a mathematical summation formula:

Call the iterator variable of a loop simple if it increases by one.
A loop will be called simple if its iterator variable is simple. In its simplest
form, the simple for loop like

is mapped to the summation

Department of Computer Science
 Some useful ssummation formulas

 Thus, a simple loop can be mapped to a

mathematical summation formula as follows:
 Use the iterator variable in the loop as the
summation index.
 Use the starting value of the iterator as the lower
limit and the last value of the iterator as the upper
limit of the summation formula.
 Each nested loop is mapped to a nested summation.

is mapped to the summation

Department of Computer Science

 Some useful ssummation formulas

 Important Summation Formulas

Department of Computer Science

 Probability
 The probability that something will occur is
given as a number in the range of 0 to 1, where
0 means it will never occur and 1 means it will
always occur.
 Bernoulli trial (or binomial trial)
 A Bernoulli trial is a random experiment that
results in two outcomes: success and failure.
 One example of a Bernoulli trial in searching
algorithm is that the key is found, or key is not
found.

Department of Computer Science

 Methodology
Mathematical Analysis of iterative Algorithms

Decide on parameter n indicating input size

Identify algorithm’s basic operation

Determine worst, average, and best case

for input of size n

Set up summation for C(n) reflecting

algorithm’s loop structure

Simplify summation using standard

formulas

Determine Orders of growth of

Department of Computer Science 24
C(n)
 Examples

Department of Computer Science

 Example of BF
Algorithm
with Constant
complexity

Department of Computer Science

 Example 1:
 Problem: Exchange the values of two variables
say x and y

 What is its time complexity?

 T(n) = c1+c2+c3
 = O(1)

Department of Computer Science

 Example of BF
Algorithm
with
logarithmic
complexity

Department of Computer Science

 Example 2:

 Use most appropriate

notation among O, Ω,
and Ɵ to compute the
time complexity of
above algorithm
 What is measure of an input’s size?
 The length of number.
 What is its basic operation/Primitive Operation?
 Division(/)
 Whether the basic operation count can be different for
inputs of the same size.
 No
Department of Computer Science
 Example 2:

Use most appropriate


notation among O, Ω,
and Ɵ to compute the
time complexity of
above algorithm
 Setup the summation? How many times is the
basic operation executed?

Department of Computer Science

 Example 3:

 What is measure of an input’s size?

 The length of array.
 What is its basic operation/Primitive Operation?
 We will assume that each three-way comparison (if-then-else)
counts as one comparison.
 Whether the basic operation count can be different for inputs of
the same size.
 Yes
Department of Computer Science
 Example 3:

 When does its best case occur?

 When element found at middle location.
 When does its worst case occur? at is its basic
operation/Primitive Operation?
 When element found at extreme location, i.e. either
first position or last position; or not exist in array.
 Thus, to find the maximum number of comparisons, we may
assume without loss of generality that x is greater than or equal
Department of Computer Science
to A[n] or x is less than or equal to A[0].
 Example 3:

Case 1: x = a[mid]
x = 10
low = 0, high = 8
mid = (0 + 8) / 2 = 4

a: 1 5 7 9 10 13 17 19 27
0 1 2 3 4 5 6 7 8

low mid high

Department of Computer Science 33

 Example 3:

Case 2: x > a[mid]

x = 19
low = 0, high = 8
mid = (0 + 8) / 2 = 4
new low = mid+1 = 5

a: 1 5 7 9 10 13 17 19 27
0 1 2 3 4 5 6 7 8

low new high

mid
low

Department of Computer Science 34

 Example 3:

Case 3: x < a[mid]

x=7
low = 0, high = 8
mid = (0 + 8) / 2 = 4
new high = mid-1 = 3

a: 1 5 7 9 10 13 17 19 27
0 1 2 3 4 5 6 7 8

low new mid high

high
Department of Computer Science 35
 Example 3

val = 7

a: 1 5 7 9 10 13 17 19 27
0 1 2 3 4 5 6 7 8

a: 1 5 7 9 10 13 17 19 27
0 1 2 3 4 5 6 7 8

a: 1 5 7 9 10 13 17 19 27
0 1 2 3 4 5 6 7 8
Department of Computer Science 36
 Binary Search: Binary Tree

An entire sorted list

First half Second half

First half Second half First half

First half First half First half

 The search divides a list into two small sub-lists

till a sub-list is no more divisible.

Department of Computer Science 37

 Binary Search: Binary Tree

Suppose that we want to search for x = 35 or x

= 100 in
In each iteration of the algorithm, the bottom
half of the array is discarded until there is only
one element:

Department of Computer Science 38

 Binary Search: Binary Tree

the maximum number of comparisons in both

trees is 4.
In general, the maximum number of
comparisons is 1 plus the height of the
corresponding decision tree
Department of Computer Science 39
 Binary search: Efficiency

 After 1 bisection N/2 items

 After 2 bisectionsN/4 = N/22 items
 . . .
 After i bisectionsN/2i =1 item

i = log2 N

Department of Computer Science 40

 Example 3:

 Setup the summation? How many times is the

basic operation executed?

Department of Computer Science

 Example of BF
Algorithm
with linear
time
complexity

Department of Computer Science

 Example 4:

 Use most appropriate

notation among O, Ω,
and Ɵ to compute the
time complexity of
above algorithm
 What is measure of an input’s size?
 The length of numbers(i.e no. of digits in a number)
 What is its basic operation/Primitive Operation?
 Multiplication(*)
 Whether the basic operation count can be different for inputs of the
same size.
 No

Department of Computer Science

 Example 4:

 Use most appropriate

notation among O, Ω,
and Ɵ to compute the
time complexity of
above algorithm
 Setup the summation
 How many times is the basic operation executed?

Department of Computer Science

 Example 5:

 Use most appropriate

notation among O, Ω,
and Ɵ to compute the
time complexity of
above algorithm
 What is measure of an input’s size?
 The length of given number
 What is its basic operation/Primitive Operation?
 Multiplication(*)
 Whether the basic operation count can be different for inputs of the
same size.
 No

Department of Computer Science

 Example 5:

Use most appropriate


notation among O, Ω,
and Ɵ to compute the
time complexity of
above algorithm
 Setup the summation? How many times is the
basic operation executed?

Department of Computer Science

 Example 6:

 Use most appropriate

notation among O, Ω,
and Ɵ to compute the
time complexity of
above algorithm
 What is measure of an input’s size?
 The length of numbers.
 What is its basic operation/Primitive Operation?
 Addition(+)
 Whether the basic operation count can be different for inputs of the
same size.
 No

Department of Computer Science

 Example 6:

Use most appropriate


notation among O, Ω,
and Ɵ to compute the
time complexity of
above algorithm
 Setup the summation? How many times is the
basic operation executed?

Department of Computer Science

 Example 7:

 Use most appropriate

notation among O, Ω,
and Ɵ to compute the
time complexity of
above algorithm
 What is measure of an input’s size?
 The length of number.
 What is its basic operation/Primitive Operation?
 Addition(+)
 Whether the basic operation count can be different for inputs of the
same size.
 No

Department of Computer Science

 Example 7:

Use most appropriate


notation among O, Ω,
and Ɵ to compute the
time complexity of
above algorithm
 Setup the summation? How many times is the
basic operation executed?

Department of Computer Science

 Example 8:

 Use most appropriate

notation among O, Ω,
and Ɵ to compute the
time complexity of
above algorithm
 What is measure of an input’s size?
 The number of elements in the array, i.e., n.
 What is its basic operation/Primitive Operation?
 Comparison(>)
 Whether the basic operation count can be different for
inputs of the same size.
 No
Department of Computer Science
 Example 8:

Use most appropriate


notation among O, Ω,
and Ɵ to compute the
time complexity of
above algorithm
 Setup the summation? How many times is the
basic operation executed?

Department of Computer Science

 Example of BF
Algorithm
with
Quadratic
time
complexity

Department of Computer Science

 Example 9:

Input size: n
Basic operation: A[i] = A[j]
Does C(n) depend on type of input?
Department of Computer Science 54
 Example 9:

Department of Computer Science 55

 Example 9: Alternatively

Department of Computer Science 56

 Example
Brute Force 10:
Examples

Basic Operation

Department of Computer Science

 Example of BF
Algorithm
with Cubic
time
complexity

Department of Computer Science

 Example 11:

Department of Computer Science 59

 Classification of Algorithms
based on
Complexity Classes

Department of Computer Science

 Classes of Algorithms

Department of Computer Science 61

 Classes of Algorithms

 Factorial
 kcn < n! < nn
where k is some
constant, and c is a
constant less than n

Department of Computer Science 62

 Basic Efficiency Classes

 O(1)
 Time complexity of a function (or set of
statements) is considered as O(1) if it doesn’t
contain loop, recursion and call to any other
non-constant time function. For example
swap() function has O(1) time complexity.

T(n) = c1+c2+c3
= O(1)

Department of Computer Science 63

 Basic Efficiency Classes

 O(loglogn)
 Time Complexity of a loop is considered as
O(loglogn) if the loop variables is
reduced/increased exponentially by a constant.

Department of Computer Science 64

 Basic Efficiency Classes

 O(loglogn)
 Time Complexity of a loop is considered as
O(loglogn) if the loop variables is
reduced/increased exponentially by a constant.

Department of Computer Science 65

 Basic Efficiency Classes

 O(logn)
 Time Complexity of a loop is considered as
O(logn) if the loop variables is
divided/multiplied by a constant amount.

Department of Computer Science 66

 Basic Efficiency Classes

 O(n)
 Time Complexity of a loop is considered as
O(n) if the loop variables is
incremented/decremented by a constant
amount.

Department of Computer Science 67

 Basic Efficiency Classes

 O(n)
 Time Complexity of a loop is considered as
O(n) if the loop variables is
incremented/decremented by a constant
amount.

Department of Computer Science 68

 Basic Efficiency Classes

 O(nc)
 Time complexity of nested loops is equal to the
number of times the inner most statement is
executed.
 O(n2)

Department of Computer Science 69

 Basic Efficiency Classes

 O(nc)
 Time complexity of nested loops is equal to the
number of times the inner most statement is
executed.
 O(n2)

Department of Computer Science 70

 How to Apply theorems?

 How to combine time complexity of

consecutive loops?

 Time complexity of above code is O(n)+O(m)

which is O(n+m)
Department of Computer Science 71
 How to Apply theorems?

 If t1(n) є O(g1(n)) and t2(n) є O(g2(n)), then

t1(n) + t2(n) є O(max{g1(n),g2(n)})

T(n) = {max(log2n, n)} = O(n)

O(1)

Department of Computer Science

 How to Apply theorems: Theorem-
1
 Suppose an algorithm is consisting of two sub
algorithms A1 and A2. Suppose the orders of
execution times of the A1 and A2 are:
 A1: f1(n) = O(g1(n)) ; A2: f2(n) = O(g2(n)) , then
 (f1+ f2)(n) = O(max{(g1(n) , (g2(n) }
O(N)

Department of Computer Science

 Theorem- 1

A1

O(N)
A2

O(1)

Department of Computer Science

 Theorem- 1

 If f1(n) є O(g1(n)) and f2(n) є O(g2(n)), then

f1(n) + f2(n) є O(max{g1(n),g2(n)})
if (i<j)
for ( i=0; i<N; i++ ) O(N)
X = X+i;
else
X=0; O(1)
Max ( O(N), O(1) ) = O (N)

Department of Computer Science

 Theorem- 2

 Let f1(n) = O(g1(n) ; f2(n) = O(g2(n) , then

 (f1* f2)(n) = O((g1(n)* g2(n))

Algorithm Example()
while(counter < n)
{
if(n%2 == 0)
{ O(N)
binarySearch(preSortedArray, counter);
}
}

Department of Computer Science

 Order of growth

The time efficiencies of a large number of algorithms fall into only a few
classes.

Department of Computer Science 77

 Order of growth

O(1): Time requirement is constant, and it is independent of the problem’s

size.
O(log2n):Time requirement for a logarithmic algorithm increases slowly as
the problem size increases.
O(n):Time requirement for a linear algorithm increases directly with the size
of the problem.
O(n*log2n): Time requirement for a n*log2n algorithm increases more
rapidly than a linear algorithm.
O(n2): Time requirement for a quadratic algorithm increases rapidly with
the size of the problem.
O(n3):Time requirement for a cubic algorithm increases more rapidly with
the size of the problem than the time requirement for a quadratic
algorithm.
O(2n) As the size of the problem increases, the time requirement for an
exponential algorithm increases too rapidly to be practical.

Department of Computer Science 78

 Order of growth

Department of Computer Science 79

 Order of growth

Department of Computer Science 80

 Limitation of Asymptotic Analysis

 Does Asymptotic Analysis always work?

 Asymptotic Analysis is not perfect, but that’s
the best way available for analyzing algorithms.
 For example, say there are two sorting algorithms
that take 1000nlogn and 2nlogn time respectively
on a machine.
 Both of these algorithms are asymptotically the
same (order of growth is nlogn).
 So, With Asymptotic Analysis, we can’t judge
which one is better as we ignore constants in
Asymptotic Analysis.

Department of Computer Science 81

 Another
Brute ForceExample
Examples

Basic Operation
n
T ( n)   2  2n  O ( n)
i 1

Basic Operation
n
T ( n)   1  n  O ( n)
i 1

Department of Computer Science

 Limitation of Asymptotic Analysis

 Does Asymptotic Analysis always work?

With Asymptotic Analysis, we can’t judge which

one is better as we ignore constants in
Asymptotic Analysis.
Department of Computer Science 83
 Limitation of Asymptotic Analysis

 Does Asymptotic Analysis always work?

 Also, in Asymptotic analysis, we always talk
about input sizes larger than a constant value.
 It might be possible that those large inputs are
never given to your software and an asymptotically
slower algorithm always performs better for your
particular situation.
 So, you may end up choosing an algorithm that
is Asymptotically slower but faster for your
software.

Department of Computer Science 84

 Advantages of Asymptotic Analysis

 Asymptotic analysis provides a high-level

understanding of how an algorithm performs
with respect to input size.
 It is a useful tool for comparing the efficiency of
different algorithms and selecting the best one
for a specific problem.
 It helps in predicting how an algorithm will
perform on larger input sizes, which is essential
for real-world applications.
 Asymptotic analysis is relatively easy to perform
and requires only basic mathematical skills.
Department of Computer Science 85
 Disadvantages of Asymptotic Analysis

 Asymptotic analysis does not provide an accurate

running time or space usage of an algorithm.
 It assumes that the input size is the only factor that
affects an algorithm’s performance, which is not
always the case in practice.
 Asymptotic analysis can sometimes be misleading,
as two algorithms with the same asymptotic
complexity may have different actual running times
or space usage.
 It is not always straightforward to determine the
best asymptotic complexity for an algorithm, as
there may be trade-offs between time and space
complexity.
Department of Computer Science 86
 Conclusion

Department of Computer Science

 Mathematical Analysis of Iterative Algorithms

Steps in mathematical analysis of non recursive

algorithms

 Decide on parameter n indicating input size

 Identify algorithm’s basic operation
 Determine worst, average, and best case for input of
size n
 Set up summation for C(n) reflecting algorithm’s loop
structure
 Simplify summation using standard formulas
 Determine Orders of growth of C(n)

Department of Computer Science 88

 Measuring Input Sizes

 So, efficiency is define as a function of input size.

 Input size depends on the problem.
Problem Input size measure
Searching for key in a list of n
Number of list’s items, i.e. n
items
Matrix dimensions or total number of
Multiplication of two matrices
elements
Checking primality of a given n’size = number of digits (in binary
integer n representation)
Typical graph problem #vertices and/or edges

Polynomial Evaluation Order of the polynomial

Department of Computer Science 89

 Units for Measuring Running Time

 Basic operation/Primitive operation : the most

important operation of the algorithm, the
operation contributing the most to the total
running time.
Problem Basic operation

Searching for key in a list of n items Key comparison

Multiplication of two
Multiplication of two matrices
numbers

Checking primality of a given integer n Division

Visiting a vertex or
Typical graph problem
traversing an edge

Polynomial Evaluation multiplication

Department of Computer Science 90
 Best-case, average-case, worst-
case
For some algorithms efficiency depends on form of
input:
 Worst case: Cworst(n) – maximum over inputs of size n
 Best case: Cbest(n) – minimum over inputs of size n
 Average case: Cavg(n) – “average” over inputs of size n
 Number of times the basic operation will be executed on
typical input
 NOT the average of worst and best case
 Expected number of basic operations considered as a
random variable under some assumption about the
probability distribution of all possible inputs

Department of Computer Science