TIC2001 Data Structures and Algorithms

Lecture Note #2a

Analysis of Algorithms
You are the boss of a logistic company and your company occupies the first of the shop
houses (n+1 of them) along this busy street. Every day you need to distribute some
packages to each of the other n shop houses. You ask two of your staff to design a plan
to complete the task.

The plan from the first staff:

Set j = 1
while j <=n do the following
take a package, deliver to shop j,
come back to company, increase j by 1

The plan from the second staff;

Take n packages, move sequentially to all n shops
And deliver 1 package to each shop.

Which plan do you approve?

2
 Objectives

• To introduce the theoretical basis for

1 measuring the efficiency of algorithms

• To learn how to use such measure to

compare the efficiency of different
2 algorithms

3
TIC2001 Lecture 2a AY23/24S1
 Outline
1. What is an Algorithm?
2. What do we mean by Analysis of
Algorithms?
3. Big-O notation – Upper Bound
4. How to find the complexity of a program?

4
TIC2001 Lecture 2a AY23/24S1
 1 What is an algorithm?

TIC2001 Lecture 2a AY23/24S1

1 Algorithm
 A step-by-step procedure for solving a problem.
 Properties of an algorithm:
 Each step of an algorithm must be exact.
 An algorithm must terminate.
 An algorithm must be effective.
 An algorithm should be general.

Exact Terminate

Effective General

6
 2 What do we mean by
Analysis of Algorithms?

TIC2001 Lecture 2a AY23/24S1

 2.1 What is Analysis of Algorithms?
 Analysis of algorithms
 Provides tools for comparing the efficiency of
different methods of solution (rather than programs)
 Efficiency = Complexity of algorithms

 A comparison of algorithms
 Should focus on significant differences in the
efficiency of the algorithms
 Should not consider reductions in computing costs
due to clever coding tricks. Tricks may reduce the
readability of an algorithm.

8
TIC2001 Lecture 2a AY23/24S1
 2.2 Determining the Efficiency/Complexity
of Algorithms
 To evaluate rigorously the resources (time and
space) needed by an algorithm and represent
the result of the analysis with a formula
 We will emphasize more on the time
requirement rather than space requirement here
 The time requirement of an algorithm is also
called its time complexity

9
TIC2001 Lecture 2a AY23/24S1
 2.3 By measuring the run time?
startTime = currentSystemTime;
total = 0;
for (int i = 0; i < 10000000; i++) {
total += i;
}
stopTime = currenSystemTime;
elapsedTime = stopTime - startTime;
output elapsedTime;

Note: The run time depends on the compiler, the computer

used, and the current work load of the computer.

10
TIC2001 Lecture 2a AY23/24S1
 2.4 Exact run time is not always needed
 Using exact run time is not meaningful
when we want to compare two algorithms
 coded in different languages,
 using different data sets, or
 running on different computers.

11
TIC2001 Lecture 2a AY23/24S1
 2.5 Determining the Efficiency of
Algorithms
 Algorithm analysis should be independent of
 Specific implementations
 Compilers and their optimizers

 Computers
 Data

12
TIC2001 Lecture 2a AY23/24S1
 2.6 Execution Time of Algorithms
 Instead of working out the exact timing, we count
the number of some or all of the primitive
operations (e.g. +, -, *, /, assignment, …)
needed.
 Counting an algorithm's operations is a way to
assess its efficiency
 An algorithm’s execution time is related to the number
of operations it requires.

13
TIC2001 Lecture 2a AY23/24S1
 2.7 Counting the number of statements
 To simplify the counting further, we can ignore
 the different types of operations, and
 different number of operations in a statement,
and simply count the number of statements
executed.

14
TIC2001 Lecture 2a AY23/24S1
 2.8 Computation cost of an algorithm
 How many operations are required?
for (int i=1; i<=n; i++) {
perform 100 operations; // A
for (int j=1; j<=n; j++) {
perform 2 operations; // B
}
}
n n n
Total Ops = A + B  100   ( 2)
i 1 i 1 j 1

n
 100n   2n  100n  2n 2  2n 2  100n
i 1

15
TIC2001 Lecture 2a AY23/24S1
 2.9 Approximation of analysis results
 Very often, we are interested only in using a simple
term to indicate how efficient an algorithm is. The
exact formula of an algorithm’s performance is not
really needed.
 Example:
Given the formula: 2n2 + 100n
 the dominating term 2n2 can tell us approximately how the
algorithm performs by providing us with a measure of the
growth rate (how the number of operations executed grows
as n increases in size) of the algorithm
 This is called asymptotic analysis of the algorithm

16
TIC2001 Lecture 2a AY23/24S1
 2.10 Asymptotic analysis
 Asymptotic analysis is an analysis of algorithms
that focuses on
 analyzing the problems of large input size,
 considering only the leading term of the formula, and
 ignoring the coefficient of the leading term
 Some notations are needed in asymptotic
analysis

17
TIC2001 Lecture 2a AY23/24S1
 2.11 Algorithm Growth Rates (1/2)
 An algorithm’s time requirement can be measured
as a function of the problem size, say n
 An algorithm’s growth rate
 Enables the comparison of one algorithm with another
 Examples
 Algorithm A requires time proportional to n2
 Algorithm B requires time proportional to n

18
TIC2001 Lecture 2a AY23/24S1
 2.12 Algorithm Growth Rates (2/2)

Problem size

Figure - Time requirements as a function of the problem size n

 Algorithm efficiency is typically a concern for large

problems only. Why?
19
TIC2001 Lecture 2a AY23/24S1
 3 Big O notation

TIC2001 Lecture 2a AY23/24S1

 3.1 Definition – Big O notation
 Given a function f(n), we say g(n) is an (asymptotic)
upper bound of f(n), denoted as f(n) = O(g(n)), if there
exist a constant c > 0, and a positive integer n0 such
that f(n)  c*g(n) for all n  n0.

 f(n) is said to be
bounded from c*g(n)
above by g(n).
 O() is called the f(n)
“big O” notation.
g(n)

n0

21
TIC2001 Lecture 2a AY23/24S1
 3.2 Ignore the coefficients of all terms
 Based on the definition, 2n2 and 30n2 have the
same upper bound n2, i.e.,
 2n2 = O(n2) f1(n) = 2n2; g(n) = n2.
Let c=3 and n0=1, since 2n2  cn2  n ≥ n0
Hence f1(n) = O(g(n))
Why?
f2(n) = 30n2; g(n) = n2.
 30n2 = O(n2) Let c=31 and n0=1, since 30n2  cn2  n ≥ n0
Hence f2(n) = O(g(n))

They differ only in the choice of c.

 Therefore, in big O notation, we can omit the
coefficients of all terms in a formula:
 Example: f(n) = 2n2 + 100n = O(n2) + O(n)

22
TIC2001 Lecture 2a AY23/24S1
 3.3 Finding the constants c and n0
 Given f(n) = 2n2 + 100n, prove that f(n) = O(n2).
Observe that: 2n2 + 100n  2n2 + n2 = 3n2
whenever n ≥ 100.
 Set the constants to be c = 3 and n0 = 100.
By definition, we have f(n) = O(n2).
Notes:
1. n2  2n2 + 100n for all n, i.e., g(n)  f(n), and yet g(n)
is an asymptotic upper bound of f(n)
2. c and n0 are not unique.
For example, we can choose c = 2 + 100 = 102, and
n0 = 1 (because f(n)  102n2  n ≥ 1)
Q: Can we write f(n) = O(n3)? Yes
23
TIC2001 Lecture 2a AY23/24S1
 3.4 Is the bound tight?
 The complexity of an algorithm can be bounded
by many functions.
 Example:
 Let f(n) = 2n2 + 100n.
 f(n) is bounded by n2, n3, n4 and many others according
to the definition of big O notation.
 Hence, the following are all correct:
 f(n) = O(n2); f(n) = O(n3); f(n) = O(n4)

 However, we are more interested in the tightest

bound which is n2 for this case.

24
TIC2001 Lecture 2a AY23/24S1
 3.5 Growth Terms: Order-of-Magnitude
 In asymptotic analysis, a formula can be simplified
to a single term with coefficient 1
 Such a term is called a growth term (rate of growth,
order of growth, order-of-magnitude)
 The most common growth terms can be ordered as
follows: (note: many others are not shown)
O(1) < O(log n) < O(n) < O(n log n) < O(n2) < O(n3) < O(2n) < O(n!)
“fastest” “slowest”

Note:
 “log” = log base 2, or log2; “log10” = log base 10; “ln” = log
base e. In big O, all these log functions are the same.
(Why?)
25
TIC2001 Lecture 2a AY23/24S1
 3.6 Examples on big O notation

 f1(n) = ½n + 4
= O(n)
 f2(n) = 240n + 0.001n2
= O(n2)
 f3(n) = n log n + log n + n log (log n)
= O(n log n)

26
TIC2001 Lecture 2a AY23/24S1
 3.7 Order-of-Magnitude Analysis and
Big O Notation (1/2)

Figure - Comparison of growth-rate functions in tabular form

27
TIC2001 Lecture 2a AY23/24S1
 3.8 Order-of-Magnitude Analysis and
Big O Notation (2/2)

Figure - Comparison of growth-rate functions in graphical form

28
TIC2001 Lecture 2a AY23/24S1
 3.9 Summary: Order-of-Magnitude
Analysis and Big O Notation
 Order of growth of some common functions:
O(1) < O(log n) < O(n) < O(n log n) < O(n2) < O(n3) < O(2n) <O(n!)

 Properties of growth-rate functions

 You can ignore low-order terms
 You can ignore a multiplicative constant in the high-order
term
 O(f(n)) + O(g(n)) = O( f(n) + g(n) )

29
TIC2001 Lecture 2a AY23/24S1
 4 How to find the complexity
of a program?

TIC2001 Lecture 2a AY23/24S1

 4.1 Some rules of thumb and examples
 Basically just count the number of statements executed.
 If there are only a small number of simple statements in a
program – O(1)
 If there is a ‘for’ loop dictated by a loop index that goes up
to n – O(n)
 If there is a nested ‘for’ loop with outer one controlled by n
and the inner one controlled by m – O(n*m)
 For a loop with a range of values n, and each iteration
reduces the range by a fixed constant fraction (eg: ½)
– O(log n)
 For a recursive method, each call is usually O(1). So
 if n calls are made – O(n)
 if n log n calls are made – O(n log n)

31
TIC2001 Lecture 2a AY23/24S1
 4.2 Examples on finding complexity (1/2)
 What is the complexity of the following code fragment?
int sum = 0;
for (int i=1; i<n; i=i*2) {
sum++;
}
 It is clear that sum is incremented only when
i = 1, 2, 4, 8, …, 2k where k = log2 n
There are k+1 iterations. So the complexity is O(k) or
O(log n)

Note:
 In Computer Science, log n means log2 n.
 When 2 is replaced by 10 in the ‘for’ loop, the complexity is O(log10 n)
which is the same as O(log2 n). (Why?)
 log10 n = log2 n / log2 10
32
TIC2001 Lecture 2a AY23/24S1
 4.2 Examples on finding complexity (2/2)
 What is the complexity of the following code fragment?
(For simplicity, let’s assume that n is some power of 3.)
int sum = 0;
for (int i=1; i<n; i=i*3) {
for (j=1; j<=i; j++) {
sum++;
}
}

 f(n) = 1 + 3 + 9 + 27 + … + 3(log3 n)
= 1 + 3 + … + n/9 + n/3 + n
= n + n/3 + n/9 + … + 3 + 1 (reversing the terms in previous step)
= n * (1 + 1/3 + 1/9 + …)
 n * (3/2)
= 3n/2
= O(n)
33
TIC2001 Lecture 2a AY23/24S1
 4.3 Non-recursive Binary Search Algorithm (1)

 Requires array to be sorted in ascending order

 Maintain subarray where x (the search key) might
be located
 Repeatedly compare x with m, the middle element
of current subarray
 If x = m, found it!
 If x > m, continue search in subarray after m
 If x < m, continue search in subarray before m

34
TIC2001 Lecture 2a AY23/24S1
 4.3 Non-recursive Binary Search Algorithm (2)

 Data in the array a[] are sorted in ascending order

public static int binSearch(int[] a, int len, int x)
{
int mid, low = 0;
int high = len - 1;
while (low <= high) {
mid = (low + high) / 2;
if (x == a[mid]) return mid;
else if (x > a[mid]) low = mid + 1;
else high = mid - 1;
}
return -1;
}

35
TIC2001 Lecture 2a AY23/24S1
 4.3 Non-recursive Binary Search Algorithm (3)

 At any point during binary search, part of array is “alive”

(might contain the point x)
 Each iteration of loop eliminates at least half of previously
“alive” elements
 At the beginning, all n elements are “alive”, and after
 After 1 iteration, at most n/2 elements are left, or alive
2
 After 2 iterations, at most (n/2)/2 = n/4 = n/2 are left

 After 3 iterations, at most (n/4)/2 = n/8 = n/23 are left

:
i
 After i iterations, at most n/2 are left

 At the final iteration, at most 1 element is left

36
TIC2001 Lecture 2a AY23/24S1
 4.3 Non-recursive Binary Search Algorithm (4)
In the worst case, we have to search all the way up to the last
iteration k with only one element left.
We have:
n/2k = 1
2k = n
k = log n
Hence, the binary search algorithm takes O(f(n)) , or O(log n)
times

TIC2001 Lecture 2a AY23/24S1 37

 4.4 Time complexity of recursion: Fibonacci numbers
 Fibonacci numbers: 1, 1, 2, 3, 5, 8, 13, 21, …
 The first two Fibonacci numbers are both 1 (arbitrary numbers)
 The rest are obtained by adding the previous two together.
 Calculating the nth Fibonacci number recursively:
Fib(n) = 1 for n=1, 2
= Fib(n-1) + Fib(n-2) for n > 2

// Precond: n > 0
public static int fib(int n) {
if (n <= 2)
return 1;
else
return fib(n-1) + fib(n-2);
}

38
TIC2001 Lecture 2a AY23/24S1
 4.4 Time complexity of recursion: Fibonacci numbers

 Function for total number of operations is almost

exactly like the recursive case of Fibonacci
 T(n) = T(n-1)+T(n-2)+4 (1 comparison, 1 addition, 2 subtraction)
 T(n) < 2T(n-1)+c (replace 4 by some constant c)
 < 2(2T(n-2)+c)+c (expand T(n-1) to be 2*T(n-2)+c)
 < 4T(n-2)+2c+c
 < 8T(n-3)+4c+2c+c (cont. the expansion)
 … (will stop when n is 1)
 < 2n(T(1))+2n-1c+ 2n-2c+…+c (T(1) is c)
 < 2n*(1+1/2+1/4+1/8+…)*c
 = O(2n)
39
TIC2001 Lecture 2a AY23/24S1
 4.5 Analysis of Different Cases
Worst-Case Analysis
 Interested in the worst-case behaviour.
 A determination of the maximum amount of time that an algorithm
requires to solve problems of size n

Best-Case Analysis
 Interested in the best-case behaviour
 Not useful

Average-Case Analysis
 A determination of the average amount of time that an algorithm
requires to solve problems of size n
 Have to know the probability distribution
 The hardest

40
TIC2001 Lecture 2a AY23/24S1
 4.6 The Efficiency of Searching Algorithms

 Example: Efficiency of Sequential Search (data not

sorted)
 Worst case: O(n)
Which case?
 Average case: O(n)
 Best case: O(1)
Why? Which case?
 Unsuccessful search?
 Q: What is the best case complexity of Binary Search
(data sorted)?
 Best case complexity is not interesting. Why?

41
TIC2001 Lecture 2a AY23/24S1
 4.7 Keeping Your Perspective
 If the problem size is always small, you can
probably ignore an algorithm’s efficiency
 Weigh the trade-offs between an algorithm’s
time requirements and its memory requirements
 Order-of-magnitude analysis focuses on large
problems
 There are other measures, such as big Omega (), big
theta (), little oh (o), and little omega (). These may be
covered in more advanced module.

42
TIC2001 Lecture 2a AY23/24S1
You are the boss of a logistic company and your company occupies the first of the shop
houses (n of them) along this busy street. Every day you need to distribute some
packages to each of the other shop houses (n-1). You ask two of your staff to design a
plan to complete the task.

The plan from the first staff:

Do the following (n -1) times with j = 2 to n,
take a package, deliver to shop j, come back to shop 1

The plan from the second staff;

Take n-1 packages
do the following (n-1) times with k = 2 to n
go to shop k, deliver a package.

Which plan do you approve?

43
 End of file

TIC2001 Lecture 2a AY23/24S1