Principles of Algorithmic Techniques

CS-E3190
Pekka Orponen
Department of Computer Science
Aalto University

Autumn 2017
Lecture 2: Algorithms and their analysis

I In algorithm design, it is important to understand how the

resource requirements of an algorithm depend on the input
characteristics.
I A standard measure is an algorithms worst-case runtime,
or time complexity.
I The dependence of this on input size can typically be
determined a priori, by just analysing the algorithm
description. (This can, however, sometimes be a
challenging mathematical problem.)

CE-3190 PAT
Autumn 2017
2/23
2.1 Analysis of algorithms: basic notions

I Generally speaking, an algorithm A computes a mapping

from a potentially infinite set of inputs (or instances) to a
corresponding set of outputs (or solutions).
I Denote by T (x) the number of elementary operations that
A performs on input x and by |x| the size of an input
instance x.
I Denote by T (n) also the worst-case time that algorithm A
requires on inputs of size n, i.e.,

T (n) = max {T (x) : |x| = n}.

CE-3190 PAT
Autumn 2017
3/23
2.2 Analysis of iterative algorithms
To illustrate the basic worst-case analysis of iterative
algorithms, let us consider yet another well-known (but bad)
sorting method.

Algorithm 1: The insertion sort algorithm

1 function I NSERT S ORT (A[1 . . . n])
2 for i 2 to n do
3 a A[i]; j i 1
4 while j > 0 and a < A[j] do
5 A[j + 1] A[j]; j j 1
6 end
7 A[j + 1] a
8 end

CE-3190 PAT
Autumn 2017
4/23
Analysis of insertion sort

Denote: Tk l = the complexity of a single execution of lines k

thru l. Then:
T5 (n, i, j) c1
T4 6 (n, i) c2 + (i 1)c1
T3 7 (n, i) c3 + c2 + (i 1)c1
Pn
T2 8 (n) c4 + i=2 (c3 + c2 + (i 1)c1 )
Pn
= c4 + (n 1)(c3 + c2 ) + c1 i=2 (i 1)
const n + c1 12 n(n 1)

Thus T (n) = T2 8 (n) = O(n2 ).

CE-3190 PAT
Autumn 2017
5/23
Basic analysis rules
Denote T [P] = the time complexity of an algorithm segment P.
I T [x e] = constant, T [read x] = constant, T [write x] =
constant.
I T [S1 ; S2 ; . . . ; Sk ] = T [S1 ] + + T [Sk ] =
O(max{T [S1 ], . . . , T [Sk ]})
I T [if P then S1 else S2 ] =
(
T [P] + T [S1 ] if P = true
T [P] + T [S2 ] if P = false
I T [while P do S] =
T [P] + (number of times P = true) (T [S] + T [P])
In analysing nested loops, proceed from innermost out. Control
variables of outer loops enter as parameters in the analysis of
inner loops.
CE-3190 PAT
Autumn 2017
6/23
Operation costs in programming languages
I If one is using a modern high-level programming language
(such as Python), one needs to be aware of the runtime
costs of basic operations.
I If an operation addresses a structured object, the actual
cost may depend highly on the language implementation.
I E.g. if A is a Python list of length n, then the cost of
obtaining a k -element slice such as A[0:k] is O(k ).
I In fact, even the cost of inserting a single element at or
near the beginning of the list, e.g. operation
A.insert(0,x) has cost O(n)! This is because Python lists
are implemented internally as simple linear arrays.
I The costs of several basic Python operations on structured
data are listed at
https://wiki.python.org/moin/TimeComplexity.

CE-3190 PAT
Autumn 2017
7/23
2.3 Analysis of recursive algorithms
Let us review the analysis of the merge sort algorithm.
Let M ERGE (A0 , A00 , A) be a subroutine which takes two sorted arrays
A0 [1 . . . m] and A00 [1 . . . (n m)] and merges them, element by
element, into a single sorted sequence in array A[1 . . . n] in O(n)
steps.

Algorithm 2: The merge sort algorithm

1 function M ERGE S ORT (A[1 . . . n])
2 if n = 1 then return
3 else
4 Introduce auxiliary arrays A0 [1 . . . bn/2c], A00 [1 . . . dn/2e]
5 A0 A[1 . . . bn/2c]
6 A00 A[bn/2c + 1 . . . n]
7 M ERGE S ORT(A0 )
8 M ERGE S ORT(A00 )
9 M ERGE(A0 , A00 , A)
10 end
CE-3190 PAT
Autumn 2017
8/23
Assume the number of elements n is a power of 2. Then the
runtime T (n) of M ERGE S ORT can be described by a recurrence
relation:
(
T (1) c1
T (n) 2T ( n2 ) + c2 n, n = 2k , k = 0, 1, 2, . . .

A straightforward approach to solving such recurrences is by

unwinding them:

T (n) 2T ( n2 ) + c2 n
2(2T ( n4 ) + c2 n2 ) + c2 n
4T ( n4 ) + 2c2 n
... [general pattern emerges]
k n
2 T ( 2k ) + k c2 n
2log2 n T (1) + log2 n c2 n
c1 n + c2 n log2 n = O(n log2 n).

CE-3190 PAT
Autumn 2017
9/23
2.4 Order of growth of functions

I Also known as the big-O notation.

I In the analysis of algorithms, attention is primarily paid to
the growth rate of the dominant term of T (n), because:
1. For large input sizes n, the other effects of T (n) are
vanishingly small; and
2. Different machine implementations of an algorithm induce
different constant factors and lower-order terms in T (n) in
any case.
3. The big-O notation is a convenient estimation tool.
I We have seen several examples of this usage already, so
lets make the notation precise.

CE-3190 PAT
Autumn 2017
10/23
Definitions and notations
I Let f : N ! R+ .
I Define the order (of growth class) of f as:
O(f ) = {t : N ! R+ |9c, n0 0:n n0 ) t(n) c f (n)}.
I Thus: t 2 O(f ), if t(n) is for all large enough n at most a
constant factor bigger than f (n).
I One usually uses the somewhat imprecise, but very
convenient notation t = O(f ) instead of the formal t 2 O(f ).
I Also, one denotes functions by means of a generic
argument n. Thus one writes, e.g.,
2n2 + 3n = O(n2 )
as a shorthand for
the function f , for which f (n) = 2n2 + 3n, is in the class
O(g), where g(n) = n2 .

CE-3190 PAT
Autumn 2017
11/23
Some typical function growth rates

T(n)
1 3
n - n
2 2
3000

5 n2

2000 100n

1000

n
5 10 15 20

CE-3190 PAT
Autumn 2017
12/23
Some examples and rules in the O-calculus
1. n2 = O(n3 ) i.e., to be precise, n2 2 O(n3 )
2. 5n3 + 3n2 = O(n3 )
3. loga n = O(logb n) for all a, b > 1
P
4. log2 (n!) = ni=1 log2 i = O(n log2 n)
5.
n
X 1
= O(1) constant
2i
i=0

6. If r < s, then nr = O(ns ), ns 6= O(nr )

7. If r < s, then r n = O(sn ), sn 6= O(r n )
8. For any r , s > 1, ns = O(r n ), r n 6= O(ns )
9. For any r , s > 0, (log2 n)s = O(nr ), nr 6= O((log2 n)s )

CE-3190 PAT
Autumn 2017
13/23
Rules of thumb for determining order of growth

In simple cases (sum of logarithms, powers, exponentials):

1. Determine highest-order term using rules 3, 68 on
previous slide:
1.1 logarithms powers exponentials
1.2 for logarithms: base doesnt matter
for powers: small exponent large exponent
for exponentials: small base large base
2. Ignore all lower-order terms
3. Ignore constant factor in highest-order term

CE-3190 PAT
Autumn 2017
14/23
Properties of order classes
I.e., further rules in the O-calculus.

1. f (n) 2 O(g(n)), g(n) 2 O(h(n)) ) f (n) 2 O(h(n)).

2. O(cf (n)) = O(f (n)), for any constant c > 0.
3. O(f (n) + g(n)) = O(max(f (n), g(n))).
4. f (n) 2 O(g(n)) ) O(f (n)) O(g(n)).

Example:

n3 + 3n2 + n + 8 2 O(|{z}
n3 + 3n 2 3
| +{zn + 8}) O(n )
g(n) f (n)

Or, more standardly, n3 + 3n2 + n + 8 = O(n3 ).

CE-3190 PAT
Autumn 2017
15/23
Comparing orders via limits
Theorem 1 Let f , g : N ! R+ . Then:
(i) If
f (n)
lim = c > 0,
n!1 g(n)

then f (n) = O(g(n)), g(n) = O(f (n)).

(ii) If
f (n)
lim = 0,
n!1 g(n)
then f (n) = O(g(n)), g(n) 6= O(f (n)).
(iii) If
f (n)
lim = 1,
n!1 g(n)
then g(n) = O(f (n)), f (n) 6= O(g(n)).

CE-3190 PAT
Autumn 2017
16/23
LHpitals rule
Theorem 2 Let f , g : R ! R. Assume that
lim f (x) = lim g(x) = 0 or 1,
x!1 x!1
and moreover that f 0 (x) and g 0 (x) exist and g 0 (x) 6= 0 for all
large enough x. Then
f (x) f 0 (x)
lim = lim 0 ,
x!1 g(x) x!1 g (x)

if the latter limit exists. p p

Example. Let us verify that ln n = O( n), but n 6= O(ln n).
p 1 1
D(ln x) = x 1 , D( x) = x 2 6= 0.
2
ln n n 1 1
) lim p = lim 1
. = lim 2n 2 = 0.
n!1 n n!1 1
n 2 n!1
2
The claim follows by Theorem 1(ii).
CE-3190 PAT
Autumn 2017
17/23
Other order-of-growth notions (1/2)

Let f : N ! R+ . Define the function classes

1
(f ) = {t : N ! R+ | 9c > 0 : f (n) t(n) cf (n) for all lar. en. n}
c
o(f ) = {t : N ! R+ | 8c > 0 : t(n) < cf (n) for all large enough n}

(f ) = {t : N ! R+ | 9c > 0 : t(n) cf (n) for infinitely many n}

Some properties:
1. g 2 (f ) , g 2 O(f ) and f 2 O(g)
2. g 2 o(f ) , g 2 O(f ) and f 2
/ O(g)
3. g 2 (f ) , g 2
/ o(f )

CE-3190 PAT
Autumn 2017
18/23
Other order-of-growth notions (2/2)

One reads:
I f 2 (g): f is of the same order as g.
I f 2 o(g): f is of lower order than g.
I f 2 (g): f is of at least the same order as g.

Note. Often in the literature (incl. in the Dasgupta et al.

textbook) one sees for the notation a different definition:
f 2 (g) , g 2 O(f ). The difference lies in whether f is
required to be bigger than g for infinitely many n or all large
enough n.

CE-3190 PAT
Autumn 2017
19/23
Computing with order-of-growth classes (1/3)

I.e., yet more rules in the O-calculus.

Let X , Y be function classes (e.g. X = O(f ), Y = O(g)), and let

be some arithmetic operation (+, , , . . .) Define:

X Y = {t(n) | t(n) = f (n) g(n), f (n) 2 X , g(n) 2 Y }.

Examples:
1. O(n2 ) + O(n3 ) = O(n3 )
2. n + O( n1 ) , {n} + O( n1 )
3. n O( n1 ) = O(n n1 ) = O(1)

CE-3190 PAT
Autumn 2017
20/23
Computing with order-of-growth classes (2/3)

Define also: [
O(X ) = O(f ),
f 2X

and correspondingly for the other order-of-growth notions.

Examples:
1. O(O(n2 )) = O(n2 )
2. O(n O(n2 )) = O(O(n3 )) = O(n3 )

CE-3190 PAT
Autumn 2017
21/23
Computing with order-of-growth classes (3/3)

1. f (n) 2 O(f (n))

2. c O(f (n)) = O(f (n)) for any constant c > 0.
3. O(f (n)) + O(g(n)) = O(f (n) + g(n)) = O(max(f (n), g(n)))
4. O(O(f (n))) = O(f (n))
5. f (n) O(g(n)) = O(f (n) g(n))
6. O(f (n)) O(g(n)) = O(f (n) g(n))

CE-3190 PAT
Autumn 2017
22/23
Estimating order-of-growth with the O-calculus
Example: Whatpis the dominating behaviour (with an error
estimate) of n( n n 1) as n ! 1?
p
n ln n ln n ln n 2
n=e n =1+ + O(( ) )
n n
p ln n ln n 2
) n( n n 1) = n ( + O(( ) )
n n
ln n 2
= ln n + n O(( ) )
n
ln2 n
= ln n + O( )
n
p
Consequently: The value of n( n n 1) grows approximately as
2
ln n. The order of the error in this estimate is O( ln n n ), which
goes to zero as n ! 1.
CE-3190 PAT
Autumn 2017
23/23