Carlos Cotrini & Andreas Streich

Computer Science II
Course at D-MAVT, ETH Zurich
Spring 2024
Announcement: Exercise Scalar Product

The task can easily be solved using numpy.

Doing this however, fails one of the test cases due to numerical reasons.
Only this time, we accept this solution.
From now on, we formulate the exercise questions more precisely and
might limit the functionality you are allowed to use. Please read the task
description completely and attentively.
Please check that your implementation passes the test cases. If you have
any questions, please contact your teaching assistant with enough time
before the deadline.

90
Announcement: bonus exercise 1

Solutions must be semantically correct and reasonably efficient!

We run new tests after the submission deadline to catch hardcoders and
semantically incorrect solutions.
If your solution is efficient enough to pass the given tests, it will most
likely be efficient enough to pass the new tests.

91
Algorithm design

every-day problem

abstraction

computational problem

design

algorithm 1 algorithm 2
comparison

92
Preview

Sorting coins

Sorting numbers

Selection sort Mergesort

Θ(n2 ) Θ(n log n)

93
4.1 Selection sort
[Bhargava Ch. 2]

94
Example: Selection Sort

5 6 2 8 4 1 (i = 0) def sort(a):
n = len(a)
1 6 2 8 4 5 (i = 1)
for i in range(n):
1 2 6 8 4 5 (i = 2) # find minimum
mini = i
1 2 4 8 6 5 (i = 3) for j in range(i+1, n):
if a[j] < a[mini]:
1 2 4 5 6 8 (i = 4)
mini = j
1 2 4 5 6 8 (i = 5) # swap minimum
a[mini],a[i] = a[i],a[mini]
1 2 4 5 6 8

95
Example: Selection Sort
Invariant:
before iteration i, a[:i]
5 6 2 8 4 1 (i = 0) contains the smallest i
elements of a sorted in
1 6 2 8 4 5 (i = 1)
ascending order.
1 2 6 8 4 5 (i = 2)
Selection of the smallest
1 2 4 8 6 5 (i = 3) element a[mini] in the
unsorted part a[i:n].
1 2 4 5 6 8 (i = 4)
Swap a[mini] with the first
1 2 4 5 6 8 (i = 5) element a[i] of the unsorted
part.
1 2 4 5 6 8
Repeat until everything is
sorted (i = n). 96
Invariants

A method to prove that an algorithm that is based on a loop is correct.

An invariant must fulfil three things:
Initialization: It must hold before the loop starts.
Continuation: When it holds before an iteration of the loop, then it also
holds until the begin of the next iteration.
Termination: When the loop ends, the invariant yields a property that
helps to show the algorithm’s correctness.

97
Invariants

Invariant: Before repetition i, a[: i] contains the i smallest elements of a

sorted.
It must fulfil three things:
Initialization: i = 0 and then a[: 0] = [] is trivially sorted.
Continuation: If a[: i] is sorted and has the smallest i elements, then
appending the smallest element from the previous a[i :] makes a[: i + 1]
also sorted.
Termination: At the end, i = n and a[: n] = a is sorted.

98
4.2 Merge Sort

[Cormen et al., Kap. 2.3.1]

99
divide et impera
Divide and Conquer
Divide the problem recursively into smaller/simpler subproblems that
contribute to the simplified computation of the overall problem.

P11 L11
P1 L1
P12 L12
Problem P Solution L
P21 L21
P2 L2
P22 L22
100
Merge sort
Divide and Conquer!
1. Divide: Divide array into two halves of (roughly) equal size.
2. Recurse: Sort both halves reursively.
3. Conquer: Combine sorted halves by merging.

7 4 1 16 9 11 10 2 12 3

71 4 71 9
16 9
16 11
2 10
3 10
2 12
11 3
12

1 2 3 4 7 9 10 11 12 16

101
Merge (Conquer): Overview
Assumption: two halves of the array a are already sorted.
1 4 7 9 16 2 3 10 11 12

b: 1 2 3 4 7 9 10 11 12 16

Minimum of a can be evaluated with one comparison.

Iteratively: merge the two presorted halves of a to b.
Invariant after i steps (for 0 ≤ i < n):
b[:i] is sorted and contains the i smallest elements of a.
the two remaining not yet merged parts are sorted.

102
Merge (Conquer): Beispiel

1 4 7 9 16 2 3 10 11 12

1 2 3 4 7 9 10 11 12 16

103
Algorithm merge

def merge(a1, a2):

b, i, j = [], 0, 0
while i < len(a1) and j < len(a2):
if a1[i] < a2[j]:
b.append(a1[i])
i += 1
else:
b.append(a2[j])
j += 1
b += a1[i:]
b += a2[j:]
return b

104
Algorithm merge_sort

def merge_sort(a):
if len(a) <= 1:
return a
else:
sorted_a1 = merge_sort(a[:len(a) // 2])
sorted_a2 = merge_sort(a[len(a) // 2:])
return merge(sorted_a1, sorted_a2)

105
Merge Sort: Step by step (after every merge)
9 8 6 7 6 4 1 2

8 9 6 7 6 4 1 2

8 9 6 7 6 2 1 4

6 7 8 9 6 2 1 4

6 7 8 9 2 6 1 4

6 7 8 9 2 6 1 4

6 7 8 9 1 2 4 6

1 2 4 6 6 7 8 9
106
5. Efficiency of algorithms

Running time analysis

Computation model
Asymptotics: O, Θ, Ω

107
Comparison of Algorithms

Resources are bounded and not free:

Computing time → Efficiency
Storage space → Efficiency

Natural questions:
How efficient is our algorithm?
Is it as efficient as a different algorithm?
Is it the most efficient algorithm?

Actually, this course is nearly only about efficiency.

108
Comparison of Algorithms by time measurements?

Depends on machine, system, workload, . . . 109

Running time analysis

Goals
Understand dependence on the input size.
Quantify the running time behavior of an algorithm independent of the
machine.
Compare running time behavior of algorithms.

110
Running Time Analysis: Dependence on input

Input: parameterized by parameter (e.g., length n of the sequence)

Running time: depending on the parameter (e.g., n2 )

The running time of an algorithm usually depends on the problem instance,

not only the input size. Often there are “good” and “bad” instances (for each
input size).

Therefore we consider algorithms sometimes ”in the average“, most often

in the ”worst case“, and rarely in the ”best case“.

111
Computation Model
Unit cost model: operations on fundamental data types have cost 1.
Fundamental operations: arithmetic computations, comparisons,
assignment, flow control (jumps), . . .
Fundamental data types: e.g., bounded integer or floating point number.
Memory model
Memory access has cost 1.
Bounded-size Objects can be dynamically allocated with cost 1.
Fields of the objects can be accessed with cost 1.

⇒ Informal summary: Operations on single data elements ( size

independent of input size) cost 1.
⇒ Example list of length n: Single operation on single element costs 1, but
operation on all elements costs n.
112
Computation Model

Blocks of operations on fundamental data types also cost 1!

The cost of operations involving lists and other complex objects depend
of the costs of the elementary operations that are need to carry out the
original operation.
a + b, where a and b are lists with length m and n, costs (approximately)
m + n.
a.index(42), where a is a list of length n, costs (approximately) n.
np.zeros((m, n)) costs (approximately) m · n.

113
Asymptotic behavior: idea
The exact running time of an algorithm usually cannot be predicted.
Idea: general trend for large inputs, e.g.,:
If the input size doubles, the running time doubles.
If the input size doubles, the running time quadruples.
n 2·n 3·n
A1 n 2·n 3·n
A2 2·n 4·n 6·n
A3 4·n+2 8·n+2 12 · n + 2
A4 n2 4 · n2 9 · n2
A5 2 · n2 8 · n2 18 · n2
A6 4 · n2 + 3 · n + 2 16 · n2 + 6 · n + 2 36 · n2 + 9 · n + 2
⇒ We only consider the asymptotic behavior of the algorithm.
⇒ We ignore all constant additive and multiplicative factors. 114
Asymptotic behavior: idea
The algorithms with running time
n, 2 · n, 4 · n + 2 all behave the same way:
⇒ If the input size doubles, the running time doubles.
⇒ Their running times behave like the function n 7→ n.
⇒ We say that their running time is in Θ(n).

n2 , 2 · n2 , 4 · n2 + 2 all behave the same way:

⇒ If the input size doubles, the running time quadruples.
⇒ Their running times behave like the function n 7→ n2 .
⇒ We say that their running time is in Θ(n2 ).

More general: The asymptotic notation Θ(g(n)) for the running time means
that the algorithm behaves for large n like g(n).
115
Asymptotic behavior: idea

Precise: Θ(n2 )
The running time behaves like n 7→ n2 .
problem size doubled ⇒ running time quadruples

Upper bound: O(n2 )

The running time behaves at most like n 7→ n2 .
problem size doubled ⇒ running time at most quadruples

Lower bound: Ω(n2 )

The running time behaves at least like n 7→ n2 .
problem size doubled ⇒ running time at least quadruples

116
Definition: asymptotic upper bound
provided: a function g : N→R
g(n) = n2

N→R|
n
O(g) = f :
∃ c > 0, ∃n0 ∈ N:
∀ n ≥ n0 :
o
0 ≤ f (n) ≤ c · g(n)

Notation: O(g(n)) := O(g(·)) = O(g)

n0

N R
Set of all functions f : → that satisfy: there is some c ∈ R+ and some
N
n0 ∈ such that 0 ≤ f (n) ≤ c · g(n) for all n ≥ n0 . 117
Definition: asymptotic lower bound
provided: a function g : N→R
N→R|
n
Ω(g) = f :
∃ c > 0, ∃n0 ∈ N:
g(n) = n
∀ n ≥ n0 :
o
0 ≤ c · g(n) ≤ f (n)

Notation: Ω(g(n)) := Ω(g(·)) = Ω(g)

n0

N R
Set of all functions f : → that satisfy: there is some c ∈ R+ and some
N
n0 ∈ such that 0 ≤ f (n) ≤ c · g(n) for all n ≥ n0 .
118
Definition: asymptotic tight bound

provided: a function g : N→R g(n) = n2

Θ(g) := Ω(g) ∩ O(g)

Simple, closed form: exercise.

Notation: Θ(g(n)) := Θ(g(·)) = Θ(g)

119
About the Notation

Common casual notation

f = O(g)
should be read as f ∈ O(g).
It holds that
f1 = O(g), f2 = O(g) ̸⇒ f1 = f2 !

n = O(n2 ) and n2 = O(n2 ) but of course n ̸= n2 .

We avoid this notation where it could lead to ambiguities.

120
Example: Selection Sort

 
n−1
X n−1
X
1+ 1 + 2 + 1
i=0 j=i+1

n(n − 1)
= = Θ(n2 )
2

121
Growth in every-day life

What grows linearly?

Counting
reading books

What grows quadratically?

clink glasses
solving a jigsaw puzzle naively

122
Small n
2n
n4
60

40 n2

20

n
ln n
2 3 4 5 6
123
Larger n
·106
2n
1

0.8

0.6

0.4 n4

0.2

n2
n
5 10 15 20 log n
124
“Large” n
·1020
1

2n
0.8

0.6

0.4

0.2

n4 n2
n
20 40 60 80 100 log n
125
Logarithms
1,000 n2

800

600

400 n3/2
n log n
200
n
log n
10 20 30 40 50
126
Time Consumption

Assumption 1 Operation = 1µs.

problem size 1 100 10000 106 109

log2 n 1µs 7µs 13µs 20µs 30µs
n 1µs 100µs 1/100s 1s 17 minutes
n log2 n 1µs 700µs 13/100µs 20s 8.5 hours
n2 1µs 1/100s 1.7 minutes 11.5 days 317 centuries
2n 1µs 1014 centuries ≈∞ ≈∞ ≈∞

127
Example

def count(a, b):

c = 0
for e in a:
Runtime Θ(n).
if e == b:
c += 1
return c

128
Example

def find(a, b):

for e in a:
Runtime: Best Case Θ(1). Worst
if e == b:
Case Θ(n). Average Case Θ(n).
return True
return False

129
Example

def zeros(n):
matrix = []
for i in range(n):
row = []
Runtime: Θ(n2 ).
for j in range(n):
row.append(0)
matrix.append(row)
return matrix

130
Example

def zeros(n):
matrix = []
for i in range(n):
row = []
Runtime: Θ(n2 ).
for j in range(i):
row.append(0)
matrix.append(row)
return matrix

131
Example

def faux_nested_loop(a):
i = 0
while i < len(a):
j = i
while j < len(a) and a[j] == a[i]:
j += 1
i = j

Runtime: O(n2 )? Yes, but Θ(n) is more precise!

132
Example

def partition(A, l, r):

pivot = A[r] # last element as pivot
i = l - 1 # Temporary pivot index

for j in range(l, r):

if A[j] <= pivot:
# Move the temporary pivot index forward
i = i + 1
# Swap current element with element at the temp. pivot index
A[i], A[j] = A[j], A[i]

# Move the pivot element to the correct pivot position

i = i + 1
A[i], A[r] = A[r], A[i]
return i

133
Example

def all_combos(a, k):

if k < 0:
print(a)  
elif 0 <= k < len(a): Runtime: Θ nk .
for j in range(len(a)):
a[k] = j
all_combos(a, k - 1)

134
Summary

every-day problem

abstraction

computational problem

design

algorithm 1 algorithm 2
comparison

1. How to design algorithms?

2. How to compare algorithms? ⇒ Asymptotics: O, Ω, Θ
135
Summary: runtimes (1/3)

How to compute runtimes?

Define parameters that quantify input size (this is usually the input’s
length n).
Basic operations with basic data (comparisons, assignment, arithmetic,
etc...) cost 1.
A loop costs the sum of the cost of each iteration.

136
Summary: runtimes (2/3)

Try to derive a simplified formula T (n) that sums up the cost of your
algorithm.
Usually, T (n) is a sum of terms. Take the dominant term f (n) without its
coefficient. The dominant term is the one for which limn→∞ fg(n)
(n)
= 0, for
any other term g(n) in T (n).
Conclude that T (n) = Θ(f (n)).
If it’s too hard to derive T (n), compute at least an upper bound U (n).
Conclude that T (n) = O(f (n)), where f (n) is U (n)’s dominant term.

137
Summary: runtime (3/3)

Note that T (n) = O(f (n)) only means that asymptotically, the runtime of
your algorithm’s runtime is eventually bounded above by a factor of
f (n). T (n) = Θ(f (n)) is more precise. It means that your algorithm runs
in time proportional to f (n).
If you manage to derive a lower bound for T (n), say L(n). Then you can
conclude that T (n) = Ω(g(n)), where g(n) is L(n)’s dominant term.
If T (n) is O(f (n)) and also Ω(f (n)), then T (n) is also Θ(f (n))! This trick
can sometimes be useful for algorithms that are hard to analyze.

138