Carlos Cotrini & Andreas Streich

Computer Science II
Course at D-MAVT, ETH Zurich
Spring 2024
Comparison of Algorithms

Resources are bounded and not free:

Computing time → Efficiency
Storage space → Efficiency

Natural questions:
How efficient is our algorithm?
Is it as efficient as a different algorithm?
Is it the most efficient algorithm?

Actually, this course is nearly only about efficiency.

214
Computation Model
Unit cost model: operations on fundamental data types have cost 1.
Fundamental operations: arithmetic computations, comparisons,
assignment, flow control (jumps), . . .
Fundamental data types: e.g., bounded integer or floating point number.
Memory model
Memory access has cost 1.
Bounded-size Objects can be dynamically allocated with cost 1.
Fields of the objects can be accessed with cost 1.

⇒ Informal summary: Operations on single data elements ( size

independent of input size) cost 1.
⇒ Example list of length n: Single operation on single element costs 1, but
operation on all elements costs n.
215
Computation Model

Blocks of operations on fundamental data types also cost 1!

The cost of operations involving lists and other complex objects depend
of the costs of the elementary operations that are need to carry out the
original operation.
a + b, where a and b are lists with length m and n, costs (approximately)
m + n.
a.index(42), where a is a list of length n, costs (approximately) n.
np.zeros((m, n)) costs (approximately) m · n.

216
Asymptotic behavior: idea
Upper bound: O(g)
The running time behaves at most like n 7→ g(n).
N → R | ∃ c > 0, ∃n0 ∈ N : ∀ n ≥ n0 : 0 ≤ f (n) ≤ c · g(n)
n o
O(g) = f :

Lower bound: Ω(g)

The running time behaves at least like n 7→ g(n).
N → R | ∃ c > 0, ∃n0 ∈ N : ∀ n ≥ n0 : 0 ≤ c · g(n) ≤ f (n)
n o
Ω(g) = f :

Precise: Θ(g)
The running time behaves precisely like n 7→ g(n).
N → R |∃ cl > 0, ∃ cu > 0, ∃n0 ∈ N : ∀ n ≥ n0 :
n
Θ(g) := Ω(g) ∩ O(g) = f :
o
cl · g(n) ≤ f (n) ≤ cu · g(n) 217
Example

def count(a, b):

c = 0
for e in a:
Runtime Θ(n).
if e == b:
c += 1
return c

218
Example

def find(a, b):

for e in a:
Runtime: Best Case Θ(1). Worst
if e == b:
Case Θ(n). Average Case Θ(n).
return True
return False

219
Example

def zeros(n):
matrix = []
for i in range(n):
row = []
Runtime: Θ(n2 ).
for j in range(n):
row.append(0)
matrix.append(row)
return matrix

220
Example

def zeros(n):
matrix = []
for i in range(n):
row = []
Runtime: Θ(n2 ).
for j in range(i):
row.append(0)
matrix.append(row)
return matrix

221
Example

def all_combos(a, k):

if k < 0:
print(a)  
elif 0 <= k < len(a): Runtime: Θ nk .
for j in range(len(a)):
a[k] = j
all_combos(a, k - 1)

222
7. Searching

Linear Search, Binary Search

[Ottman/Widmayer, Kap. 3.2]

[Cormen et al., Kap. 2: Problems 2.1-3, 2.2-3, 2.3-5]

223
7.1 Linear Search

224
Search in Array

Provided:
Array a with n elements (a[0], . . . , a[n − 1]).
Key b

b = 42
22 20 32 10 35 24 42 38 28 41
k=6
0 1 2 3 4 5 6 7 8 9

Wanted:
index k, 0 ≤ k < n with a[k] = b or ”not found”.

225
Linear Search
Traverse the array a from a[0] until a[n − 1] or until found.
def LinearSearch(a, b):
for i, x in enumerate(a):
if x == b:
return i
return "nicht gefunden"
Best case: 1 comparison.
22 20 32 10 35 24 42 38 28 41 b = 22
0 1 2 3 4 5 6 7 8 9
Worst case: n comparisons.
22 20 32 10 35 24 42 38 28 41 b = 41
0 1 2 3 4 5 6 7 8 9 226
7.2 Binary Search

227
Search in a sorted array
Provided:
b = 23
10 20 22 24 28 32 35 38 41 42
not found!
0 1 2 3 4 5 6 7 8 9

Sorted array a with n elements (a[0], . . . , a[n − 1]) such that

a[0] ≤ a[1] ≤ · · · ≤ a[n − 1].
Key b
Wanted:
index k, 0 ≤ k < n with a[k] = b or ”not found”.
Strategies:
Linear Search: Worst case Θ(n)
Binary Search: Worst case Θ(log n)
228
Binary Search: Example b = 23
10 20 22 24 28 32 35 38 41 42 b < 28
0 1 2 3 4 5 6 7 8 9

10 20 22 24 28 32 35 38 41 42 b > 20
0 1 2 3 4 5 6 7 8 9

10 20 22 24 28 32 35 38 41 42 b > 22
0 1 2 3 4 5 6 7 8 9

10 20 22 24 28 32 35 38 41 42 b < 24
0 1 2 3 4 5 6 7 8 9
Invariant:
10 20 22 24 28 32 35 38 41 42 If b is in a
0 1 2 3 4 5 6 7 8 9 then in a[l:r+1].
229
Binary Search Algorithm

def bin_search(a, l, r, b):

if r < l:
return None
else:
m = (l + r) // 2
if a[m] == b:
return m
elif b < a[m]:
return bin_search(a, l, m-1, b)
else: # a[m] > b
return bin_search(a, m+1, r, b)

230
Analysis (informal)

In each step, half of the remaining numbers get eliminated.

twice as many numbers ⇒ one step more
logarithmic behavior in the worst case: Θ(log n)

231
Analysis (formal)
Recurrence (n = 2k )

d falls n = 1,
T (n) = 
T ( n2 ) + c falls n > 1.
Compute1 :
n n
   
T (n) = T +c=T + 2c = ...
2 4
n
 
=T i +i·c
2
n
 
=T + log2 n · c = d + c · log2 n ∈ Θ(log n)
n
1
Versuche eine geschlossene Form zu finden, indem die Rekurrenz, ausgehend von T (n),
wiederholt eingesetzt wird.Try to find a closed form of T by applying the recurrence
repeatedly (starting with T (n)).
232
7.3 Merge Sort

[Cormen et al., Kap. 2.3.1]

233
Merge sort
Divide and Conquer!
1. Divide: Divide array into two halves of (roughly) equal size.
2. Recurse: Sort both halves reursively.
3. Conquer: Combine sorted halves by merging.

7 4 1 16 9 11 10 2 12 3

71 4 71 9
16 9
16 11
2 10
3 10
2 12
11 3
12

1 2 3 4 7 9 10 11 12 16

234
Algorithm merge

def merge(a1, a2):

b, i, j = [], 0, 0
while i < len(a1) and j < len(a2):
if a1[i] < a2[j]:
b.append(a1[i])
i += 1
else:
b.append(a2[j])
j += 1
b += a1[i:]
b += a2[j:]
return b

235
Algorithm merge_sort

def merge_sort(a):
if len(a) <= 1:
return a
else:
sorted_a1 = merge_sort(a[:len(a) // 2])
sorted_a2 = merge_sort(a[len(a) // 2:])
return merge(sorted_a1, sorted_a2)

236
Runtime MergeSort

Let T (n) be the runtime of MergeSort.


1 if n = 1
T (n) =
2T ( n ) + an + b if n > 1
2

⇒ T (n) ∈ Θ(n log n)

237
Derivation

1 if n = 1
T (n) = 
2T ( n2 ) + an + b if n > 1

Apply recursively:
T (n) = 2T (n/2) + an + b = 2(2T (n/4) + an/2 + b) + an + b
= 2(2(2T (n/8) + an/4 + b) + an/2 + b) + an + b = ...
= 2(2(...(2(2T (n/2k ) + an/2k−1 + b)...) + an/22 + b) + an/21 + b) + an + b
= 2k T (1) + 2k−1 an/2k−1 + 2k−2 an/2k−2 + ... + 2k−k an/2k−k +2k b
| {z }
k terms
= n + ank + nb = n(b + 1) + an log2 n ∈ Θ(n log n)

238
7.4 Quick Sort

[Cormen et al., Kap. 7, Bhargava Ch. 4]

239
Quick Sort

What is the disadvantage of merge sort?

Requires additional Θ(n) storage for merging.
When could we reduce the merge costs?
Left part smaller elements than right part.
3
4 47 6
3 97 9
6 10
14 13
11 10
12 13
11 14
12

3 4 6 7 9 10 11 12 13 14

How to achieve this?

Pivot and Partition!
240
Use a pivot

≥
< < < ≥
< ≥
< p
< ≥
< ≥ ≥
< p
≥
0 k n−1

1. Choose a (an arbitrary) pivot p

2. Partition a in two parts:
one part L with the elements with a[i] < p
and another part R with a[i] ≥ p

3. Quick Sort: Recursion on parts L and R

241
Algorithm quick_sort

< < ≥
< ≥
< ≥
< p
< ≥
< ≥ ≥
< p
≥
k
l r

def quicksort(a, l, r):

if l < r:
k = partition(a, l, r)
quicksort(a, l, k - 1)
quicksort(a, k + 1, r)

242
Algorithm partition

def partition(a, l, r):

p = a[r]
j = l Invariant: Before iteration i ...
for i in range(l, r): all elements in a[l:j] are < p.
if a[i] < p:
a[i], a[j] = a[j], a[i] all elements in a[j:i] are ≥ p.
j += 1 Running time: Θ(r − l) ⊆ Θ(n).
a[j], a[r] = a[r], a[j]
return j

243
Quick Sort (arbitrary pivot)

2 4 5 6 8 3 7 9 1

2 1 3 6 8 5 7 9 4

1 2 3 6 8 5 7 9 4

1 2 3 4 5 8 7 9 6

1 2 3 4 5 6 7 9 8

1 2 3 4 5 6 7 8 9

244
Choice of the pivot.
The minimum is a bad pivot: worst case Θ(n2 )

p1 p2 p3 p4 p5 p6 p7 p8 p9 p10

A good pivot has a linear number of elements on both sides.

≥ϵ·n ≥ϵ·n

⇒ Practically the pivot is often the median of three elements.

For example: Median3(a[l], a[r], a[⌊(l + r)/2⌋]).
245
Analysis (randomized quick sort)

Theorem 1
In the worst case quick sort has an running time of Θ(n2 ).

On expectation randomized quick sort with a random pivot has a running

time of Θ(n · log n).

(without proof.)

246