UNIT I

INTRODUCTION

Notion of an Algorithm – Fundamentals of Algorithmic Problem Solving – Important Problem

Types – Fundamentals of the Analysis of Algorithmic Efficiency –Asymptotic Notations and
their properties. Analysis Framework – Empirical Analysis - Mathematical analysis for
Recursive and Non-recursive algorithms- Visualization.

Algorithm
● An algorithm is a set of rules for carrying out calculation either by hand or on a machine.
● An algorithm is a finite step-by-step procedure to achieve a required result.
● An algorithm is a sequence of computational steps that transform the input into the
output.
● An algorithm is a sequence of operations performed on data that have to be organized
in data structures.
● An algorithm is an abstraction of a program to be executed on a physical machine
(model of Computation).

1.1 NOTION OF ALGORITHM

● An algorithm is a sequence of unambiguous instructions for solving a problem, i.e.,

for obtaining a required output for any legitimate input in a finite amount of time.

problem

algorithm

input “Computer” output

Algorithms
● It is not depended on programming language, machine.
● Are mathematical entities, which can be thought of as running on some sort of idealized
computer with an infinite random access memory.
● Algorithm design is all about the mathematical theory behind the design of good
programs.
● Algorithmic is a branch of computer science that consists of designing and analyzing
computer algorithms.
● The “design” pertain to
● The description of algorithm at an abstract level by means of a pseudo language, and
● Proof of correctness that is, the algorithm solves the given problem in all cases.
● The “analysis” deals with performance evaluation (complexity analysis).
● Random Access Machine (RAM) model.
Euclid’s Algorithm

Problem: Find gcd(m,n), the greatest common divisor of two nonnegative, not both zero
integers m and n.
● Examples: gcd(60,24) = 12, gcd(60,0) = 60, gcd(0,0) = ?
● Euclid’s algorithm is based on repeated application of equality
- gcd(m,n) = gcd(n, m mod n)
until the second number becomes 0, which makes the problem trivial.
● Example: gcd(60,24) = gcd(24,12) = gcd(12,0) = 12.

Two descriptions of Euclid’s algorithm

Step 1 If n = 0, return m and stop; otherwise go to Step 2

Step 2 Divide m by n and assign the value for the remainder to r
Step 3 Assign the value of n to m and the value of r to n. Go to Step 1.

ALGORITHM Euclid(m, n)
//Computes gcd(m, n) by Euclid’s algorithm
//Input: Two nonnegative, not-both-zero integersm and n
//Output: Greatest common divisor of m and n
whilen _= 0 do
r←m mod n
m←n
n←r
returnm

Consecutive integer checking algorithm for computing gcd(m, n)

Step 1 Assign the value of min{m, n} to t.
Step 2 Divide m by t.If the remainder of this division is 0, go to Step 3;
Otherwise, go to Step 4.
Step 3 Divide n by t. If the remainder of this division is 0, return the value of t
as the answer and stop; otherwise, proceed to Step 4.
Step 4 Decrease the value of t by 1. Go to Step 2.
Middle-school procedure for computing gcd(m, n)
Step 1 Find the prime factors of m.
Step 2 Find the prime factors of n.
Step 3 Identify all the common factors in the two prime expansions found in
Step 1 and Step 2. (If p is a common factor occurring pm and pn times in m and n,
respectively, it should be repeated min{pm, pn} times.)
Step 4 Compute the product of all the common factors and return it as the greatest common
divisor of the numbers given.
Thus, for the numbers 60 and 24, we get
60 = 2 .2 .3 .5
24 = 2 .2 .2 .3
gcd(60, 24) = 2 . 2 .3 = 12.
ALGORITHM Sieve(n)
//Implements the sieve of Eratosthenes
//Input: A positive integer n >1
//Output: Array L of all prime numbers less than or equal to n
forp←2 to n do A[p]←p
forp←2 to_√n_do //see note before pseudocode
ifA[p] _=0 //p hasn’t been eliminated on previous passes
j← p ∗p
whilej ≤ n do
A[j ]←0 //mark element as eliminated
j←j + p
//copy the remaining elements of A to array L of the primes
i←0
forp←2 to n do
ifA[p] _= 0
L[i]←A[p]
i←i + 1
returnL

Example: 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

1.2 FUNDAMENTALS OF ALGORITHMIC PROBLEM SOLVING

What does it mean to understand the problem?

How the objects would be represented?

How the operations would be implemented?
Design an algorithm

Build a computational model of the solving process

Prove correctness: correct output for every legitimate input in finite time

Based on correct math formula

By induction

Tracing can prove incorrectness, it cannot prove correctness in the general case.

Analyze the algorithm

Efficiency: time and space

Simplicity
Generality (range of inputs, special cases)
Optimality

Optimal algorithm: no other algorithm can do better.

Coding

How the objects and operations in the algorithm are represented in the chosen programming
language? Can every problem be solved by an algorithm?

Algorithm Problem solving

● Understand the problem

● Decide on
• Computational Means
• Exact vs. Approximate solving
• Data structures
• Algorithm design techniques
● Design an algorithm
● Prove correctness
● Analyse an algorithm
● Code an algorithm

Understand the problem

● First important step

● What must be done rather than how to do it
● Input-instance of the problem
● Ascertaining the capability of computational device
● Generic one-processor, random-access-machine (RAM)-instructions are executed one
by one.
● Computer resources-memory, BW, CPU measures efficiency of algorithm.

Choosing between exact and approximate problem solving

● An algorithm that can solve the problem exactly is called an exact algorithm. The
algorithm that can solve the problem approximately is called an approximation algorithm.
The problems that can be approximately are
• extracting square roots
• Solving non-linear equations
• Evaluate definite integrals
for some, algorithm for solving a problem exactly is not acceptable because it can be slow
due to its intrinsic complexity of that problem. For ex, like travelling salesman problem which
finds shortest tour through n cities.

Deciding on appropriate data structures

Algorithm + Data Structures = Program

Algorithmic design technique

● An algorithm design technique is a general approach to solving problems
mathematically that is applicable to a variety of problems from different areas of computing.
● Algorithm design technique provides guidance for designing algorithms for new
problems or problems for which there is no satisfactory algorithm. It makes it possible to
classify algorithm according to underlying design idea.

Methods of specifying an algorithm

● Present: pseudocode

● Earlier: flowchart

Proving an algorithm’s correctness

● Mathematical induction for recursion

● Modus ponens

● E.g. algorithm stops since number gets smaller on each iteration

● Approximation algorithms are more difficult

Analysis of algorithms

How good is the algorithm?

● time efficiency
● space efficiency
1.3 IMPORTANT PROBLEM TYPES
● Sorting
● Searching
● String processing
● Graph problems
● Combinatorial problems
● Geometric problems
● Numerical problems

Sorting

The sorting problem is to rearrange the items of a given list in nondecreasing order.

Two properties of sorting algorithms deserve special mention:

A sorting algorithm is called stable if it preserves the relative order of any two equal elements
in its input.
-In other words, if an input list contains two equal elements in positions i and j where i < j, then
in the sorted list they have to be in positions i’ andj’, respectively, such that i’< j’.

The second notable feature of a sorting algorithm is the amount of extra memory the algorithm
requires. An algorithm is said to be in-place if it does not require extra memory, except,
possibly, for a few memory units.

Searching

The searching problem deals with finding a given value, called a search key, in a
given set.
In applications where the underlying data may change frequently relative to the Number of
searches, searching has to be considered in conjunction with two other Operations: an addition
to and deletion from the data set of an item. In such Situations, data structures and algorithms
should be chosen to strike a balance among the requirements of each operation.

String processing

A string is a sequence of characters from an alphabet. Strings of particular interest are text
strings, which comprise letters, numbers, and special characters; bit strings, which comprise
zeros and ones; and gene sequences, which can be modeled by strings of characters from the
four-character alphabet {A,C, G, T}. It should be pointed out, however, that string-processing
algorithms have been important for computer science for a long time in conjunction with
computer languages and compiling issues.
One particular problem—that of searching for a given word in a text—has attracted special
attention from researchers. They call it string matching. Several algorithms that exploit the
special nature of this type of searching have been invented.
Graph problems:
Graph can be thought of as a collection of points called vertices, some of which are connected
by line segments called edges.

The traveling salesman problem (TSP) is the problem of finding the shortest tour through n
cities that visits every city exactly once.

The graph-coloring problem seeks to assign the smallest number of colors to the vertices of
a graph so that no two adjacent vertices are the same color. This problem arises in several
applications, such as event scheduling: if the events are represented by vertices that are
connected by an edge if and only if the corresponding events cannot be scheduled at the same
time, a solution to the graph-coloring problem yields an optimal schedule.

Combinatorial problems

The travelling salesman problem and the graph-coloring problem are examples of
combinatorial problems.

combinatorial problems are the most difficult problems in computing, from both a theoretical
and practical standpoint. Their difficulty stems from the following facts. First, the number of
combinatorial objects typically grows extremely fast with a problem’s size, reaching
unimaginable magnitudes even for moderate-sized instances. Second, there are no known
algorithms for solving most such problems exactly in an acceptable amount of time.

Some combinatorial problems can be solved by efficient algorithms, but they should be
considered fortunate exceptions to the rule. The shortest-path problem mentioned earlier is
among such exceptions.

Geometric problems:

Geometric algorithms deal with geometric objects such as points, lines, and polygons.
Two classic problems of computational geometry:
The closest-pair problem is self-explanatory: given n points in the plane, find the closest pair
among them.
The convex-hull problem asks to find the smallest convex polygon that would include all the
points of a given set.

Numerical problems:

Numerical problems, another large special area of applications, are problems that involve
mathematical objects of continuous nature: solving equations and systems of equations,
computing definite integrals, evaluating functions, and so on.

1.4 FUNDAMENTALS OF ANALYSIS OF ALGORITHM EFFICIENCY

In this a general framework for analyzing the efficiency of algorithms. We already mentioned
in that there are two kinds of efficiency: time efficiency and space efficiency. Time efficiency,
also called time complexity, indicates how fast an algorithm in question runs. Space efficiency,
also called space complexity, refers to the amount of memory units required by the algorithm
in addition to the space needed for its input and output. In the early days of electronic
computing, both resources—time and space—were at a premium. Half a century of relentless
technological innovations have improved the computer’s speed and memory size by many
orders of magnitude. Now the amount of extra space required by an algorithm is typically not
of as much concern, with the caveat that there is still, of course, a difference between the fast
main memory, the slower secondary memory, and the cache. The time issue has not diminished
quite to the same extent, however. In addition, the research experience has shown that for most
problems, we can achieve much more spectacular progress in speed than in space. Therefore,
following a well-established tradition of algorithm textbooks, we primarily concentrate on time
efficiency, but the analytical framework introduced here is applicable to analyzing space
efficiency as well.

Measuring an Input’s Size

Let’s start with the obvious observation that almost all algorithms run longer on larger inputs.
For example, it takes longer to sort larger arrays, multiply larger matrices, and so on. Therefore,
it is logical to investigate an algorithm’s efficiency as a function of some parameter n indicating
the algorithm’s input size.1 In most cases, selecting such a parameter is quite straightforward.
For example, it will be the size of the list for problems of sorting, searching, finding the list’s
smallest element, and most other problems dealing with lists. For the problem of evaluating a
polynomial p(x) = anx n + . . . + a0 of degree n, it will be the polynomial’s degree or the number
of its coefficients, which is larger by 1 than its degree. You’ll see from the discussion that such
a minor difference is inconsequential for the efficiency analysis.
There are situations, of course, where the choice of a parameter indicating an input size does
matter. One such example is computing the product of two n × n matrices. There are two natural
measures of size for this problem. The first and more frequently used is the matrix order n. But
the other natural contender is the total number of elements N in the matrices being multiplied.
(The latter is also more general since it is applicable to matrices that are not necessarily square.)
The choice of an appropriate size metric can be influenced by operations of the algorithm
in question. For example, how should we measure an input’s size for a spell-checking
algorithm? If the algorithm examines individual characters of its input, we should measure the
size by the number of characters; if it works by processing words, we should count their number
in the input.
 We should make a special note about measuring input size for algorithms solving
problems such as checking primality of a positive integer n. Here, the input is just one number,
and it is this number’s magnitude that determines the input size. In such situations, it is
preferable to measure size by the number b of bits in the n’s binary representation: b = log2 n
+ 1.
This metric usually gives a better idea about the efficiency of algorithms in question.

1.5 ASYMPTOTIC NOTATIONS

Principal indicator of efficiency = Order of growth of basic operation count

A way of comparing functions that ignores constant factors and small input sizes.

O(g(n)): set of all functions f(n) with a smaller or same order of growth as g(n) (to within a
constant multiple, as n!).

"(g(n)): set of all functions f(n) with a larger or same order of growth as g(n) (to within a
constant multiple, as n!).

(g(n)): set of all functions f(n) with the same order of growth as g(n) (to within a constant
multiple, as n!).

Big-Oh Notation

Let t(n) = O(g(n)): A function t(n) is said to be in O(g(n)), if it is bounded above by some
constant multiple of g(n) for all large nThere exist positive constant c and non-negative integer
n0 such that t(n) # c g(n) for every n $ n0

Example: 100n+5 is O(n2)

Big-Omega Notation

Let t(n) =(g(n)): A function t(n) is said to be in O(g(n)), if it is bounded below by some

constant multiple of g(n) for all large n There exist positive constant c and non-negative integer
n0 such that t(n) $ c g(n) for every n $ n0

Example: 2n3-65 is (n2)

Big-Theta Notation

Let t(n) =(g(n)): A function t(n) is said to be in (g(n)), if it is bounded both above and below
by some constant multiples of g(n) for all large nThere exist positive constants c1

andc2 and non-negative integer n0 such that c2g(n) $ t(n) $ c1g(n) for every n $ n0

Example: (1/2)n(n-1) is (n2)

Comparing Growth Rate: Using Limits

Compute the limit of the ratio of two functions under consideration using the limit-based
approach is more convenient than the one based in the definition.

Basic Asymptotic Efficiency Classes

Caution: In defining asymptotic efficiency classes, the values of multiplicative constants are
usually left unspecified.

CONDITIONAL ASYMPTOTIC NOTATION

Let f be an eventually-nonnegative sequence of real numbers and let X be a set. The function
O( f |X) yields a non empty set of functions from N to R and is defined as follows:

O( f |X) = {t;t ranges over elements of RN:

Wc,N (c > 0 ^Vn (n _ N ^ n 2 X ) t(n) _ c · f (n) ^ t(n) _ 0))}.

Let f be an eventually-nonnegative sequence of real numbers and let X be a set. The function

( f|X) yielding a non empty set of functions from N to R is defined as follows:

(f|X) = {t;t ranges over elements of RN:

Wd,N (d > 0 ^Vn (n _ N ^ n 2 X ) t(n) _ d · f (n) ^ t(n) _ 0))}.

Let f be an eventually-nonnegative sequence of real numbers and let X be a set. The function (
f|X) yields a non empty set of functions from N to R and is defined by the condition (f |X) =
{t;t ranges over elements of RN:

Wc,d,N (c > 0 ^ d > 0 ^Vn (n _ N ^ n 2 X ) d · f (n) _ t(n) ^ t(n) _ c · f (n)))}.

The following proposition is true.For every eventually-nonnegative sequence f of real numbers

and for every set X holds

( f|X) = O( f |X)\ ( f |X).

Let f be a sequence of real numbers and let b be a natural number. The functor fb yielding a
sequence of real numbers is defined as follows:

For every n holds fb(n) = f (b ·n).

Let f be an eventually-nonnegative sequence of real numbers and let b be a natural number. We

say that f is smooth w.r.t. b if and only if:

f is eventually-nondecreasing and fb 2 O( f ).

Let f be an eventually-nonnegative sequence of real numbers. We say that f is smooth if and

only if:
For every natural number b such that b _ 2 holds f is smooth w.r.t. b.

REMOVING CONDITION FROM THE CONDITIONAL ASYMPTOTIC

NOTATION

Let f be an eventually-nonnegative sequence of real numbers. Given a natural number b such

that b _ 2 and f is smooth w.r.t. b. Then f is smooth.

Let f be an eventually-nonnegative sequence of real numbers, t be an eventually non-negative

eventually-nondecreasing sequence of real numbers, and b be a natural number.

Suppose f is smooth and b _ 2 and t 2 O( f |{bn : n ranges over natural numbers}). Then

t 2 O( f ).Let f be an eventually-nonnegative sequence of real numbers, t be an eventually non-

negative eventually-non decreasing sequence of real numbers, and b be a natural
number.Suppose f is smooth and b _ 2 and t 2 ( f |{bn : n ranges over natural numbers}).
Then t2 ( f ).

Let f be an eventually-nonnegative sequence of real numbers, t be an eventually non negative

eventually-non decreasing sequence of real numbers, and b be a natural number.Suppose f is
smooth and b _ 2 and t 2 ( f |{bn : n ranges over natural numbers}). Then t2 ( f ).

PROPERTIES OF BIG-OH NOTATION

Big O notation (also known as Big Oh notation, Landau notation, Bachmann–Landau

notation, and asymptotic notation) describes the limiting behavior of a function when the
argument tends towards a particular value or infinity, usually in terms of simpler functions. Big
O notation allows its users to simplify functions in order to concentrate on their growth rates:
different functions with the same growth rate may be represented using the same O notation.

This notation is now frequently also used in computational complexity theory to describe an
algorithm's usage of computational resources: the worst case or average case running time or
memory usage of an algorithm is often expressed as a function of the length of its input using
big O notation. This allows algorithm designers to predict the behavior of their algorithms and
to determine which of multiple algorithms to use, in a way that is independent of computer
architecture or clock rate. Big O notation is also used in many other fields to provide similar
estimates.
A description of a function in terms of big O notation usually only provides an upper bound on
the growth rate of the function; associated with big O notation are several related notations,
using the symbols o, Ω, ω, and Θ, to describe other kinds of bounds on asymptotic growth
rates.

If a function f(n) can be written as a finite sum of other functions, then the fastest growing one
determines the order of f(n).

When we defined the Rational class, we didn't reduce the fractions, so we ended up with
numbers like 10/8 and 4/6. To avoid this, we could reduce the fractions in the constructor by
dividing both the numerator and the denominator by the GCD of the two.

The mathematical definition is:

The greatest common divisor (GCD) of two integersm and n is the greatest integer that
divides both m and n with no remainder.

So, we'd like a procedure

Thus, the problem is:

Given integers m and n such that m>= n> 0, find the GCD of m and n.

To solve this problem, the mathematical definition isn't enough. It tells us what we want, but
not how to get it. Before we can write a procedure, we need to know how to compute the
value.

We need an algorithm: a method for computing process.

v v
PROCE
PROCEDURE s ALGORITHM s
SS
. .

step by step description of the the idea of how to perform the

activity
activity activity

(The same algorithm can be described in many different ways.)

So far, the procedures we have written contained only simple formulae and occasional
conditional statements. We have essentially translated the specifications directly into code.
Now we need to come up with an algorithm, a way to compute the results that does not fall
out immediately from the statement of the problem.

Before considering possible GCD algorithms, let's design algorithms for some simpler
problems.

Example: factorial
Recall that n! = n*(n-1)*(n-2)*...*2*1, and that 0! = 1. In other words,

We want to build a procedure

Let's try to devise an algorithm straight from the mathematical definition.

int factorial(int n) {
if (n == 0)
return 1;
else
return (n * factorial(n-1));
}

Will this work? Let's try using the substitution model.

factorial(0)
=> 1

factorial(3)
3 * factorial(2)
3 * 2 * factorial(1)
3 * 2 * 1 * factorial(0)
3*2*1*1
=> 6

This looks good. We've done a reduction of the factorial problem to a smaller instance of the
same problem. This idea of reducing a problem to itself is known as recursion.

Let's pick apart the code:

int factorial(int n) {
if (n == 0) // Termination condition (base case)
return 1; // Result for base case
else
return (n * factorial(n-1));
// ^ --------------
// | ^
// | |___ Recursive call
// |
// |___ Combining step (used to combine other values with
// results of the recursive call)
}

● Always identify the base case and associated result first.

● Make sure the recursive call is for a smaller problem (one "closer" to the base case)

Another way to think about the execution of a recursive procedure is with the "actors" model
(or dataflow model). In this simple model, we draw a box for each procedure invocation and
show the flow of data into and out of each box. To do this, we put the output arrow on the left
with the input.

For example,

1.6 ANALYSIS FRAMEWORK

Run-time analysis

Run-time analysis is a theoretical classification that estimates and anticipates the increase in
running time (or run-time) of an algorithm as its input size (usually denoted as n) increases.
Run-time efficiency is a topic of great interest in computer science: A program can take
seconds, hours or even years to finish executing, depending on which algorithm it implements
(see also performance analysis, which is the analysis of an algorithm's run-time in practice).

Shortcomings of empirical metrics

Since algorithms are platform-independent (i.e. a given algorithm can be implemented in an

arbitrary programming language on an arbitrary computer running an arbitrary operating
system), there are significant drawbacks to using an empirical approach to gauge the
comparative performance of a given set of algorithms.

Take as an example a program that looks up a specific entry in a sortedlist of size n. Suppose
this program were implemented on Computer A, a state-of-the-art machine, using a linear
search algorithm, and on Computer B, a much slower machine, using a binary search algorithm.
Benchmark testing on the two computers running their respective programs might look
something like the following:

Computer A run- Computer B run-

n (list
time time
size)
(in nanoseconds) (in nanoseconds)

15 7 100,000

65 32 150,000

250 125 200,000

1,000 500 250,000

Based on these metrics, it would be easy to jump to the conclusion that Computer A is
running an algorithm that is far superior in efficiency to that of Computer B. However, if the
size of the input-list is increased to a sufficient number, that conclusion is dramatically
demonstrated to be in error:

Computer A run- Computer B run-

n (list size) time time
(in nanoseconds) (in nanoseconds)

15 7 100,000

65 32 150,000

250 125 200,000

1,000 500 250,000

... ... ...

1,000,000 500,000 500,000

4,000,000 2,000,000 550,000

16,000,000 8,000,000 600,000

... ... ...

63,072 × 31,536 × 1012 ns, 1,375,000 ns,

1012 or 1 year or 1.375 milliseconds

Computer A, running the linear search program, exhibits a linear growth rate. The program's
run-time is directly proportional to its input size. Doubling the input size doubles the run
time, quadrupling the input size quadruples the run-time, and so forth. On the other hand,
Computer B, running the binary search program, exhibits a logarithmic growth rate. Doubling
the input size only increases the run time by a constant amount (in this example, 50,000 ns).
Even though Computer A is ostensibly a faster machine, Computer B will inevitably surpass
Computer A in run-time because it's running an algorithm with a much slower growth rate.

Orders of growth
Main article: Big O notation

Informally, an algorithm can be said to exhibit a growth rate on the order of a mathematical
function if beyond a certain input size n, the function f(n) times a positive constant provides an
upper bound or limit for the run-time of that algorithm. In other words, for a given input size n
greater than some n0 and a constant c, the running time of that algorithm will never be larger
than c × f(n). This concept is frequently expressed using Big O notation. For example, since
the run-time of insertion sortgrows quadratically as its input size increases, insertion sort can
be said to be of order O(n²).

Big O notation is a convenient way to express the worst-case scenario for a given algorithm,
although it can also be used to express the average-case — for example, the worst-case scenario
for quicksort is O(n²), but the average-case run-time is O(n log n).[7]
Empirical orders of growth

Assuming the execution time follows power rule, t ≈ k na, the coefficient a can be found [8] by
taking empirical measurements of run time at some problem-size points ,
and calculating so that . If the order of
growth indeed follows the power rule, the empirical value of a will stay constant at different
ranges, and if not, it will change - but still could serve for comparison of any two given
algorithms as to their empirical local orders of growth behaviour. Applied to the above table:

Computer A run- Local order of Computer B run- Local order of

n (list
time growth time growth
size)
(in nanoseconds) (n^_) (in nanoseconds) (n^_)

15 7 100,000

65 32 1.04 150,000 0.28

250 125 1.01 200,000 0.21

1,000 500 1.00 250,000 0.16

... ... ...

1,000,00
500,000 1.00 500,000 0.10
0

4,000,00
2,000,000 1.00 550,000 0.07
0

16,000,0
8,000,000 1.00 600,000 0.06
00

... ... ...

It is clearly seen that the first algorithm exhibits a linear order of growth indeed following the
power rule. The empirical values for the second one are diminishing rapidly, suggesting it
follows another rule of growth and in any case has much lower local orders of growth (and
improving further still), empirically, than the first one.
Evaluating run-time complexity

The run-time complexity for the worst-case scenario of a given algorithm can sometimes be
evaluated by examining the structure of the algorithm and making some simplifying
assumptions. Consider the following pseudocode:

get a positive integer from input

if n > 10
print "This might take a while..."
for i = 1 to n
for j = 1 to i
print i * j
print "Done!"

A given computer will take a discrete amount of time to execute each of the instructions
involved with carrying out this algorithm. The specific amount of time to carry out a given
instruction will vary depending on which instruction is being executed and which computer is
executing it, but on a conventional computer, this amount will be deterministic.[9] Say that the
actions carried out in step 1 are considered to consume time T1, step 2 uses time T2, and so
forth.

In the algorithm above, steps 1, 2 and 7 will only be run once. For a worst-case evaluation, it
should be assumed that step 3 will be run as well. Thus the total amount of time to run steps
1-3 and step 7 is:

The loops in steps 4, 5 and 6 are trickier to evaluate. The outer loop test in step 4 will execute
( n + 1 ) times (note that an extra step is required to terminate the for loop, hence n + 1 and
not n executions), which will consume T4( n + 1 ) time. The inner loop, on the other hand, is
governed by the value of i, which iterates from 1 to i. On the first pass through the outer loop,
j iterates from 1 to 1: The inner loop makes one pass, so running the inner loop body (step 6)
consumes T6 time, and the inner loop test (step 5) consumes 2T5 time. During the next pass
through the outer loop, j iterates from 1 to 2: the inner loop makes two passes, so running the
inner loop body (step 6) consumes 2T6 time, and the inner loop test (step 5) consumes 3T5
time.

Altogether, the total time required to run the inner loop body can be expressed as an
arithmetic progression:

which can be factored[10] as

The total time required to run the outer loop test can be evaluated similarly:

which can be factored as

Therefore the total running time for this algorithm is:

which reduces to

As a rule-of-thumb, one can assume that the highest-order term in any given function
dominates its rate of growth and thus defines its run-time order. In this example, n² is the
highest-order term, so one can conclude that f(n) = O(n²). Formally this can be proven as
follows:
Prove that

(for n ≥ 0)

Let k be a constant greater than or equal to [T1..T7]

(for n ≥ 1)

Therefore

for

A more elegant approach to analyzing this algorithm would be to declare that [T1..T7] are all
equal to one unit of time, in a system of units chosen so that one unit is greater than or equal
to the actual times for these steps. This would mean that the algorithm's running time breaks
down as follows:[11]

(for n ≥ 1)

1.7 EMPIRICAL ANALYSIS

Empirical analysis is a type of research dedicated to the discovery of concrete, verifiable

evidence. Guided by the scientific method, empirical analysis allows researchers to remove
personal bias and instead use concrete, accurate and repeatable real-world evidence to draw
conclusions.

we saw how algorithms, both nonrecursive and recursive, can be analyzed

mathematically. Though these techniques can be applied success-fully to many simple
algorithms, the power of mathematics, even when enhanced with more advanced techniques
(see [Sed96], [Pur04], [Gra94], and [Gre07]), is far from limitless. In fact, even some
seemingly simple algorithms have proved to be very difficult to analyze with mathematical
precision and certainty. As we pointed out in Section 2.1, this is especially true for the average-
case analysis.

The principal alternative to the mathematical analysis of an algorithm’s ef-ficiency is its

empirical analysis. This approach implies steps spelled out in the following plan.

General Plan for the Empirical Analysis of Algorithm Time Efficiency

• Understand the experiment’s purpose.

• Decide on the efficiency metric M to be measured and the measurement unit (an
operation count vs. a time unit).
• Decide on characteristics of the input sample (its range, size, and so on).
• Prepare a program implementing the algorithm (or algorithms) for the
• experimentation.
• Generate a sample of inputs.
• Run the algorithm (or algorithms) on the sample’s inputs and record the data observed.
• Analyze the data obtained.

Let us discuss these steps one at a time. There are several different goals one can pursue in
analyzing algorithms empirically. They include checking the accuracy of a theoretical assertion
about the algorithm’s efficiency, comparing the efficiency of several algorithms for solving the
same problem or different implementations of the same algorithm, developing a hypothesis
about the algorithm’s efficiency class, and ascertaining the efficiency of the program
implementing the algorithm on a particular machine. Obviously, an experiment’s design should
de-pend on the question the experimenter seeks to answer.
In particular, the goal of the experiment should influence, if not dictate, how the algorithm’s
efficiency is to be measured. The first alternative is to insert a counter (or counters) into a
program implementing the algorithm to count the number of times the algorithm’s basic
operation is executed. This is usually a straightforward operation; you should only be mindful
of the possibility that the basic operation is executed in several places in the program and that
all its executions need to be accounted for. As straightforward as this task usually is, you should
always test the modified program to ensure that it works correctly, in terms of both the problem
it solves and the counts it yields.

The second alternative is to time the program implementing the algorithm in question. The
easiest way to do this is to use a system’s command, such as the time command in UNIX.
Alternatively, one can measure the running time of a code fragment by asking for the system
time right before the fragment’s start (tstart) and just after its completion (tfinish), and then
computing the difference between thetwo (tfinish− tstart).7 In C and C++, you can use the function
clock for this purpose; in Java, the method currentTimeMillis() in the System class is available.

It is important to keep several facts in mind, however. First, a system’s time is typically not
very accurate, and you might get somewhat different results on repeated runs of the same
program on the same inputs. An obvious remedy is to make several such measurements and
then take their average (or the median) as the sample’s observation point. Second, given the
high speed of modern com-puters, the running time may fail to register at all and be reported
as zero. The standard trick to overcome this obstacle is to run the program in an extra loop
many times, measure the total running time, and then divide it by the number of the loop’s
repetitions. Third, on a computer running under a time-sharing system such as UNIX, the
reported time may include the time spent by the CPU on other programs, which obviously
defeats the purpose of the experiment. Therefore, you should take care to ask the system for
the time devoted specifically to execution of your program. (In UNIX, this time is called the
“user time,” and it is automatically provided by the time command.)

Thus, measuring the physical running time has several disadvantages, both principal
(dependence on a particular machine being the most important of them) and technical, not
shared by counting the executions of a basic operation. On the other hand, the physical running
time provides very specific information about an algorithm’s performance in a particular
computing environment, which can be of more importance to the experimenter than, say, the
algorithm’s asymptotic efficiency class. In addition, measuring time spent on different
segments of a program can pinpoint a bottleneck in the program’s performance that can be
missed by an abstract deliberation about the algorithm’s basic operation. Getting such data—
called profiling—is an important resource in the empirical analysis of an algorithm’s running
time; the data in question can usually be obtained from the system tools available in most
computing environments.

Whether you decide to measure the efficiency by basic operation counting or by time clocking,
you will need to decide on a sample of inputs for the experiment. Often, the goal is to use a
sample representing a “typical” input; so the challenge is to understand what a “typical” input
is. For some classes of algorithms—e.g., for algorithms for the traveling salesman problem that
we are going to discuss later in the book—researchers have developed a set of instances they
use for benchmark-ing. But much more often than not, an input sample has to be developed by
the experimenter. Typically, you will have to make decisions about the sample size (it is
sensible to start with a relatively small sample and increase it later if necessary), the range of
instance sizes (typically neither trivially small nor excessively large), and a procedure for
generating instances in the range chosen. The instance sizes can either adhere to some pattern
(e.g., 1000, 2000, 3000, . . . , 10,000 or 500, 1000, 2000, 4000, . . . , 128,000) or be generated
randomly within the range chosen.

The principal advantage of size changing according to a pattern is that its impact is easier to
analyze. For example, if a sample’s sizes are generated by doubling, you can compute the ratios
M(2n)/M(n) of the observed metric M (the count or the time) to see whether the ratios exhibit
a behavior typical of algorithms in one of the basic efficiency classes discussed in Section 2.2.
The major disadvantage of nonrandom sizes is the possibility that the algorithm under
investigation exhibits atypical behavior on the sample chosen. For example, if all the sizes in
a sample are even and your algorithm runs much more slowly on odd-size inputs, the empirical
results will be quite misleading.

Another important issue concerning sizes in an experiment’s sample is whether several

instances of the same size should be included. If you expect the observed metric to vary
considerably on instances of the same size, it would be probably wise to include several
instances for every size in the sample. (There are well-developed methods in statistics to help
the experimenter make such de-cisions; you will find no shortage of books on this subject.) Of
course, if several instances of the same size are included in the sample, the averages or medians
of the observed values for each size should be computed and investigated instead of or in
addition to individual sample points.

Much more often than not, an empirical analysis requires generating random numbers. Even if
you decide to use a pattern for input sizes, you will typically want instances themselves
generated randomly. Generating random numbers on a digital computer is known to present a
difficult problem because, in principle, the problem can be solved only approximately. This is
the reason computer scien-tists prefer to call such numbers pseudorandom. As a practical
matter, the easiest and most natural way of getting such numbers is to take advantage of a
random number generator available in computer language libraries. Typically, its output will
be a value of a (pseudo)random variable uniformly distributed in the interval between 0 and 1.
If a different (pseudo)random variable is desired, an appro-priate transformation needs to be
made. For example, if x is a continuous ran-dom variable uniformly distributed on the interval
0 ≤ x < 1, the variable y = l+ x(r − l) will be uniformly distributed among the integer values
between integers l and r − 1 (l < r).

Alternatively, you can implement one of several known algorithms for gener-ating
(pseudo)random numbers. The most widely used and thoroughly studied of such algorithms is
the linear congruential method.

ALGORITHM Random(n, m, seed, a, b)

//Generates a sequence of n pseudorandom numbers according to the linear congruential

method

//Input: A positive integer n and positive integer parameters m, seed, a, b //Output: A sequence
r1, . . . , rn of n pseudorandom integers uniformly distributed among integer values between 0
and m − 1 //Note: Pseudorandom numbers between 0 and 1 can be obtained by treating the
integers generated as digits after the decimal point

r0 ← seed
for i ← 1 to n do
ri ← (a ∗ ri−1 + b) mod m

The simplicity of this pseudocode is misleading because the devil lies in the details of choosing
the algorithm’s parameters. Here is a partial list of recommen-dations based on the results of a
sophisticated mathematical analysis (see [KnuII, pp. 184–185] for details): seed may be chosen
arbitrarily and is often set to the current date and time; m should be large and may be
conveniently taken as 2w, where w is the computer’s word size; a should be selected as an
integer between 0.01m and 0.99m with no particular pattern in its digits but such that a mod 8
= 5; and the value of b can be chosen as 1.

The empirical data obtained as the result of an experiment need to be recorded and then
presented for an analysis. Data can be presented numerically in a table or graphically in a
scatterplot, i.e., by points in a Cartesian coordinate system. It is a good idea to use both these
options whenever it is feasible because both methods have their unique strengths and
weaknesses.

The principal advantage of tabulated data lies in the opportunity to manip-ulate it easily. For
example, one can compute the ratios M(n)/g(n) where g(n) is a candidate to represent the
efficiency class of the algorithm in question. If the algorithm is indeed in (g(n)), most likely
these ratios will converge to some pos-itive constant as n gets large. (Note that careless novices
sometimes assume that this constant must be 1, which is, of course, incorrect according to the
definition of (g(n)).) Or one can compute the ratios M(2n)/M(n) and see how the running time
reacts to doubling of its input size. As we discussed in Section 2.2, such ratios should change
only slightly for logarithmic algorithms and most likely converge to 2, 4, and 8 for linear,
quadratic, and cubic algorithms, respectively—to name the most obvious and convenient cases.

On the other hand, the form of a scatterplot may also help in ascertaining the algorithm’s
probable efficiency class. For a logarithmic algorithm, the scat-terplot will have a concave
shape (Figure 2.7a); this fact distinguishes it from all the other basic efficiency classes. For a
linear algorithm, the points will tend to aggregate around a straight line or, more generally, to
be contained between two straight lines (Figure 2.7b). Scatterplots of functions in (n lg n)
and (n2) will have a convex shape (Figure 2.7c), making them difficult to differentiate. A
scatterplot of a cubic algorithm will also have a convex shape, but it will show a much more
rapid increase in the metric’s values. An exponential algorithm will most probably require a
logarithmic scale for the vertical axis, in which the val-ues of loga M(n) rather than those of
M(n) are plotted. (The commonly used logarithm base is 2 or 10.) In such a coordinate system,
a scatterplot of a truly exponential algorithm should resemble a linear function because M(n)
≈ can im-plies logb M(n) ≈ logb c + n logb a, and vice versa.
One of the possible applications of the empirical analysis is to predict the al-gorithm’s
performance on an instance not included in the experiment sample. For example, if you observe
that the ratios M(n)/g(n) are close to some constant c for the sample instances, it could be
sensible to approximate M(n) by the prod-uct cg(n) for other instances, too. This approach
should be used with caution, especially for values of n outside the sample range.
(Mathematicians call such predictions extrapolation, as opposed to interpolation, which deals
with values within the sample range.) Of course, you can try unleashing the standard tech-
niques of statistical data analysis and prediction. Note, however, that the majority of such
techniques are based on specific probabilistic assumptions that may or may not be valid for the
experimental data in question.

It seems appropriate to end this section by pointing out the basic differ-ences between
mathematical and empirical analyses of algorithms. The princi-pal strength of the mathematical
analysis is its independence of specific inputs; its principal weakness is its limited applicability,
especially for investigating the average-case efficiency. The principal strength of the empirical
analysis lies in its applicability to any algorithm, but its results can depend on the particular
sample of instances and the computer used in the experiment.
Exercises 2.6

1. Consider the following well-known sorting algorithm, which is studied later in the
book, with a counter inserted to count the number of key comparisons.

ALGORITHM SortAnalysis(A[0..n − 1])

//Input: An array A[0..n − 1] of n orderable elements

//Output: The total number of key comparisons made count ← 0

for i ← 1 to n − 1 do

v ← A[i] j ← i − 1

while j ≥ 0 and A[j ] > v do count ← count + 1

A[j + 1] ← A[j ] j ← j − 1

A[j + 1] ← v return count

Is the comparison counter inserted in the right place? If you believe it is, prove it; if you believe
it is not, make an appropriate correction.
 2. a. Run the program of Problem 1, with a properly inserted counter (or coun-ters) for
the number of key comparisons, on 20 random arrays of sizes 1000, 2000, 3000, . . . , 20,000.

Analyze the data obtained to form a hypothesis about the algorithm’s average-case
efficiency.

Estimate the number of key comparisons we should expect for a randomly generated
array of size 25,000 sorted by the same algorithm.

3. Repeat Problem 2 by measuring the program’s running time in milliseconds.

4. Hypothesize a likely efficiency class of an algorithm based on the following empirical

observations of its basic operation’s count:

5. What scale transformation will make a logarithmic scatterplot look like a linear one?

6. How can one distinguish a scatterplot for an algorithm in (lg lg n) from a scatterplot
for an algorithm in (lg n)?

7. a. Find empirically the largest number of divisions made by Euclid’s algo-rithm for
computing gcd(m, n) for 1≤ n ≤ m ≤ 100.

For each positive integer k, find empirically the smallest pair of integers 1≤ n ≤ m ≤
100 for which Euclid’s algorithm needs to make k divisions in order to find gcd(m, n)

1.8 MATHEMATICAL ANALYSIS FOR RECURSIVE AND NON-RECURSIVE

ALGORITHMS

This corresponds very closely to what actually happens on the execution stack in the
computer's memory.
 Nonrecursive Algorithms

Steps in mathematical analysis of nonrecursive algorithms:

● Decide on parameter n indicating input size

● Identify algorithm’s basic operation
● Determine worst, average, and best case for input of size n
● Set up summation for C(n) reflecting algorithm’s loop structure
● Simplify summation using standard formulas (see Appendix A)

Example: Selection sort

Input: An array A[0..n-1]

Output: Array A[0..n-1] sorted in ascending order

for i ← 0 to n-2 do
min ← i
for j = i + 1 to n – 1 do
if A[j] < A[min]
min ← j
swap A[i] and A[min]

basic operation: comparison

Inner loop:

n-1
S(i) =∑1 = (n-1) – (i + 1) + 1 = n – 1 - i
j = i+1

Outer loop:
n-2n-2n-2n-2
Op =∑ S(i) = ∑ (n – 1 – i) = ∑ (n – 1) - ∑ i
i = 0i = 0 i = 0i = 0

Basic formula:

n
∑i = n(n+1) / 2
i=0

Op = (n – 1 )(n -1 ) – (n-2)(n-1)/2 = (n – 1) [2(n – 1) – (n – 2)] / 2 = = (n – 1) n / 2 = O(n2)

Recursive Algorithms

Steps in mathematical analysis of recursive algorithms:

• Decide on parameter n indicating input size

• Identify algorithm’s basic operation
• Determine worst, average, and best case for input of size n
• Set up a recurrence relation and initial condition(s) for C(n)-the number of times the
basic operation will be executed for an input of size n (alternatively count recursive
calls)
• Solve the recurrence to obtain a closed form or estimate the order of magnitude of the
solution (see Appendix B)

Important recurrence types:

● Linear: One (constant) operation reduces problem size by one.

T(n) = T(n-1) + c
T(1) = d
Solution: T(n) = (n-1)c + d

● Quadratic: A pass through input reduces problem size by one.

T(n) = T(n-1) + cn
T(1) = d
Solution: T(n) = [n(n+1)/2 – 1] c + d

● Logarithmic: One (constant) operation reduces problem size by half.

T(n) = T(n/2) + c
T(1) = d
Solution: T(n) = c log n + d

● n log n: A pass through input reduces problem size by half.

T(n) = 2T(n/2) + cn
T(1) = d
Solution: T(n) = cn log n + d n

Example 1

n! = n*(n-1)!
0! = 1

T(n) = T(n-1) + 1
T(1) = 1

Telescoping:

T(n) = T(n-1) + 1
T(n-1) = T(n-2) + 1
T(n-2) = T(n-3) + 1
…
T(2) = T(1 ) + 1

Add the equations and cross equal terms on opposite sides:

T(n) = T(1) + (n-1) = n

Example 2: Binary search

T(n) = T(n/2) + 1
T(n/2) = T(n/4) + 1
…
T(2) = T(1) + 1

Add the equations and cross equal terms on opposite sides:

T(n) = T(1) + log(n) = O(log(n))

Master Theorem: A general divide-and-conquer recurrence

T(n) = aT(n/b) + f (n), where f (n) Θ(nk)

1. a < bk : T(n) Θ(nk)

2. a = bk : T(n) Θ(nk log n )
3. a > bk : T(n) Θ(n (log b a))

Note: the same results hold with O instead of Θ.

RECURRENCE EQUATIONS
Linear recurrence equations
A recurrence of the form c0ar + c1ar−1 + c2ar−2 . . . + ckar−k = f(r) where ci are constants
is called a linear recurrence equation of order k. Most of the above examples fall under this
class. If f(r) = 0 then it is homogeneous linear recurrence.

Homogeneous equations
Let us first determine the number of solutions. It appears that we must know the values of a1,
a2 . . . akto compute the values of the sequence according to the recurrence. In absence of this
there can be different solutions based on different boundary conditions. Given the k boundary
conditions, we can uniquelydetermine the values of the sequence. Note that this is not true for
a non-linear recurrence like

ar2 + ar−1 = 5 with a0 = 1

This observation (of unique solution) makes it somewhat easier for us to guess some solution
and verify.Let us guess a solution of the form ar = Aαr where A is some constant. This may be
justified from the solution of Example A.1. By substituting this in the homogeneous linear
recurrence and simplification, we obtain the following equation

c0αk + c1αk−1 . . . + ck = 0

This is called the characteristic equation of the recurrence relation and this degree k equation
has k roots, say α1, α2 . . . αk. If these are all distinct then the following is a solution to the
recurrence ar(h) = A1αr1+ A2αr2 + . . .Akαrk which is also called the homogeneous solution
to linear recurrence. The values of A1,A2. . .Ak can be determined from the k boundary
conditions (by solving k simultaneous equations).

When the roots are not unique, i.e. some roots have multiplicity then for multiplicity m, αn,
nαn, n2αn . . . nm−1αn are the associated solutions. This follows from the fact that if α is a
multiple root of the characteristic equation, then it is also the root of the derivative of the
equation.

Inhomogeneous equations
If f(n) 6= 0, then there is no general methodology. Solutions are known for some particular
cases, known as particular solutions. Let a(h) n be the solution by ignoring f(n) and let a(p) n
be a particular solution then it can be verified that

an= a(h) n + a(p) n

is a solution to the non-homogeneous recurrence.The following is a table of some particular
solutions

da constant B

dn B1n + B0

dn2 B2n2 + B1n + B0

edn, e, d are constants Bdn.

Here B,B0,B1,B2 are constants to be determined from initial conditions. When f(n) = f1(n) +
f2(n) is a sum of the above functions then we solve the equation for f1(n) and f2(n) separately
and then add them in the end to obtain a particular solution for the f(n).

SOLVING RECURRENCE EQUATIONS

Use a recurrence equation to specify how it is related to other terms in the sequence.

R Solve takes recurrence equations and solves them to get explicit formulas for a[n].

This solves a simple recurrence equation.

In[1]:= RSolve[{a[n] == 2 a[n-1], a[1] == 1}, a[n], n]

Out[1]=

This takes the solution and makes an explicit table of the first ten a[n].

In[2]:= Table[a[n] /. First[%], {n, 10}]

Out[2]=
Solving a recurrence equation.

This solves a recurrence equation for a geometric series.

In[3]:= RSolve[{a[n] == r a[n-1] + 1, a[1] == 1}, a[n], n]

Out[3]=

This gives the same result.

In[4]:= RSolve[{a[n+1] == r a[n] + 1, a[1] == 1}, a[n], n]

Out[4]=

This gives an algebraic solution to the Fibonacci recurrence equation.

In[5]:= RSolve[{a[n] == a[n-1] + a[n-2], a[1] == a[2] == 1}, a[n], n]

Out[5]=

RSolve can be thought of as a discrete analog of DSolve. Many of the same functions
generated in solving differential equations also appear in finding symbolic solutions to
recurrence equations.

This generates a gamma function, which generalizes the factorial.

In[6]:= RSolve[{a[n] == n a[n-1], a[1] == 1}, a[n], n]

Out[6]=

This second-order recurrence equation comes out in terms of Bessel functions.

In[7]:= RSolve[{a[n + 1] == n a[n] + a[n - 1],
a[1] == 0, a[2] == 1}, a[n], n]

Out[7]=

RSolve does not require you to specify explicit values for terms such as a[1]. Like DSolve, it
automatically introduces undetermined constants C[i] to give a general solution.

This gives a general solution with one undetermined constant.

In[8]:= RSolve[a[n] == n a[n-1], a[n], n]

Out[8]=

RSolve can solve equations that do not depend only linearly on a[n]. For nonlinear equations,
however, there are sometimes several distinct solutions that must be given. Just as for
differential equations, it is a difficult matter to find symbolic solutions to recurrence
equations, and standard mathematical functions only cover a limited set of cases.
Here is the general solution to a nonlinear recurrence equation.
In[9]:= RSolve[{a[n] == a[n + 1] a[n - 1]}, a[n], n]
Out[9]=
This gives two distinct solutions.
In[10]:= RSolve[a[n] == (a[n + 1] a[n - 1])^2, a[n], n]
Out[10]=

RSolve can solve not only ordinary difference equations in which the arguments of a differ
by integers, but also q-difference equations in which the arguments of a are related by
multiplicative factors.

This solves the -difference analog of the factorial equation.

In[11]:= RSolve[a[q n] == n a[n], a[n], n]

Out[11]=

Here is a second-order -difference equation.

In[12]:= RSolve[a[n] == a[q n] + a[n/q], a[n], n]

Out[12]=

Solving systems of recurrence equations.

This solves a system of two coupled recurrence equations.

In[13]:= RSolve[{a[n] == b[n - 1] + n, b[n] == a[n - 1] - n,

a[1] == b[1] == 1}, {a[n], b[n]}, n]

Out[13]=

Solving partial recurrence equations.

Just as one can set up partial differential equations that involve functions of several variables,
so one can also set up partial recurrence equations that involve multidimensional sequences.
Just as in the differential equations case, general solutions to partial recurrence equations can
involve undetermined functions.

This gives the general solution to a simple partial recurrence equation.

In[14]:= RSolve[a[i + 1, j + 1] == i j a[i, j], a[i, j], {i, j}]

Out[14]=
1.9 ALGORITHM VISUALIZATION

In addition to the mathematical and empirical analyses of algorithms, there is yet a third way
to study algorithms. It is called algorithm visualization and can be defined as the use of images
to convey some useful information about algorithms. That information can be a visual
illustration of an algorithm’s operation, of its per-formance on different kinds of inputs, or of
its execution speed versus that of other algorithms for the same problem. To accomplish this
goal, an algorithm visualiza-tion uses graphic elements—points, line segments, two- or three-
dimensional bars, and so on—to represent some “interesting events” in the algorithm’s
operation.

There are two principal variations of algorithm visualization:

● Static algorithm visualization

● Dynamic algorithm visualization, also called algorithm animation

Static algorithm visualization shows an algorithm’s progress through a series of still images.
Algorithm animation, on the other hand, shows a continuous, movie-like presentation of an
algorithm’s operations. Animation is an arguably more sophisticated option, which, of course,
is much more difficult to implement.

Early efforts in the area of algorithm visualization go back to the 1970s. The watershed event
happened in 1981 with the appearance of a 30-minute color sound film titled Sorting Out
Sorting. This algorithm visualization classic was produced at the University of Toronto by
Ronald Baecker with the assistance of D. Sherman [Bae81, Bae98]. It contained visualizations
of nine well-known sorting algorithms (more than half of them are discussed later in the book)
and provided quite a convincing demonstration of their relative speeds.

The success of Sorting Out Sorting made sorting algorithms a perennial fa-vorite for algorithm
animation. Indeed, the sorting problem lends itself quite naturally to visual presentation via
vertical or horizontal bars or sticks of different heights or lengths, which need to be rearranged
according to their sizes (Figure 2.8). This presentation is convenient, however, only for
illustrating actions of a typical sorting algorithm on small inputs. For larger files, Sorting Out
Sorting used the ingenious idea of presenting data by a scatterplot of points on a coordinate
plane, with the first coordinate representing an item’s position in the file and the second one
representing the item’s value; with such a representation, the process of sorting looks like a
transformation of a “random” scatterplot of points into the points along a frame’s diagonal
(Figure 2.9). In addition, most sorting algorithms
work by comparing and exchanging two given items at a time—an event that can be animated
relatively easily.

Since the appearance of Sorting Out Sorting, a great number of algorithm animations have been
created, especially after the appearance of Java and the
World Wide Web in the 1990s. They range in scope from one particular algorithm to a group
of algorithms for the same problem (e.g., sorting) or the same applica-tion area (e.g., geometric
algorithms) to general-purpose animation systems. At the end of 2010, a catalog of links to
existing visualizations, maintained under the NSF-supported AlgoVizProject, contained over
500 links. Unfortunately, a survey of existing visualizations found most of them to be of low
quality, with the content heavily skewed toward easier topics such as sorting [Sha07].

There are two principal applications of algorithm visualization: research and education.
Potential benefits for researchers are based on expectations that algo-rithm visualization may
help uncover some unknown features of algorithms. For example, one researcher used a
visualization of the recursive Tower of Hanoi algo-rithm in which odd- and even-numbered
disks were colored in two different colors. He noticed that two disks of the same color never
came in direct contact during the algorithm’s execution. This observation helped him in
developing a better non-recursive version of the classic algorithm. To give another example,
Bentley and McIlroy [Ben93] mentioned using an algorithm animation system in their work on
improving a library implementation of a leading sorting algorithm.

The application of algorithm visualization to education seeks to help students learning

algorithms. The available evidence of its effectiveness is decisively mixed. Although some
experiments did register positive learning outcomes, others failed to do so. The increasing body
of evidence indicates that creating sophisticated software systems is not going to be enough. In
fact, it appears that the level of student involvement with visualization might be more important
than specific features of visualization software. In some experiments, low-tech visualizations
prepared by students were more effective than passive exposure to sophisticated software
systems.

To summarize, although some successes in both research and education have been reported in
the literature, they are not as impressive as one might expect. A deeper understanding of human
perception of images will be required before the true potential of algorithm visualization is
fulfilled.