CS 854 – Advanced Algorithm Analysis

Instructor: Dr. Zuhair Zafar

Lecture# 1 & 2: Introduction & Overview

1
CS 854 – Advanced Algorithm Analysis

Instructor: Dr. Zuhair Zafar

Lecture# 7: Sorting Algorithms

2
Recap
• Asymptotic Notations
• Types of Asymptotic Notations
• Recurrences
• Methods to solve recurrences
• Substitution Method
• Recursion Tree Method
• Master Method

• Why do we need to solve recurrences? What is the objective

of solving recurrences?
3
Agenda
• What is an algorithm?

• Algorithm vs. Program

• Properties of Algorithms

• History of Algorithms

• Design of Algorithms

• Designing Techniques

• Algorithm Efficiency
4
General Conduct
• Be respectful of others
• Only speak at your turn and preferably raise your hand if you
want to say something
• Do not cut off others when they are talking
• Be punctual

5
Algorithm Definition
• An algorithm is a step-by-step procedure for solving a particular
problem in a finite amount of time.

• More generally, an algorithm is any well defined computational

procedure that takes collection of elements as input and
produces a collection of elements as output.

Input ALGORITHM
Some mysterious Output
X processing Y = F(X)
F: X→Y
6
Algorithm -- Examples
• Repairing a lamp
• A cooking recipe
• Calling a friend on the phone
• The rules of how to play a game
• Directions for driving from A to B
• A car repair manual
• Human Brain Project
• Internet & Communication Links (Graph)
• Matrix Multiplication

7
Algorithm vs. Program
• A computer program is an instance, or concrete representation,
for an algorithm in some programming language

• Set of instructions which the computer follows to solve a problem

Problem Algorithm: A sequence of

instructions describing how
to do a task

8
High Level
Language
Program
Solving Problems (1)
When faced with a problem:
1. First clearly define the problem

2. Think of possible solutions

3. Select the one that seems the best under the prevailing
circumstances

4. And then apply that solution

5. If the solution works as desired, fine; else go back to step 2

9
Solving Problems (2)
• It is quite common to first solve a problem for a particular case
• Then for another
• And, possibly another
• And watch for patterns and trends that emerge
• And to use the knowledge from these patterns and trends in
coming up with a general solution
• And this general solution is called …………….
“Algorithm”

10
One Problem, Many Algorithms
Problem
• The statement of the problem specifies, in general terms, the
desired input/output relationship.
Algorithm
• The algorithm describes a specific computational procedure for
achieving input/output relationship.
Example
• Sorting a sequence of numbers into non-decreasing order.
Algorithms
• Various algorithms e.g. merge sort, quick sort, heap sorts etc.

11
Problem Instances
• An input sequence is called an instance of a Problem

• A problem has many particular instances

• An algorithm must work correctly on all instances of the

problem it claims to solve

• Many interesting problems have infinitely many instances

– Since computers are finite, we usually need to limit the number and/or
size of possible instances in this case
– This restriction doesn’t prevent us from doing analysis in the abstract
12
Properties of Algorithms
• It must be composed of an ordered sequence of precise steps.

• It must have finite number of well-defined instructions /steps.

• The execution sequence of instructions should not be ambiguous.

• It must be correct.

• It must terminate.

13
Syntax & Semantics
An algorithm is “correct” if its: WARNINGS:
• Semantics are correct
• Syntax is correct 1. An algorithm can be
syntactically correct, yet
semantically incorrect –
Semantics: dangerous situation!
Colorless
• The conceptgreen
embeddedideas
in sleep furiously!
an algorithm (the soul!) 2. Syntactic correctness is
easier to check as
compared to semantic
Syntax: correctness
• The actual representation of
an algorithm (the body!)
14
Brief History
• The study of algorithms began with mathematicians and was a
significant area of work in the early years. The goal of those early
studies was to find a single, general algorithm that could solve all
problems of a single type.

• Named after 9th century Persian Muslim mathematician

Abu Abdullah Muhammad ibn Musa al-Khwarizmi who
lived in Baghdad and worked at the Dar al-Hikma

• Dar al-Hikma acquired & translated books on science & philosophy,

particularly those in Greek, as well as publishing original research.

• The word algorism originally referred only to the rules of

15
performing arithmetic using Hindu-Arabic numerals, but later
evolved to include all definite procedures for solving problems.
Al-Khwarizmi’s Golden Principle
All complex problems can be and must be solved
using the following simple steps:

1. Break down the problem into small, simple sub-problems

2. Arrange the sub-problems in such an order that each of them

can be solved without effecting any other

3. Solve them separately, in the correct order

4. Combine the solutions of the sub-problems to form the

solution of the original problem

16
Designing of Algorithms
• Selecting the basic approaches to the solution of the problem
• Choosing data structures
• Putting the pieces of the puzzle together
• Expressing and implementing the algorithm
• clearness, conciseness, effectiveness, etc.

Major Factors in Algorithms Design

• Correctness: An algorithm is said to be correct if for every input, it
halts with correct output. An incorrect algorithm might not halt at
all OR it might halt with an answer other than desired one. Correct
algorithm solves a computational problem
• Algorithm Efficiency: Measuring efficiency of an algorithm
17
• do its analysis i.e. growth rate.
• Compare efficiencies of different algorithms for the same problem.
Designing of Algorithms
• Most basic and popular algorithms are search and sort algorithms
• Which algorithm is the best?
• Depends upon various factors, for example in case of sorting
• The number of items to be sorted
• The extent to which the items are already sorted
• Possible restrictions on the item values
• The kind of storage device to be used etc.

18
Important Designing Techniques
• Brute Force–Straightforward, naive approach–Mostly expensive

• Divide-and-Conquer –Divide into smaller sub-problems

• e.g merge sort

• Iterative Improvement–Improve one change at a time.

• e.g greedy algorithms

• Decrease-and-Conquer–Decrease instance size

• e.g fibonacci sequence

• Transform-and-Conquer–Modify problem first and then solve it

• e.g repeating numbers in an array
19

• Dynamic programming–Dependent sub-problems, reuse results

 Algorithm Efficiency
• Several possible algorithms exist that can solve a particular problem
• each algorithm has a given efficiency
• compare efficiencies of different algorithms for the same problem
• The efficiency of an algorithm is a measure of the amount of
resources consumed in solving a problem of size n
• Running time (number of primitive steps that are executed)
• Memory/Space
• Analysis in the context of algorithms is concerned with predicting
the required resources
• There are always tradeoffs between these two efficiencies
• allow one to decrease the running time of an algorithm solution by
increasing space to store and vice-versa
• Time is the resource of most interest 20
• By analyzing several candidate algorithms, the most efficient one(s)
can be identified
 Algorithm Efficiency
Two goals for best design practices:
1. To design an algorithm that is easy to understand, code, debug.
2. To design an algorithm that makes efficient use of the computer’s
resources.
How do we improve the time efficiency of a program?
The 90/10 Rule
• 90% of the execution time of a program is spent in executing 10% of
the code. So, how do we locate the critical 10%?
• software metrics tools
• global counters to locate bottlenecks (loop executions, function calls)

Time Efficiency improvement

• Good programming: Move code out of loops that does not belong there
• Remove any unnecessary I/O operations
• Replace an inefficient algorithm (best solution)
21
Moral - Choose the most appropriate algorithm(s) BEFORE
program implementation