Chapter - 3
Chapter - 3
01/30/2024 by Kelemu L. 1
Greedy algorithm
• Greedy design technique is primarily used in Optimization problems.
• Optimization problems are problems where in we would like to find the best of all possible
solutions (a solution which has the optimal (maximum maximization or minimum
minimization) value satisfying the given constraints ).
• It helps in constructing a solution for a problem through a sequence of steps
(each step can be a partial solution progressively extended to get the
complete solution )
• Each step chosen has to satisfy the constraints given in the problem.
• Each step is chosen such that it is the best alternative among all feasible choices that are
available at the moment.
• The choice of a step once made cannot be changed in subsequent steps.
• It works in a top-down approach.
01/30/2024 by Kelemu L. 2
Characteristics of Greedy algorithm
• For a problem to be solved using the Greedy approach, it must follow a few major
characteristics:
• There is an ordered list of resources(profit, cost, value, etc.)
• Maximum/minimum of all the resources(max profit, max value, etc.) are taken.
• These algorithms are simple, straightforward and easy to implement.
• They take decisions on the basis of information at hand without worrying about the effect
these decisions may have in the future (can be one of its drawback).
• They work in stages and never reconsider any decision.
• If an optimal solution to the problem can be found by choosing the best choice at each step
without reconsidering the previous steps once chosen, the problem can be solved using a
greedy approach.
• This property is called greedy choice property.
• For example, in the fractional knapsack problem, the maximum value/weight is taken first according to available capacity .
01/30/2024 by Kelemu L. 3
• This algorithm may not produce the best result for all the problems.
• It's because it always goes for the local best choice to produce the global best result.
• Finds best local optimal choice with the hope of finding a global optimum
• If the optimal overall solution to the problem corresponds to the optimal solution to
its subproblems, then the problem can be solved using a greedy approach.
• This property is called optimal substructure.
• Some common use cases for the greedy algorithm include
• Scheduling and Resource Allocation: to schedule jobs or allocate resources in an efficient
manner.
• Minimum Spanning Trees: to find the minimum spanning tree of a graph,
• which is the subgraph that connects all vertices with the minimum total edge weight.
• Coin Change Problem: to make change for a given amount with the minimum number of coins
• Huffman Coding: to generate a prefix-free code for data compression,
• by constructing a binary tree in a way that the frequency of each character is taken into consideration.
01/30/2024 by Kelemu L. 4
Basic terminologies
• In Greedy method the problems have 'n' inputs called as candidate set, from
which a subset is selected to form a solution for the given problem.
• Any subset that satisfies the given constraints is called a feasible solution.
• We need to find a feasible solution that maximizes or minimizes an objective function and
such solution is called an optimal solution.
• Basic structures
• Declare an empty result = 0.
• We make a greedy choice to select, If the choice is feasible add it to the final result.
• return the result.
01/30/2024 by Kelemu L. 5
• The solution is said to be a feasible solution if it satisfies the following
constraints.
• Explicit constraints: - The elements of the output set must be taken from the
input set.
• Implicit constraints:-The objective function defined in the problem.
• The goal must be achieved
• Eg. Sorting a = {5, 3, 2, 9}{1, 3, 5, 7} is feasible ?
01/30/2024 by Kelemu L. 6
Change making example
• Suppose, we want to make change for an amount ‘A’ using fewest # of currency
notes. Assume the available denominations are Birr 1, 2, 5, 10, 20, 50, 100, 500,
and 1000.
• To make a change for A=28 Birr, with the minimum number of notes, one would
first choose a note of denomination Birr 20, 5, 2 and 1.
• In the above example currency notes denomination is candidate set.
• constraint is our solution make the exact target amount of cash.
• Hence, any feasible solution i.e. sum of selected notes should be equal to target amount
• objective function is our solution should consist of the fewest number of currency
notes.
• Hence, any optimal solution which is one of the feasible solutions that optimizes the
objective function.
• There can be more than one optimal solution
01/30/2024 by Kelemu L. 7
Greedy general method
• n is the size of input a
• initially solution is empty and progressively it will be modified with feasible
solutions.
01/30/2024 by Kelemu L. 8
• Greedy method consists of 3 functions (steps).
• Select: it selects an input from array a[ ] (candidate set) and puts in the variable x.
• Feasible: it is a Boolean function which checks whether the selected input meets the
constraints or not.
• Union: if the selected input i.e. 'x' makes the solution feasible,
• then x is included in the solution and objective function get updated
01/30/2024 by Kelemu L. 9
• The choice of each step in a greedy approach is done based in the following:
• It must be feasible: it has to satisfy the problem’s constraints
• It must be locally optimal: it has to be the best local choice among all feasible
choices available on that step.
• It must be unalterable (irrevocable): once made, it cannot be changed on
subsequent steps of the algorithm
01/30/2024 by Kelemu L. 10
Knapsack problem
• A thief robbing a store finds n items, the items each worth vi Birr and weights wi
grams, where vi and wi are positive numbers. He wants to take as valuable load as
possible but he can carry at most W grams in his knapsack(bag). Which item should
he take?
• What is the constraint, feasible solution and optimal solution of the above
scenario ?
• They are two types of knapsack problem
• 0-1 knapsack problem: Here the items may not be broken into smaller pieces, so thief may
decide either to take an item or to leave to it(binary choice).
• It cannot be efficiently solved by greedy algorithm
• Fractional (General) Knapsack problem: Here thief can take the fraction of items, meaning
that the items can be broken into smaller pieces so that thief may be able to carry a fraction xi
of item i.
• This can be solved easily by greedy.
01/30/2024 by Kelemu L. 11
• If a fraction xi , 0 ≤ xi ≤ 1, of objects i is placed into the knapsack, then a profit of
pi xi is earned.
• The objective is to obtain a filling of the knapsack that maximizes the total profit earned.
• Since the knapsack capacity is m, we require the total weight of all chosen objects to be at
most m.
• Formally the problem can be stated as
• Maximize Σpixi eq 1
• Subject to ΣWi xi ≤ m eq 2
• And 0 ≤ xi ≤1, 1≤ i ≤n eq 3
01/30/2024 by Kelemu L. 13
• Solution 1 is not optimal
01/30/2024 by Kelemu L. 15
Analysis of Greedy Knapsack
• If the items are already sorted into decreasing order of vi/wi, then time
complexity is O(n)
• Time minimum required to sort the array: O(N*log(N))
• Therefore Time complexity including sorting is O(n log n)
• O(N*log(N)) + O(N) ≈ O(N*log(N)).
• Correctness: The greedy technique is one of the most efficient techniques to
solve the knapsack problem, but the major drawback is its correctness.
• The greedy technique produces sub-optimum solutions, which might not always lead to
the optimum solution.
• Ex: P: 9, 8, 5, 4, and W: 2 ,3 ,2 ,2 Capacity: 6
• The optimum solution to the above problem is 18,
• but when the greedy technique is used, it results in 17.
01/30/2024 by Kelemu L. 16
Coin Change Problem
• Problem Statement: Given coins of several denominations find out a way to give
a customer an amount with fewest number of coins.
• Example: if denominations are 1,5,10, 25 and 100 and the change required is 30,
the solutions are,
• Amount: 30
• Solutions: 3 x 10 ( 3 coins ), 6 x 5 ( 6 coins )
• 1 x 25 + 5 x 1 ( 6 coins )
• 1 x 25 + 1 x 5 ( 2 coins )
• The last solution is the optimal one as it gives us change only with 2 coins.
01/30/2024 by Kelemu L. 17
Job Scheduling algorithm
• Algorithm JOB_SCHEDULING( J, D, P )
• // Description : Schedule the jobs using the greedy approach which maximizes the profit
• // Input :
• J: Array of N jobs
• D: Array of the deadline for each job
• P: Array of profit associated with each job
• // Output : Set of scheduled job which gives maximum profit
• Sort all jobs in J in decreasing order of profit
• S ← Φ // S is set of scheduled jobs, initially it is empty
• SP ← 0 // Sum is the profit earned
• for i ← 1 to N do
• if Job J[i] is feasible then
• Schedule the job in the latest possible free slot meeting its deadline.
• S ← S ∪ J[i]
• SP ← SP + P[i]
• end
• end
01/30/2024 by Kelemu L. 18
Job Scheduling algorithm
• We are given a set of n jobs. Associated with job i is an integer deadline di ≥ 0
and a profit Pi ≥ 0.For any job i profit Pi is earned if the job is completed by its
deadline. To complete a job, one has to process job on a machine for one unit of
time. Only one machine is available for processing jobs. A feasible solution for
this problem is a subset J of jobs such that each job in this subset can be
completed by its deadline.
• Eg; Let n=4. (P1,P2,P3,P4) = (100,10,15,27) and (d1,d2,d3,d4)=(2,1,2,1). d1 =2
means first job should be completed by first 2 units of time. d2 = 1 means second
job should be completed by first 1 unit of time.
01/30/2024 by Kelemu L. 19
The feasible solutions and their values are
• The feasible solutions and their values are
Feasible processing value
solution sequence
1 (1 , 2) 2,1 110
2 (1 , 3) 1 , 3 or 3,1 115
3 (1 , 4) 4 ,1 127
4 (2 , 3) 2, 3 25
5 (3 , 4) 4,3 42
6 (1) 1 100
7 (2) 2 10
8 (3) 3 15
9 (4) 4 27
• Solution 3 is optimal.
• In this solution job 1 & 4 are produced and the value is 127.
01/30/2024 by Kelemu L. 20
Exercise
• Problem: Solve the following instance of “job scheduling with deadlines”
problem : n = 7, profits (p1, p2, p3, p4, p5, p6, p7) = (3, 5, 20, 18, 1, 6, 30) and
deadlines (d1, d2, d3, d4, d5, d6, d7) = (1, 3, 4, 3, 2, 1, 2).
• Schedule the jobs in such a way to get maximum profit.
01/30/2024 by Kelemu L. 21
Minimum Spanning Tree (MST)
• A tree is defined to be an undirected, acyclic and connected graph
• (or more simply, a graph in which there is only one path connecting each pair of vertices).
• Assume there is an undirected, connected graph G.
• A spanning tree is a sub-graph of G, is a tree, and contains all the vertices of G
• A spanning tree of a connected graph is its connected acyclic subgraph (i.e., a tree) that
contains all the vertices of the graph.
• A MST of a weighted connected graph is its spanning tree of the smallest weight,
• where the weight of a tree is defined as the sum of the weights on all its edges.
• MST is applicable for
• The design of a network: to determine the least costly paths with no cycles in this network,
thereby connecting everyone at a minimum cost.
• Finding airline routes: to optimize airline routes by finding the least costly paths with no
cycles
01/30/2024 by Kelemu L. 22
• MST
01/30/2024 by Kelemu L. 23
Prim’s Algorithm
• Prim's algorithm constructs a MST through a sequence of expanding sub- trees.
• The initial subtree in such a sequence consists of a single vertex selected arbitrarily from
the set V of the graph's vertices.
• On each iteration it expands the current tree in the greedy manner by simply
attaching to it the nearest vertex not in that tree.
• The algorithm stops after all the graph's vertices have been included in the tree
being constructed.
• Since the algorithm expands a tree by exactly one vertex on each of its iterations, the total
number of such iterations is n - 1, where n is the number of vertices in the graph.
• The tree generated by the algorithm is obtained as the set of edges.
01/30/2024 by Kelemu L. 24
• Take vertices containing the minimum edge and expand the tree by
comparing(take the minimum one) the edges connected to the tree.
01/30/2024 by Kelemu L. 25
Brief example
• Consider the connected graph below
01/30/2024 by Kelemu L. 26
Algorithm (prim’s)
• Alg
01/30/2024 by Kelemu L. 27
Analysis of Efficiency
• The efficiency of Prim’s algorithm depends on the data structures chosen for the
graph itself and for the priority queue of the set V − VT whose vertex priorities are
the distances to the nearest tree vertices.
• If a graph is represented by its weight matrix and the priority queue is implemented as an
unordered array, the algorithm’s running time will be in O(|V|2).
• Prim’s algorithm has three for loops.
• The first for loop finds the near of all nodes which require O(n) time.
• The second for loop is to find the remaining n-2 edges and the third for loop updates near of each node
after adding a vertex to MST.
• Since the third for loop is within the second for loop, it requires O(n 2) time.
• Hence, the overall time complexity of Prim’s algorithm is O(n 2).
• We can implement the priority queue as a min-heap.
• Deletion of the smallest element from and insertion of a new element into a min-heap of size n are O(log n)
operations.
• If a graph is represented by its adjacency lists and the priority queue is implemented as a
min-heap, the running time of the algorithm is in O(|E| log |V |).
01/30/2024 by Kelemu L. 28
Kruskal’s Algorithm
• This algorithm starts with a list of edges sorted in non decreasing order of
weights.
• It repeatedly adds the smallest edge to the spanning tree that does not create a
cycle (makes it difficult to implement).
• Initially, each vertex is in its own tree in the forest.
• Then, the algorithm considers each edge ordered by increasing weights.
• If the edge (u, v) connects two different trees, then (u, v) is added to the set of
edges of the MST and two trees connected by an edge (u, v) are merged in to a
single tree.
• If an edge (u, v) connects two vertices in the same tree, then edge (u, v) is
discarded.
01/30/2024 by Kelemu L. 29
Algorithm
• Step 1: Sort all edges in increasing order of their edge weights.
• Step 2: Pick the smallest edge.
• Step 3: Check if the new edge creates a cycle or loop in a spanning tree.
• Step 4: If it doesn’t form the cycle, then include that edge in MST. Otherwise, discard it.
• Step 5: Repeat from step 2 until it includes |V| - 1 edges in MST.
01/30/2024 by Kelemu L. 30
Example 1
01/30/2024 by Kelemu L. 31
Example 2
The Edges of the
Edge Weight
Graph
E F 2
F D 2
B C 3
C F 3
C D 4
B F 5
B D 6
A B 7
A C 8
01/30/2024 by Kelemu L. 32
Analysis of Efficiency
• The crucial check whether two vertices belong to the same tree can be found out
using union- find algorithms.
• The Kruskal’s algorithm first creates n trees from n vertices which is done in O(n)
time.
• Then, a heap is created in O(n) time using heapify procedure.
• The least weight edge is at the root of the heap.
• Hence, the edges are deleted one by one from the heap and either added to the MST or
discarded if it forms a cycle.
• This deletion process requires O(nlog2n).
• Efficiency of Kruskal’s algorithm is based on the time needed for sorting the edge
weights of a given graph.
• Hence, with an efficient sorting algorithm, the time efficiency of Kruskal's algorithm will be
in O (|E| log |E|).
01/30/2024 by Kelemu L. 33
Graph basic concepts
• Graphs are data structures used to represent "connections" between pairs of elements.
• These elements are called nodes.
• They represent real-life objects, persons, or entities.
• The connections between nodes are called edges.
• Can be:
• Weighted Graphs: is a graph whose edges have a "weight" or "cost".
• Undirected: if for every pair of connected nodes, you can go from one node to the other
in both directions.
• Directed: if for every pair of connected nodes, you can only go from one node to
another in a specific direction.
• We use arrows instead of simple lines to represent directed edges.
01/30/2024 by Kelemu L. 34
Shortest Path Problem
• consider a number of cities connected with roads and a traveler wants to travel
form his home city A to the destination B with a minimum cost.
• Let us consider the graph G = (V, E), a weighting function w(e) for the edges in E
and a source node v0.
• The problem is to determine the shortest path from v0 to all the remaining nodes
of G.
• The solution to this problem is suggested by E.W. Dijkstra and the algorithm is
popularly known as Dijustra’s algorithm.
01/30/2024 by Kelemu L. 35
Dijkstra's Algorithm
• Dijkstra's Algorithm is the best-known algorithm for the single-source shortest-
paths problem.
• This algorithm is applicable to undirected and directed graphs with nonnegative
weights only.
• Working - Dijkstra's algorithm finds the shortest paths to a graph's vertices in
order of their distance from a given source.
• First, it finds the shortest path from the source to a vertex nearest to it, then to a second
nearest, and so on.
• The algorithm keeps track of the currently known shortest distance from each node to the
source node and it updates these values if it finds a shorter path.
• Once the algorithm has found the shortest path between the source node and another
node, that node is marked as "visited" and added to the path.
• The process continues until all the nodes in the graph have been added to the path.
01/30/2024 by Kelemu L. 36
Requirements
• Dijkstra's Algorithm can only work with graphs that have positive weights.
• If there is a negative weight in the graph, then the algorithm will not work
properly.
• The graph should be weighted graph
• The distance from the source node to itself is 0.
• The distance from the source node to all other nodes has not been determined
yet, so we use the infinity symbol to represent this initially.
01/30/2024 by Kelemu L. 37
Example
• Find the shortest distance from node 0 to all other nodes.
01/30/2024 by Kelemu L. 38
• Navigate from node 0 to the neighboring nodes
• This doesn't mean that we are immediately adding the two adjacent nodes to the
shortest path.
• Before adding a node to this path, we need to check if we have found the shortest path to
reach it.
• We are simply making an initial examination process to see the options available.
01/30/2024 by Kelemu L. 39
• After updating the distances of the adjacent nodes, we need to:
• Select the node that is closest to the source node based on the current known distances.
• Mark it as visited.
• Add it to the path.
Node 1 has the shortest distance to the source node (a distance of 2), so we add it to the
path.
01/30/2024 by Kelemu L. 40
• Now we need to analyze the new adjacent nodes to find the shortest path to
reach them.
• Node 3 and node 2 are both adjacent to nodes that are already in the path
because they are directly connected to node 1 and node 0, respectively, as you
can see below.
• Since we already have the distance from the source node to node 2 written down
in our list, we don't need to update the distance this time.
• We only need to update the distance from the source node to the new adjacent node
(node 3):
01/30/2024 by Kelemu L. 41
01/30/2024 by Kelemu L. 42
• We need to check the new adjacent nodes that we have not visited so far.
• This time, these nodes are node 4 and node 5 since they are adjacent to node 3.
• For node 4: the distance is 17 from the path 0 -> 1 -> 3 -> 4.
• For node 5: the distance is 22 from the path 0 -> 1 -> 3 -> 5.
01/30/2024 by Kelemu L. 43
• We need to choose which unvisited node will be marked as visited now.
• In this case, it's node 4 because it has the shortest distance in the list of distances.
• We add it graphically in the diagram:
01/30/2024 by Kelemu L. 44
• We check the adjacent nodes: node 5 and node 6.
• For node 5:
• The first option is to follow the path 0 -> 1 -> 3 -> 5, which has a distance of 22 from the
source node (2 + 5 + 15).
• This distance was already recorded in the list of distances in a previous step.
• The second option would be to follow the path 0 -> 1 -> 3 -> 4 -> 5, which has a distance of
23 from the source node (2 + 5 + 10 + 6).
• Clearly, the first path is shorter, so we choose it for node 5.
• For node 6:
• The path available is 0 -> 1 -> 3 -> 4 -> 6,
• which has a distance of 19 from the source node (2 + 5 + 10 + 2).
01/30/2024 by Kelemu L. 45
• We mark the node with the shortest (currently known) distance as visited.
• In this case, node 6.
01/30/2024 by Kelemu L. 46
• Only one node has not been visited yet, node 5.
• Three different paths that we can take to reach node 5 from the nodes that have
been added to the path:
• Option 1: 0 -> 1 -> 3 -> 5 with a distance of 22 (2 + 5 + 15).
• Option 2: 0 -> 1 -> 3 -> 4 -> 5 with a distance of 23 (2 + 5 + 10 + 6).
• Option 3: 0 -> 1 -> 3 -> 4 -> 6 -> 5 with a distance of 25 (2 + 5 + 10 + 2 + 6).
01/30/2024 by Kelemu L. 47
• The red lines mark the edges that belong to the shortest path.
• You need to follow these edges to follow the shortest path to reach a given node
in the graph starting from node 0.
01/30/2024 by Kelemu L. 48
Exercise
1.Draw a table showing the intermediate distance values of all the nodes at each
iteration of the algorithm.
2.Show the final shortest path tree
01/30/2024 by Kelemu L. 49
Huffman Coding
• Huffman code was introduced by David Huffman at MIT.
• Has a good application in lossless data compression.
• It encodes the entropy of the data in terms of variable length code.
• Fix length codes are not always desirable.
• The ANSI uses 8 bits. As such, all other encoding methods, e.g. UTF-8, EBCDIC etc., use
fixed length codes.
• Fix length codes are better to access, but it’s not efficient in terms of memory
utilization.
• Idea is to assign short code to more frequent symbols and long code to less frequent
symbols.
01/30/2024 by Kelemu L. 50
• Assume that we want to encode following six alphabets: S = <A, B, C, D, E, F>.
• If we use a 4 bit fixed length code for them, and assume that the code for each
alphabet is as follows :
• A : 0000, B : 0101, C : 1010, D : 0011, E : 1100, F : 0110
• If we want to encode the string “a b b d d e e b a c d f f a b”,
• then we need a total of 15 * 4 = 60 characters.
• This is memory intensive
• Instead use variable length code
• 0=A, 101=B, 100=C, 111=D, 1100=E, 1111=F
• Total bit length is 18 instead of 24
01/30/2024 by Kelemu L. 51
• Prefix Code:
• Variable length code should be such that decoding will not create any ambiguity.
Let’s take code for A = 0, B = 01 and C = 001.
• If our message is ABC, than its encoding will be 001001.
• As A is prefix of B and C and AB is prefix of C, it is difficult to decode the string.
• 001001 can be decoded in many ways like CC, ABAB, CAB, ABC etc.
• To prevent such ambiguity, code of any character must not be prefix of any other
code.
• Such codes are known as prefix code.
• Prefix code always provide optimal text data compression without any ambiguity.
01/30/2024 by Kelemu L. 52
Algorithm for Huffman Coding
• Algorithm HUFFMAN_CODE (PQ)
// PQ is the priority queue, in which priority is frequency of each character.
// PQ contains all n characters in decreasing order of their priority
for i ← 1 to n – 1 do
z ← CreatNode()
x ← LeftChild[z] ← deque(PQ)
y ← RightChild[z] ← deque(PQ)
z.priority ← x.priority + y.priority
enqueue (PQ, z)
end
return deque(Q)
01/30/2024 by Kelemu L. 53
Example
• Given that for character set S = <A, B, C, D, E> occurrence in text file is P = <35,
12, 8, 25, 20>. Find prefix code for each symbol.
• Solution:
• Step 1 : Arrange all characters in decreasing/increasing order of their frequency.
S = <A, D, E, B, C> and corresponding P = <35, 25, 20, 12, 8>
• Step 2 : Merge last two nodes and arrange it again in order
• Step 3 : Merge last two nodes and arrange it again in order,
• Label all left arc by 1 and all right arc by 0
01/30/2024 by Kelemu L. 54
• Visit all leaf nodes and read its edge value to find its prefix code.
• Character Prefix Code
•A 11
•B 001
•C 000
•D 10
•E 01
01/30/2024 by Kelemu L. 55
Exercise
• Find Huffman code for each symbol in following text
ABCCDEBABFFBACBEBDFAAAABCDEEDCCBFEBFCAE
• Note that count the number of occurrences of each character and follow the
steps.
01/30/2024 by Kelemu L. 56
Complexity Analysis of Huffman Coding
• Sorting of n characters according to their frequency can be achieved in O(nlog2n)
time.
• Least frequent two elements are added, it can be done in constant time.
• Merged node should be inserted in priority queue, that will take linear time O(n).
• So over all time required by Huffman code algorithm is
• O(nlog2 n) + O(n) = O(nlog2 n).
01/30/2024 by Kelemu L. 57
Dynamic Programming
01/30/2024 by Kelemu L. 58