Chapter 2: Fundamental Algorithmic Strategies

Table of Contents

 Chapter Learning Outcomes

 Introduction
 Brute Force
 Greedy
 Dynamic Programming
 Branch and Bound Methodologies
 Backtracking Methodologies
 Heuristics
 Heuristics in Problem-Solving
 Algorithm Analysis and Comparison
 Summary

Fundamental algorithmic strategies 1

  Chapter Learning Outcomes
 Understanding the concepts of below listed concepts
- Understand Algorithm Design Strategies
- Divide and Conquer
- Greedy Algorithms
- Dynamic Programming
- Backtracking

 Hands-on Problem Solving

 Algorithm Analysis and Comparison

 Introduction
Fundamental algorithmic strategies 2
Algorithmic strategies are systematic approaches to solving computational problems.
These strategies provide a framework for designing efficient algorithms to tackle a
wide range of problems across different domains, from data processing to
optimization. A deep understanding of these strategies allows for the development of
solutions that are both time-efficient and space-efficient, making them crucial for
solving large-scale, real-world problems.

Here is an overview of some key fundamental algorithmic strategies:

1. Divide and Conquer

The Divide and Conquer strategy involves breaking down a problem into smaller
subproblems, solving each subproblem independently, and then combining the
solutions to the subproblems to get the overall solution. This strategy is typically
used when the problem can be naturally divided into smaller, similar subproblems.

 Example: Merge Sort, Quick Sort, Binary Search.

 Key steps:
1. Divide the problem into smaller subproblems.
2. Solve the subproblems recursively.
3. Combine the solutions of the subproblems.

Advantages:

 Can lead to efficient solutions, especially for large problems.

 Simplifies complex problems by reducing them to easier subproblems.

2. Greedy Algorithms

A Greedy Algorithm is an approach that makes the locally optimal choice at each
step with the hope that these local solutions lead to a globally optimal solution. The
greedy approach is effective when a problem has the greedy-choice property and the
optimal substructure, meaning the global solution can be constructed by combining
local solutions.

 Example: Kruskal's Algorithm for Minimum Spanning Tree, Huffman Coding,

Activity Selection Problem.
 Key characteristics:
o A choice is made based on the current situation, without considering
future consequences.
o Does not always guarantee an optimal solution, but works efficiently in
many cases.

3. Dynamic Programming (DP)

Dynamic Programming is used for solving problems that exhibit overlapping

subproblems and optimal substructure. DP involves solving smaller subproblems
Fundamental algorithmic strategies 3
and storing their results to avoid redundant calculations, thus optimizing the overall
computation.

 Example: Fibonacci Sequence, Knapsack Problem, Longest Common

Subsequence, Matrix Chain Multiplication.
 Key steps:
1. Break the problem into overlapping subproblems.
2. Solve each subproblem and store the result (memoization or
tabulation).
3. Combine the results of the subproblems to find the final solution.

Advantages:

 Can lead to highly efficient solutions for complex problems.

 Reduces the time complexity by eliminating the need to solve the same
subproblems multiple times.

4. Backtracking

Backtracking is a technique used to solve problems by trying out all possible

solutions incrementally and abandoning a solution as soon as it is determined that it
cannot lead to a valid solution. It is often used for problems involving constraints,
such as searching through all possible configurations.

 Example: N-Queens Problem, Sudoku Solver, Graph Coloring.

 Key steps:
1. Try to extend the current solution step by step.
2. If an invalid configuration is found, backtrack and explore other options.

Advantages:

 Allows exploring all possible solutions.

 Effective for constraint satisfaction problems.

Why Algorithmic Strategies Matter

The choice of algorithmic strategy plays a critical role in determining the efficiency of
a solution. The effectiveness of an algorithm depends on:

 Time Complexity: How long it takes to run as a function of the input size.
 Space Complexity: How much memory is required for the solution.

By understanding the core algorithmic strategies, developers and engineers can

choose the most appropriate approach for a given problem, balancing complexity
and performance. Additionally, mastering these strategies is essential for solving
problems in fields such as artificial intelligence, machine learning, optimization, and
software development.
Fundamental algorithmic strategies 4
  Brute Force
Definition:
 Brute-force algorithm is a basic method for solving problems by
exhaustively exploring all possible solutions.
 It systematically examines every possible combination of solutions until it
finds the optimal one.
Key Characteristics:
1. Exhaustive Search: Brute-force algorithm evaluates every possible
solution in the solution space.
2. Straightforward Implementation: Brute-force algorithms are relatively
simple to implement, as they follow a systematic approach without
requiring complex optimizations.
3. Deterministic: Given enough time and resources, the algorithm always
guarantees finding the optimal solution.
4. Inefficiency for Large Problems: Brute-force algorithms become
impractical as the problem size increases due to the exponential growth in
the search space.
Brute Force Algorithm: How It Works

1. Identify the problem's input space.

o The input space is the set of all possible solutions or candidate
solutions to the problem.

2. Generate all possible solutions.

o This step involves generating every possible combination or
configuration that could potentially be a solution. For example, if you're
trying to find the maximum sum of elements in a list, you might have to
examine every possible subset of the list.

3. Check each solution.

o For each generated candidate solution, check if it satisfies the
problem’s conditions. If it does, then it is a valid solution.

4. Return the correct solution.

o If a valid solution is found, return it. If no solution is found, return a
failure indication.

Examples:
1. Subset Sum Problem:
 Given a set of integers and a target sum, determine if there exists
Fundamental algorithmic strategies 5
 a subset whose elements sum up to the target.
 Brute-force approach: Generate all possible subsets and check each
one for the target sum.
2. Permutations:
 Given a set of elements, find all possible permutations of those
elements.
 Brute-force approach: Generate all permutations using recursive or
iterative methods.
3. String Matching:
 Given a text string and a pattern, find all occurrences of the pattern
within the text.
 Brute-force approach: Slide the pattern over the text and compare
each substring to the pattern.
Limitations:
 Inefficient for large problem sizes due to the exhaustive search.
 Not suitable for problems with a vast solution space where more
efficient methods are required.
Advantages:
 Conceptually simple and easy to understand.
 Guarantees finding the optimal solution if one exists within the search space.
When to Use Brute Force?

1. Small Input Sizes:

If the problem size is small and the number of possible solutions is
manageable, brute force may be a good approach.
2. Lack of Knowledge of Optimized Algorithms:
In some cases, brute force is used when no efficient algorithms are known or
when developing an optimized algorithm is complex or time-consuming.
3. Finding a Reference Solution:
Brute force can be used to find the correct solution in a reference
implementation or as a benchmark against which optimized algorithms can be
tested.
4. Exhaustive Search Problems:
For problems where every possible solution must be examined, brute force is
a natural fit.

Fundamental algorithmic strategies 6

  Greedy
Definition:
 Greedy algorithms are problem-solving strategies that make locally optimal
choices at each step with the hope of finding a global optimum solution.
Key Characteristics:
1. Greedy Choice Property: Greedy algorithms make decisions based on
the best available option at the current step without reconsidering
previous choices.
2. Not Always Optimal: While greedy algorithms often provide quick
solutions, they may not always yield the most optimal solution globally.
3. Efficiency: Greedy algorithms are typically efficient and have a lower
computational overhead compared to brute-force approaches.
4. Applicability: Greedy algorithms are well-suited for optimization problems
where finding a near-optimal solution quickly is acceptable.
Examples:
1. Coin Change Problem:

 Given a set of coin denominations and a target amount, find the

minimum number of coins needed to make the change.
 Greedy approach: Continuously select the largest denomination
coin that does not exceed the remaining amount until the target is
reached.
2. Fractional Knapsack Problem:

 Given a set of items with weights and values, and a knapsack

capacity, determine the maximum value of items that can be placed
into the knapsack.
 Greedy approach: Sort items by value-to-weight ratio and greedily
select items to fill the knapsack until it reaches capacity.
3. Dijkstra's Shortest Path Algorithm:

 Find the shortest path from a source vertex to all other vertices in a
weighted graph.
 Greedy approach: Iteratively select the vertex with the smallest
tentative distance from the source and update distances to
adjacent vertices accordingly.
Limitations:
 Greedy algorithms may not always produce the globally optimal solution.
Fundamental algorithmic strategies 7
  They require careful analysis to ensure that the greedy choice property
leads to the optimal solution.
Advantages:
 Greedy algorithms are often simple to implement and computationally
efficient.
 They provide quick solutions for optimization problems where near-
optimal solutions are acceptable.
Example Application:
 Huffman Coding: Greedy algorithms can be used to construct optimal prefix-
free codes for data compression by repeatedly merging the two least
frequent characters until a single tree is formed.
Steps Involved in Greedy Algorithms:

1. Define the objective: Understand the goal of the problem (e.g., minimizing
cost or maximizing profit).
2. Characterize the problem: Break the problem into subproblems and identify
the greedy choice that needs to be made at each step.
3. Make the greedy choice: Choose the best available option at each step
without considering future implications.
4. Check for optimality: After making each choice, check whether the current
partial solution can be extended to a valid solution. If it's not, discard and try
the next best option.
5. Repeat the process: Continue making greedy choices until a complete
solution is found.

 Dynamic Programming
Definition:
 Dynamic Programming (DP) is a method for solving complex problems by
breaking them down into simpler subproblems and solving each
subproblem only once.
 It stores the solutions to subproblems in a table or array to avoid redundant
computations.
Key Characteristics:
1. Optimal Substructure: The problem can be divided into smaller,
overlapping subproblems, and the optimal solution can be constructed from
optimal solutions to these subproblems.

Fundamental algorithmic strategies 8

 2. Overlapping Subproblems: The same subproblems are solved multiple
times in a recursive solution, and dynamic programming saves the results of
these subproblems to avoid redundant computations.
3. Memoization or Tabulation: Dynamic programming can be implemented
using either memoization (top-down) or tabulation (bottom-up) approach to
store and retrieve solutions to subproblems.
Examples:
1. Fibonacci Sequence:

 Compute the nth Fibonacci number.

 Dynamic programming approach: Store the results of previously
computed Fibonacci numbers to avoid redundant calculations.
2. Longest Common Subsequence (LCS):

 Find the longest subsequence present in given sequences X and Y.

 Dynamic programming approach: Build a table to store the length of
the LCS for all subproblems.
3. Knapsack Problem:

 Given a set of items with weights and values, and a knapsack

capacity, determine the maximum value of items that can be placed
into the knapsack.
 Dynamic programming approach: Create a table to store the
maximum value that can be achieved with different item subsets and
capacities.
Limitations:
 Dynamic programming may require significant memory to store solutions
to subproblems, especially for problems with large input sizes.
 It may not be suitable for problems with overlapping subproblems or optimal
substructure.
Advantages:
 Dynamic programming efficiently solves problems with overlapping
subproblems by avoiding redundant computations.
 It provides an elegant solution to many optimization problems by breaking
them down into smaller, solvable subproblems.
Example Application:
 Edit Distance: Dynamic programming can be used to calculate the
minimum number of operations (insertions, deletions, substitutions)

Fundamental algorithmic strategies 9

 required to convert one string into another, which is useful in spell
checking, DNA sequencing, and natural language processing.
Steps to Solve a Problem Using Dynamic Programming:

1. Characterize the structure of the optimal solution:

Break down the problem into smaller subproblems. Identify how these
subproblems combine to form the overall solution.
2. Define the state:
Define what each subproblem’s solution represents. This is usually done
using an array or a table to store the solutions to subproblems.
3. Formulate the recurrence relation:
Establish a recurrence relation that describes how to compute the solution to
a problem using solutions to smaller subproblems. This is the heart of
dynamic programming.
4. Memoization or Tabulation:
o Use memoization (top-down) by solving subproblems recursively and
storing the results.
o Use tabulation (bottom-up) by iteratively filling in a table to solve the
subproblems.

5. Compute the final solution:

Once all subproblems are solved, the solution to the original problem can be
found in the table or from the stored results.

 Branch and Bound Methodologies

Definition:
 Branch-and-Bound is a problem-solving paradigm that systematically
explores the solution space by dividing it into smaller subspaces (branches)
and bounding the search within certain limits.
 It efficiently prunes branches that cannot lead to an optimal solution,
reducing the search space.
Key Characteristics:
1. Divide and Conquer: Branch-and-Bound breaks down the problem into
smaller subproblems and explores each subproblem independently.
2. Bounding: It establishes criteria (bounds) to eliminate subspaces or
branches that cannot contain the optimal solution.
3. Exploration Strategy: Branch-and-Bound typically employs strategies
such as depth-first search or breadth-first search to traverse the solution
space efficiently.

Fundamental algorithmic strategies 10

 4. Optimality Guarantee: Branch-and-Bound ensures finding an optimal
solution or proving its optimality by exploring the entire solution space.
Key Concepts of Branch and Bound:

1. Branching: Branching involves dividing the problem into subproblems, each

of which represents a "branch" in the search tree. The branching process
systematically explores different decisions or solutions by dividing the original
problem into smaller subproblems.
2. Bounding: Bounding is the process of determining the best possible (upper
or lower) bound for the objective function in a subproblem. This bound is used
to decide whether to explore that subproblem further or prune it away if it
cannot lead to a better solution than the current best.
o Upper Bound: The best solution found so far (typically for minimization
problems, it is the smallest known solution).
o Lower Bound: A calculated bound that is guaranteed to be worse than
the optimal solution but still provides a limit to the subproblem's
objective function.

3. Pruning: Pruning refers to the process of discarding or excluding

subproblems that cannot yield better solutions than the current best. This is
done using the bounds: if the bound of a subproblem is worse than the best-
known solution, it is not necessary to explore further.
4. Search Tree: The solution space is represented as a tree, where each node
corresponds to a subproblem. The root node represents the original problem,
and the branches represent the subproblems formed by dividing the original
problem.

General Steps of Branch and Bound:

1. Initialization: Start by initializing the best solution found so far (called the
incumbent). The root of the search tree represents the original problem, and
an initial bound is computed for the root.
2. Branching: Split the problem into smaller subproblems, creating child nodes.
Each subproblem corresponds to a branch in the search tree.
3. Bounding: For each node in the search tree (i.e., each subproblem), compute
an upper or lower bound of the objective function. This bound will guide the
decision of whether to further explore the node.
4. Pruning: Evaluate each subproblem:
o If a subproblem has a bound worse than the current best solution
(incumbent), it is pruned (discarded).
o If a subproblem’s bound is better than the incumbent, explore it further
(branch it).

5. Explore Remaining Nodes: Continue branching and bounding, and explore

subproblems that have the potential to yield a better solution than the
incumbent. This is done by using a queue (or a priority queue) to explore
Fundamental algorithmic strategies 11
 nodes with the best potential first (often using best-first or depth-first search
strategies).
6. Termination: The algorithm terminates when all subproblems are either
solved or pruned. The best solution found during the search is the optimal
solution to the problem.

Types of Branch and Bound:

1. Best-First Search: In Best-First Search, nodes with the best bound

(according to some criterion, such as the smallest upper bound for
minimization problems) are explored first. A priority queue (min-heap or max-
heap) is commonly used to implement this strategy.
2. Depth-First Search: This is a simpler approach where the algorithm explores
one branch of the search tree as deeply as possible before backtracking and
exploring other branches. It is generally faster but might not always find the
optimal solution as efficiently as Best-First.
3. Breadth-First Search: In this approach, the algorithm explores all the nodes
at a given depth (i.e., level) in the search tree before moving on to the next
level. This method is less commonly used in B&B algorithms.

Examples:
1. Traveling Salesman Problem (TSP):

 Given a set of cities and distances between them, find the shortest
possible route that visits each city exactly once and returns to the
origin city.
 Branch-and-Bound approach: Explore the solution space by
considering all possible permutations of city visits, pruning branches
based on lower bounds.
2. Integer Linear Programming (ILP):

 Optimize a linear objective function subject to linear equality and

inequality constraints, with the additional requirement that some or
all variables take integer values.
 Branch-and-Bound approach: Divide the search space into integer-
feasible regions and use bounds to prune infeasible regions.
3. Bin Packing Problem:

 Given a set of items with weights and a set of bins with

capacities, minimize the number of bins needed to pack all items.
 Branch-and-Bound approach: Explore different packing
arrangements of items into bins and prune branches that exceed bin
capacities.

Fundamental algorithmic strategies 12

Limitations:
 Branch-and-Bound may become computationally expensive for large
problem instances due to the exponential growth in the solution space.
 It requires careful selection of bounding strategies and exploration
techniques to ensure efficiency.
Advantages:
 Branch-and-Bound guarantees finding an optimal solution or proving
optimality by exploring the entire solution space.
 It efficiently prunes branches that cannot lead to an optimal solution,
reducing the search space.
Example Application:
Job Scheduling: Branch-and-Bound can be employed to solve the job scheduling
problem, where tasks need to be assigned to machines to minimize makespan or
total completion time, by exploring different scheduling options and pruning
infeasible or suboptimal solutions.

 Backtracking Methodologies
Definition:
 Backtracking is a problem-solving technique used to systematically search
for solutions by exploring all possible candidates and backtracking from
those candidates as soon as it determines that they cannot lead to a valid
solution.
It is particularly useful for problems with constraints or decisions that need to
be made incrementally.
Key Concepts of Backtracking:

1. Incremental Building of Solutions:

o The algorithm builds a potential solution piece by piece. At each step, a
choice is made from the available options (or decisions) to move
forward in the problem-solving process.
2. Pruning or Backtracking:
o If at any point it becomes clear that a solution cannot be completed
(i.e., a partial solution violates some constraint), we backtrack to the
previous step and try the next available option. This process effectively
prunes the search tree by eliminating invalid or non-promising
branches early.
3. Recursive Nature:
o Backtracking is typically implemented using recursion, where the
function calls itself with the next decision, and in case of failure, it
returns to the previous state and tries a different path.
Fundamental algorithmic strategies 13
 4. Exploration of Solution Space:
o Backtracking explores all possible paths in the solution space. It does
so by branching out and pruning infeasible paths whenever a constraint
is violated, ensuring that the search for solutions is efficient.

General Steps of Backtracking:

1. Choose:
At each step, choose an option or decision from a set of possible choices that
would extend the current partial solution.
2. Explore:
Recursively explore the next step based on the current choice. If the solution
is complete and valid, return it. If not, move on to the next possible choice.
3. Check Constraints:
After choosing an option, check whether the current solution still satisfies the
problem’s constraints. If the constraints are violated, backtrack.
4. Backtrack:
If the current path leads to a dead-end (i.e., no valid solution can be built from
the current partial solution), undo the last choice (backtrack) and try the next
possibility.
5. Terminate:
The algorithm terminates when all possibilities have been explored or a valid
solution has been found.

Types of Problems Solved Using Backtracking:

1. Combinatorial Problems: These problems involve selecting or arranging

objects from a set. For example, generating all subsets of a set, finding
permutations or combinations, or solving the Subset Sum Problem.
2. Constraint Satisfaction Problems: Problems that require finding a solution
that satisfies certain constraints. Examples include the N-Queens problem,
Sudoku, and the Graph Coloring problem.
3. Optimization Problems: These involve finding the best solution from a set of
feasible solutions. For example, Knapsack Problem, Traveling Salesman
Problem (TSP), or String Matching.

Key Characteristics:
1. Systematic Search: Backtracking explores the solution space
systematically, considering one candidate solution at a time.
2. Incremental Construction: It incrementally builds candidate solutions,
making decisions at each step based on constraints and requirements.
3. Backtracking Mechanism: When the algorithm determines that the
current candidate solution cannot be extended further to satisfy the
problem constraints, it backtracks to the previous decision point and
explores other options.
Fundamental algorithmic strategies 14
 Pruning: Backtracking may include pruning techniques to eliminate branches
of the search space that cannot lead to valid solutions, improving efficiency.
Examples:
1. N-Queens Problem:

 Place N queens on an N×N chessboard so that no two queens attack

each other.
 Backtracking approach: Place queens one by one and backtrack
when a conflict is encountered, exploring all possible placements.
2. Sudoku Solver:

 Fill a 9×9 grid with digits such that each column, row, and 3×3
subgrid contains all the digits from 1 to 9 without repetition.
 Backtracking approach: Fill the grid cell by cell, trying different digit
placements and backtracking when a conflict arises.
3. Graph Coloring:

 Assign colors to vertices of a graph such that no two adjacent vertices

share the same color with the minimum number of colors.
Backtracking approach: Color vertices one by one and backtrack when a
coloring violates the graph's constraints.
Limitations:
 Backtracking may become inefficient for large problem instances due to
the exponential growth in the solution space.
 Pruning strategies are crucial for improving efficiency, and selecting
appropriate constraints is essential.
Advantages:
 Backtracking is a versatile technique applicable to various
combinatorial optimization problems with constraints.
It systematically explores the solution space and efficiently prunes branches
that cannot lead to valid solutions.
Example Application:
 Cryptarithmetic Puzzles: Backtracking can be used to solve cryptarithmetic
puzzles, where letters represent digits, and the goal is to find a digit
assignment that satisfies the puzzle's arithmetic constraints by
systematically exploring different digit assignments and backtracking when
inconsistencies arise.

 Heuristics
Fundamental algorithmic strategies 15
Definition:
 Heuristics are problem-solving techniques or rules of thumb that provide
practical solutions to problems, often by sacrificing optimality for efficiency.
They are strategies or guidelines used to quickly find satisfactory solutions
when an exhaustive search or optimal solution is impractical.
Key Concepts of Heuristics:

1. Approximation: Heuristics provide approximate solutions, typically faster and

at a lower computational cost than exact algorithms. They are used when an
optimal solution is not required or too costly to compute.
2. Rule of Thumb: A heuristic is essentially a "rule of thumb" or a practical
approach to solving problems that aren't guaranteed to find the best solution
but usually work well under most circumstances.
3. Problem-Specific: Heuristics are often tailored to specific types of problems
and are designed based on domain knowledge or characteristics of the
problem. The effectiveness of a heuristic depends on how well it aligns with
the structure of the problem.
4. Greedy Approach: Some heuristics follow a greedy approach, making locally
optimal choices at each step with the hope of finding a global optimum.
5. Search Efficiency: Heuristics help to speed up search processes by guiding
the search towards promising areas of the solution space, thereby reducing
the number of options to explore.

Types of Heuristics:

1. Greedy Heuristics: In greedy algorithms, the heuristic picks the locally

optimal choice at each step, hoping that these choices will lead to a globally
optimal solution. It does not reconsider decisions made earlier. For example:
o Greedy Best-First Search: In pathfinding problems like the A
algorithm*, the heuristic is used to estimate the distance to the goal.
The search chooses paths that seem closest to the goal, based on the
heuristic.

2. Admissible Heuristics: These heuristics are guaranteed to never

overestimate the true cost to reach the goal. An admissible heuristic is
particularly important in search algorithms like A search*, where it is
necessary to ensure optimality. It provides a lower bound on the cost to reach
the goal. For example:
o Euclidean distance: In pathfinding, it may be used to estimate the
straight-line distance between two points.

3. Informed Heuristics: These heuristics use domain-specific information to

guide the search process. They provide more information than just random
choices and help the algorithm make more intelligent decisions. Examples
include:

Fundamental algorithmic strategies 16

 o Manhattan Distance: Used in grid-based pathfinding problems, it
calculates the total grid distance from the current point to the goal by
summing the horizontal and vertical distances.

4. Uninformed Heuristics (Blind Search): These heuristics do not use any

domain-specific knowledge and treat the search as a blind exploration of the
solution space. They rely on brute-force search methods and explore all
possibilities without guidance. Examples include:
o Breadth-First Search: The algorithm explores all nodes level by level,
without any additional heuristics to prioritize the search.

5. Constraint Satisfaction Heuristics: In problems like Sudoku, N-Queens, or

Graph Coloring, heuristics can help to quickly prune infeasible solutions.
Common heuristics include:
o Most Constrained Variable (MCV): Choosing the variable with the
fewest legal values left.
o Least Constraining Value (LCV): Selecting the value that constrains
the least number of other variables.

6. Local Search Heuristics: These heuristics operate by moving through the

solution space from one potential solution to another, evaluating solutions
along the way. They are especially useful in optimization problems. Examples
include:
o Hill Climbing: This heuristic starts from a random solution and
iteratively moves to neighboring solutions that improve the objective
function.
o Simulated Annealing: A probabilistic technique that allows for
occasional acceptance of worse solutions to escape local optima.

Key Characteristics:
1. Efficiency: Heuristics prioritize finding solutions quickly, even if they may
not be optimal or guaranteed to be the best.
2. Simplicity: Heuristics are typically simple and easy-to-understand
strategies that do not require complex computations.
3. Domain-Specific: Heuristics are often tailored to specific problem
domains, leveraging domain knowledge or patterns to guide problem-
solving.
4. Trade-off: Heuristics often involve a trade-off between solution quality and
computational efficiency, providing satisfactory solutions within a
reasonable time frame.

Fundamental algorithmic strategies 17

Examples:
1. Greedy Heuristic:

 Greedy algorithms are heuristic approaches that make locally optimal

choices at each step, hoping to find a globally optimal solution.
 Example: Dijkstra's algorithm for finding the shortest path in a graph
employs a greedy heuristic by selecting the vertex with the smallest
tentative distance at each step.
2. Nearest Neighbor Heuristic:
 In the traveling salesman problem, the nearest neighbor heuristic involves
selecting the nearest unvisited city as the next destination at each step.
 Example: The nearest neighbor heuristic provides a quick solution
to the TSP but may not guarantee the shortest possible tour.
3. Rule-Based Heuristics:

 Rule-based heuristics involve predefined rules or guidelines to guide

decision-making in problem-solving.
 Example: In chess, players often follow heuristics such as
controlling the center of the board, developing pieces, and
protecting the king for strategic advantage.
Limitations:
 Heuristics may produce suboptimal solutions or fail to find a solution in some
cases.
They rely on simplifications and assumptions, which may not always hold true in
complex problem domains.
Advantages:
 Heuristics provide practical and efficient solutions to complex problems,
often in situations where finding an optimal solution is impractical.
They are easy to implement and understand, making them accessible for problem-
solving in various domains.
Example Application:
Route Planning: Heuristic algorithms such as A* search are used in route planning
applications to quickly find near-optimal paths between locations by prioritizing
promising routes based on estimated distances to the destination, even though the
actual shortest path may not be known in advance.

 Heuristics in Problem-Solving

Fundamental algorithmic strategies 18

Definition:
 Heuristics are problem-solving strategies or rules of thumb that guide
decision-making in situations where exhaustive search or optimal
solutions are impractical.
They involve using practical techniques, experience, and intuition to quickly find
satisfactory solutions, often without guaranteeing optimality.
Key Characteristics:
1. Rule-based Approach: Heuristics rely on predefined rules or guidelines to
make decisions quickly and efficiently.
2. Fast Decision-Making: Heuristics prioritize speed over optimality,
allowing for rapid problem-solving in complex or uncertain
environments.
3. Domain-Specific: Heuristics are often tailored to specific problem
domains, leveraging domain knowledge and expertise.
4. Adaptability: Heuristics can adapt to changing circumstances or new
information, making them flexible problem-solving tools.
Examples:
1. Greedy Heuristic:

 Selecting the locally optimal choice at each step, hoping to

reach a satisfactory solution.
 Example: Traveling salesman problem solved using the nearest
neighbor heuristic.
2. Availability Heuristic:

 Basing judgments or decisions on readily available information or

recent experiences.
 Example: Assessing the risk of an event based on vivid or
memorable examples rather than statistical data.
3. Anchoring Heuristic:

 Relying heavily on the first piece of information encountered when

making decisions.
 Example: Negotiating prices, where the initial offer sets a
reference point for subsequent negotiations.
Application Domains:
Heuristics are widely applied in various domains, including:
 Problem-solving in artificial intelligence and optimization.

Fundamental algorithmic strategies 19

  Decision-making in psychology, economics, and behavioral sciences.
 Search algorithms and route planning in computer science and logistics.
 Game-playing strategies in game theory and recreational activities.
Limitations:
 Heuristics may lead to suboptimal or biased decisions in certain situations.
 They can overlook less apparent but more optimal solutions,
especially in complex or dynamic environments.
 Reliance on heuristics may result in cognitive biases and errors in judgment.
Advantages:
 Heuristics enable quick decision-making and problem-solving in situations
where exhaustive search or optimal solutions are impractical.
 They leverage available information and domain knowledge to guide
decision-making efficiently.
Example Application:
In route planning applications, heuristics are used to quickly find approximate
solutions to the traveling salesman problem by guiding the search towards promising
routes without exhaustively exploring all possibilities.

 Types of Heuristic Strategies in Problem-Solving:

1. Greedy Heuristics:
o Greedy heuristics involve making the locally optimal choice at each
step with the hope that these choices will lead to a globally optimal
solution. These strategies focus on immediate benefits rather than
long-term consequences.
o Example: In the Traveling Salesman Problem (TSP), a greedy
heuristic might choose the nearest unvisited city as the next city to visit,
hoping this will lead to the shortest overall route.

2. Admissible Heuristics:
o An admissible heuristic never overestimates the cost to reach the goal.
This type of heuristic is useful in search algorithms like A* search,
where the goal is to find the shortest path to a goal.
o Example: Euclidean distance is an admissible heuristic for
pathfinding problems on a plane, as it never overestimates the true
distance between two points.

3. Informed Search Heuristics:

o These heuristics use domain-specific knowledge to estimate the cost or
distance to a solution. Informed heuristics improve the efficiency of
Fundamental algorithmic strategies 20
 search algorithms by guiding the search toward more promising
solutions.
o Example: A* Search Algorithm uses an informed heuristic that
combines the actual cost of the path so far and the estimated cost to
the goal (e.g., straight-line distance).

4. Uninformed Heuristics (Blind Search):

o Uninformed heuristics, or blind search strategies, do not use domain-
specific information. These methods search the solution space in a
systematic but non-directive manner.
o Example: Breadth-First Search (BFS) is an uninformed search
algorithm that explores all possible solutions level by level without any
heuristic guidance.

5. Constraint Satisfaction Heuristics:

o In problems like Sudoku, N-Queens, or Graph Coloring, heuristics
are used to reduce the search space by prioritizing certain variables or
values.
o Example: The Most Constrained Variable (MCV) heuristic in
constraint satisfaction problems selects the variable that has the fewest
legal values left, potentially reducing the search tree’s size.

6. Local Search Heuristics:

o Local search heuristics operate by iteratively improving a current
solution by exploring its neighbors, typically in problems where a
complete search of the solution space is impractical.
o Example: Hill climbing is a local search heuristic that starts from a
random solution and iteratively moves to a neighboring solution that
improves the objective function (e.g., increasing the value or reducing
the cost).

7. Simulated Annealing:
o Simulated annealing is a probabilistic technique used to approximate
the global optimum of a problem. It allows occasional acceptance of
worse solutions to escape local minima, simulating the process of
cooling metal.
o Example: In the TSP, simulated annealing could allow visiting a distant
city temporarily if it helps avoid getting trapped in a local optimum.

 Example Applications of Fundamental Algorithmic Strategies

Fundamental algorithmic strategies 21

Introduction:
Now that we've discussed fundamental algorithmic strategies such as Brute-Force,
Greedy, Dynamic Programming, Branch-and-Bound, and Backtracking, let's explore
how these techniques are applied to solve real-world problems.
Problem-Solving Examples:
1. Bin Packing Problem:

 Brute-Force: Enumerate all possible combinations of items into bins

and select the combination that minimizes the number of bins used.
 Dynamic Programming: Optimize the solution by storing
intermediate results and reusing them to efficiently compute the
minimum number of bins required.
 Heuristics: Use greedy heuristics to select items for packing based
on criteria such as weight or volume, aiming to fill bins optimally.
2. Knapsack Problem:

 Brute-Force: Try all possible subsets of items and select the one
with the maximum value within the knapsack capacity.
 Dynamic Programming: Build a table to store the maximum value
that can be achieved with different item subsets and capacities,
efficiently solving the problem.
 Branch-and-Bound: Explore different combinations of items and
prune branches that exceed the knapsack capacity, improving
efficiency.
3. Traveling Salesman Problem (TSP):

 Brute-Force: Enumerate all possible permutations of cities and

calculate the total distance traveled for each permutation.
 Dynamic Programming: Use dynamic programming to solve
smaller subproblems and store their solutions, reducing redundant
computations in the overall solution.
 Heuristics: Employ nearest neighbor or genetic algorithms to
quickly find approximate solutions that are close to optimal for large
instances of the problem.
Cross-Strategy Applications:
 Job Scheduling (Branch-and-Bound + Heuristics):
 Use Branch-and-Bound to explore different scheduling options and
prune infeasible solutions based on constraints such as resource
availability and task dependencies.

Fundamental algorithmic strategies 22

  Apply heuristics to quickly generate initial feasible solutions or
guide the search towards promising regions of the solution space,
improving efficiency.
 Optimal Route Planning (Greedy + Dynamic Programming):
 Use Greedy algorithms to make locally optimal decisions at
each step, such as selecting the nearest neighbor or shortest
edge, to construct a feasible route.
 Apply Dynamic Programming to optimize the route by considering all
possible permutations of visited cities and selecting the one with the
minimum total distance.

 Illustrations of Algorithmic Strategies in Problem-

Solving
Introduction:
 Algorithmic strategies such as Brute-Force, Greedy, Dynamic
Programming, Branch-and- Bound, and Backtracking provide different
approaches to problem-solving, each suited to specific types of problems.
 In this section, we will illustrate these strategies through three classic
optimization problems: Bin Packing, Knapsack, and Traveling Salesman
Problem (TSP).
1. Bin Packing Problem:

 Given a set of items with weights and a set of bins with capacities,
minimize the number of bins needed to pack all items.
 Brute-Force Approach:
 Generate all possible combinations of items and bins.
 Check each combination to see if it satisfies the capacity
constraints.
 Select the combination with the minimum number of bins.
 Greedy Approach:
 Sort items by non-increasing order of weight.
 Place each item into the first bin that can accommodate it.
 If no bin can accommodate the item, use a new bin.
 Dynamic Programming Approach:
 Formulate the problem as a knapsack problem with bin

Fundamental algorithmic strategies 23

 capacities as constraints.
 Use dynamic programming to find the optimal packing of items
into bins.
2. Knapsack Problem:

 Given a set of items with weights and values, and a knapsack

capacity, determine the maximum value of items that can be placed
into the knapsack.
 Brute-Force Approach:
 Generate all possible combinations of items.
 Check each combination to see if it exceeds the knapsack
capacity.
 Select the combination with the maximum value within the
capacity.
 Greedy Approach:
 Sort items by non-increasing order of value-to-weight ratio.
 Add items to the knapsack greedily until the capacity is reached.
 Dynamic Programming Approach:
 Formulate the problem as a recursive function with
subproblems representing different item subsets and capacities.
 Use memoization or tabulation to store and retrieve solutions to
subproblems efficiently.
3. Traveling Salesman Problem (TSP):

 Given a set of cities and distances between them, find the shortest
possible route that visits each city exactly once and returns to the
starting city.
 Brute-Force Approach:
 Generate all possible permutations of cities.
 Calculate the total distance for each permutation.
 Select the permutation with the minimum total distance.
 Branch-and-Bound Approach:
 Explore the solution space using depth-first search or
branch-and-bound technique.
 Use lower bounds to prune branches that cannot lead to an

Fundamental algorithmic strategies 24

 optimal solution.
 Continuously update the best solution found during exploration.
 Heuristic Approach:
 Use heuristics such as nearest neighbor or genetic algorithms
to quickly find approximate solutions.
 Iterate and refine the solutions to improve accuracy.

 Heuristics: Characteristics and Application Domains

Introduction:
 Heuristics are problem-solving techniques that prioritize speed and
practicality over optimality, often relying on rules of thumb or past
experiences to guide decision-making.
 Understanding the characteristics and application domains of
heuristics is essential for effectively employing them in problem-solving
scenarios.
1. Characteristics of Heuristics:

 Rule-Based: Heuristics rely on predefined rules or guidelines to

make decisions quickly.
 Fast Decision-Making: They prioritize speed over optimality,
allowing for rapid problem-solving.
 Domain-Specific: Heuristics are often tailored to specific
problem domains, leveraging domain knowledge.
 Adaptability: Heuristics can adapt to changing circumstances or
new information, making them flexible problem-solving tools.
2. Examples of Heuristics:
 Greedy Heuristic: Selecting the locally optimal choice at each step,
hoping to reach a satisfactory solution.
 Availability Heuristic: Basing judgments or decisions on
readily available information or recent experiences.
 Anchoring Heuristic: Relying heavily on the first piece of
information encountered when making decisions.
3. Application Domains of Heuristics:

 Artificial Intelligence: Heuristics are widely used in AI algorithms

such as search algorithms, constraint satisfaction, and game
playing.
Fundamental algorithmic strategies 25
  Operations Research: Heuristics play a vital role in optimization
problems like scheduling, routing, and resource allocation.
 Psychology and Behavioral Sciences: Heuristics are studied in the
context of human decision-making processes and cognitive biases.
 Computer Science: Heuristics are applied in various areas,
including algorithms, machine learning, and data analysis.
4. Advantages and Limitations:

 Advantages: Heuristics enable quick decision-making and

problem-solving in situations where exhaustive search or optimal
solutions are impractical.
 Limitations: They may lead to suboptimal or biased decisions
and can overlook less apparent but more optimal solutions.

 Algorithm Analysis and Comparison

Algorithm Analysis and Comparison are fundamental aspects of computer

science that allow us to evaluate the efficiency and performance of algorithms. They
help determine how an algorithm performs in terms of resource usage (like time and
space) and how scalable it is with increasing problem sizes. Understanding how to
analyze and compare algorithms enables software engineers and computer
scientists to select the best algorithm for a given problem, optimizing both speed and
resource usage.

In this context, we focus on two primary aspects:

1. Time Complexity: The amount of time an algorithm takes to run as a function

of the size of the input.
2. Space Complexity: The amount of memory an algorithm uses as a function
of the size of the input.

1. What is Algorithm Analysis?

Algorithm analysis is the process of determining the computational efficiency of an

algorithm. The analysis is typically done in terms of time complexity and space
complexity, which provide insights into the algorithm's behavior as the input size
increases.

Goals of Algorithm Analysis:

 Efficiency: To find algorithms that solve problems in the least time and with
minimal resources.

Fundamental algorithmic strategies 26

  Scalability: To ensure that the algorithm can handle large inputs as the
problem grows.
 Predictability: To predict the performance of an algorithm under varying
conditions and input sizes.

2. Time Complexity Analysis

Time complexity refers to the amount of time an algorithm takes to execute as a

function of the input size. It provides a way to measure the efficiency of an algorithm.

Common Time Complexity Classifications:

1. Constant Time: O(1):

o The execution time of the algorithm does not depend on the input size.
The algorithm takes the same amount of time regardless of the size of
the input.
o Example: Accessing an element in an array by index.

2. Logarithmic Time: O(log n):

o The execution time increases logarithmically with the input size.
Logarithmic time complexities often appear in algorithms that divide the
problem in half at each step (e.g., binary search).
o Example: Binary search on a sorted array.

3. Linear Time: O(n):

o The execution time grows linearly with the input size. This typically
occurs when an algorithm needs to examine every element in the input
exactly once.
o Example: Linear search or traversing a list of n elements.

4. Linearithmic Time: O(n log n):

o The execution time grows faster than linear but slower than quadratic
time. This time complexity is typical for efficient sorting algorithms.
o Example: Merge Sort, Quick Sort, and Heap Sort.

5. Quadratic Time: O(n²):

o The execution time grows quadratically with the input size. This often
happens in algorithms that involve nested loops.
o Example: Bubble Sort, Selection Sort, Insertion Sort.

Algorithm Comparison

Once an algorithm is analyzed, the next step is to compare it against other

algorithms to determine the best one for a given problem. The comparison process
involves evaluating both the time complexity and space complexity of algorithms.

Fundamental algorithmic strategies 27

Key Comparison Metrics:

1. Efficiency:
o Time efficiency: Analyzing how long the algorithm takes to run with
respect to input size.
o Space efficiency: Analyzing how much memory the algorithm requires
for computation.

2. Scalability:
o How well does the algorithm perform as the input size increases? An
algorithm with a better time complexity (e.g., O(log n) vs O(n²)) is more
scalable.

3. Robustness:
o Analyzing how the algorithm handles various types of input, including
edge cases (e.g., empty input, large inputs).

4. Practical Performance:
o Sometimes, despite the worst-case theoretical performance, an
algorithm may perform well in practice due to constant factors, input
characteristics, or specific optimizations. For example, quick sort (O(n
log n)) often performs better than other O(n log n) algorithms in real-
world scenarios due to smaller constant factors.

5. Memory Usage:
o Some algorithms may require significant memory (like recursive
algorithms that use the call stack), while others may be more space-
efficient (e.g., iterative algorithms).

Example of Algorithm Comparison

Consider the task of sorting an array. Let's compare the Bubble Sort algorithm and
the Quick Sort algorithm:

 Bubble Sort:
o Time Complexity: O(n²) in the worst and average cases.
o Space Complexity: O(1) (in-place sorting, no extra space needed).
o This algorithm is inefficient for large datasets due to its quadratic time
complexity.
 Quick Sort:
o Time Complexity: O(n log n) on average, but O(n²) in the worst case
(when the pivot selection is poor).
o Space Complexity: O(log n) for recursion stack.
o Although quick sort has a worst-case time complexity of O(n²), in
practice, it is much faster than bubble sort, especially on large
datasets, because it has better average-case time complexity.

Fundamental algorithmic strategies 28

Comparison Conclusion:

 Quick Sort is more efficient for larger inputs than Bubble Sort, especially in
terms of time complexity.
 Bubble Sort might be preferable for small datasets or when a simple
algorithm is needed, but it generally performs poorly on larger inputs due to its
O(n²) time complexity.
 For space efficiency, both algorithms are space-efficient in that they both
have low space complexity (with Quick Sort using slightly more space due to
recursion).

 Summary

Fundamental algorithmic strategies provide the core approaches used in designing

and solving computational problems efficiently. These strategies are essential for
ensuring that algorithms are effective, scalable, and feasible for real-world
applications. Below is a summary of key algorithmic strategies:

1. Brute Force:
o A straightforward approach where all possible solutions are
systematically tried until the correct one is found.
o Simple but inefficient for large problems due to its exhaustive nature.
o Example: Brute force search in finding the maximum value in an
unsorted array.
2. Greedy Algorithms:
o These algorithms make locally optimal choices at each step with the
hope that these choices will lead to a globally optimal solution.
o Greedy algorithms are often faster and simpler but may not always
provide the best solution for every problem.
o Example: The Greedy Knapsack Problem or Huffman coding.
3. Dynamic Programming (DP):
o A method for solving complex problems by breaking them down into
simpler subproblems and solving each subproblem just once, storing
the result for future reference.
o Particularly useful for optimization problems where overlapping
subproblems occur.
o Example: Fibonacci sequence, Longest Common Subsequence
(LCS).
4. Divide and Conquer:
o The problem is divided into smaller, more manageable subproblems
that are solved independently and then combined to form the solution
to the original problem.
o It is especially efficient for problems that can be recursively divided into
smaller parts.

Fundamental algorithmic strategies 29

 o Example: Merge Sort, Quick Sort.
5. Branch and Bound:
o A method used for solving optimization problems, where solutions are
systematically explored by breaking down the problem into smaller
subproblems (branching) and pruning suboptimal solutions (bounding).
o Suitable for problems like Traveling Salesman Problem (TSP).
6. Backtracking:
o An incremental approach that builds solutions step by step,
abandoning (backtracking) solutions that fail to meet the problem's
constraints.
o Used for problems like constraint satisfaction, where a set of possible
solutions is being explored.
o Example: N-Queens problem, Sudoku solving.
7. Heuristics:
o Problem-solving strategies that use practical, often approximate
methods to find solutions that are "good enough" within a reasonable
time.
o These approaches are efficient but do not guarantee the optimal
solution.
o Example: A search algorithm*, Simulated Annealing.
8. Randomization:
o Randomization introduces randomness into the algorithm’s logic, which
can help avoid worst-case scenarios and make the algorithm more
adaptable to different problem instances.
o Example: Randomized Quick Sort, Monte Carlo methods.
9. Graph Algorithms:
o Focus on solving problems related to graph structures, such as finding
shortest paths, detecting cycles, or finding spanning trees.
o Common graph algorithms include Dijkstra's algorithm, Kruskal’s
algorithm, and Floyd-Warshall.
10. Network Flow Algorithms:
o These algorithms are designed for optimization problems involving flow
through networks, such as finding the maximum flow in a network.
o Example: Ford-Fulkerson algorithm for finding the maximum flow.

Key Takeaways:

 Efficiency is the central goal in algorithm design, whether it’s time complexity,
space complexity, or scalability.
 Different strategies are suited to different types of problems, and often, hybrid
approaches are used in practice.
 Greedy algorithms are fast but may not always find the best solution, while
Dynamic Programming and Divide and Conquer offer more structured,
optimized approaches for larger, more complex problems.
 Understanding each algorithmic strategy is key to selecting the appropriate
one based on problem requirements and constraints.

Fundamental algorithmic strategies 30

Fundamental algorithmic strategies 31