CONTENTS

Linear Programming
• Standard Form of LPP
• Slack Variable
• Surplus Variable
• Methods for Solving
o Graphical Method
o Simplex Method
o Two Phase Method
o Big M Method
o Revised Simplex Method
o Big M + Revised Simplex Method

Non – Linear Programming

• Convex Function
• Concave Function
• Some Facts about Convex/Concave Functions
• Determining Convexity using the Hessian Matrix
• Convex Programming Problems
• KKT Conditions for Constrained Optimization Problem
• Desirable Properties of an Algorithm used for solving Unconstrained
Minimization Problems
o Descent Property
o Quadratic Termination Property
o Globally Convergent
o Order of Convergence
• Unimodal Function
• Line Search Methods for Unimodal Functions
o The Golden Section Rule
o The Fibonacci Search Method
o Relation Between the Above Two Methods
• Methods for Solving Unconstrained Minimization Problems
o The Steepest Descent Method
o Newton’s Method
o Modified Newton’s Method
o The Conjugate Gradient Method
o Fletcher and Reeves’ Method
o Davidon – Fletcher – Powell (DFP) Method
• Methods for Solving Convex Quadratic Programming Problems
o Wolfe’s Method
• Methods for Solving Constrained Optimization Problems
o Method of Lagrange Multipliers
o Frank and Wolfe’s Method
o The Penalty Function Method
▪ Numerial Implementation
o The Barrier Function Method
▪ Numerical Implementation
• Multi Objective Programming Problems
o Goal Programming
▪ Aspiration Level
▪ Goal
▪ Goal Deviation
▪ Hard Goals and Soft Goals
o Formulation of Goal Programming Problem
o General Model of Goal Programming Problem
o Methods for Solving Linear Goal Programming Problem
▪ Graphical Method
▪ Sequential Goal Programming Technique
• Evolutionary Methods and Global Optimization
o Simulated Annealing
▪ Procedure
o Genetic Algorithms
▪ Procedure
 CE – 565

Standard Form of LPP

max z = c1 x1 + c2 x2 + ⋯ + cn xn , subject to
n

∑ aij xj = bi (i = 1,2, … , m)
j=1
xi ≥ 0 (i = 1,2, … , m)

Matrix Form
max z = c T x , subject to
Ax = b
x≥0

• Slack Variable: Variables which are non – negative and added to ≤

type constraint.

• Surplus Variable: Variables which are non – negative and added to ≥

type constraint.

1) Graphical Method
• The set of points S satisfying all the constraints is called the feasible
region.
• The feasible region is a closed, bounded convex set having finitely
many corner points.
• The optimal solution occurs at the boundary of the feasible region.

2) Simplex Method
• Select a set of m variables such that their coefficient matrix (B) is
identity.

• In case it is not possible to do so, add artificial variables (xa ) as per

the requirement and use either Two Phase Method or Big M Method.
• xB = Vector of basic variables = [xB1 xB2 … xBm ]T = B−1 b
• cBT = Vector of the coefficients of basic variables
• cj = jth column of C
• aj = jth column of A
• yj = B −1 aj
• z(xB ) = cBT xB
• zj = cBT yj

𝐱𝐁 𝐲𝟏 𝐲𝟐 … 𝐲𝐣 … 𝐲𝐧 𝐱𝐁
𝐲𝐣
xB1 = y11 y12 … y1j … y1n r1
xB2 = y21 y22 … y2j … y2n r2
… … … … … … … …
xBk = yk1 yk2 … 𝐲𝐤𝐣 … ykn 𝐫𝐤
… … … … … … … …
xBm = ym1 ym2 … ymj … ymn rm
z(xB ) = z1 − c1 z2 − c2 … 𝐳𝐣 − 𝐜𝐣 … zn − cn

• Result 1: If all zi − ci ≥ 0 then the optimal solution is obtained.

• Result 2: If some zi − ci < 0 and corresponding to that all yij ≤ 0
then the given LPP has unbounded solution.
• Result 3: If some zi − ci < 0 and for that some yij > 0 then there
exists a new bfs x̂B such that z(x̂B ) ≥ z(xB ).

Procedure
1. Identify the variable xj which will enter the basis:
zj − cj = min{zi − ci : zi − ci < 0 and some yij > 0}
i
2. Identify the variable xBk which will leave the basis:
rk = min{ri : yij > 0}
i
3. Replace xBk with xj .

4. Perform the row operation:

Rk
Rk →
ykj

5. Perform the row operations:

R i → R i − yij R k , ∀ i ≠ k

3) Two Phase Method

Let us assume that we added k artificial variables.

Phase 1
max z = −xa1 − xa2 − ⋯ − xak , subject to
Ax = b
x≥0

• Proceed as per the Simplex Method and try to remove the artificial
variables (xa ) from the basis.

• If z(xB ) < 0 or some xai > 0 ➔ Inconsistent Constraints.

• If some xai = 0 then try to exchange the variable.

• It is possible to exchange xai with xj if the corresponding yij ≠ 0.
• If exchange is not possible ➔ Redundancy.

Phase 2
• Consider the original objective function.
• Delete the artificial variables.
• Remove the columns corresponding to artificial variables.
• Change the last row as per the new values.
• Proceed as per the Simplex Method to obtain the optimal solution.
4) Big M Method
Let us assume that we added k artificial variables.

• Replace the objective function with the following:

z − Mxa1 − Mxa2 − ⋯ − Mxak

• Proceed as per the Simplex Method assuming that M → ∞.

5) Revised Simplex Method

max z , subject to
z − c T x{R} = 0
Ax = bx ≥ 0
x≥0

The same can be written as:

max z , subject to
A{R} x{R} = b{R}
x≥0

1 −c T 0 z −cj
A{R} = [ ], b{R} = [ ] , x{R} = [ ] , aj{R} =[a ]
0 A b x j

1 −c T
B −1 1 cBT B−1
B{R} = [ ] ⇒ B{R} = [ ]
0 B 0 B−1

Basis 𝐞 𝛃𝟏 𝛃𝟐 … 𝛃𝐦−𝟏 𝐱 𝐁{𝐑} 𝐲𝐣 𝐱 𝐁{𝐑}

(𝐳) (𝐱 𝐁𝟐 ) (𝐱 𝐁𝟑 ) (𝐱 𝐁𝐦 ) 𝐲𝐣
𝐱 𝐁𝟏 (𝐳) y1j
𝐱 𝐁𝟐 y2j r2
… −1 −1 … …
B{R} B{R} b{R}
𝐱 𝐁𝐤 ykj 𝐫𝐤
… … …
𝐱 𝐁𝐦 ymj rm
Procedure
1. Identify the variable xj which will enter the basis:
−1
zi − ci = (First row of B{R} ) × ai{R}
zj − cj = min{zi − ci : zi − ci < 0}
i

−1
2. Compute yj = B{R} aj{R} .

3. Identify the variable xBk which will leave the basis:

rk = min{ri : yij > 0}
i

4. Replace xBk with xj .

−1
5. Compute the new value of B{R} :
E = [e1 e2 … ek−1 ξ ek+1 … em ]
ei = Column vector having the ith element as 1, rest 0s
y1j y2j yk−1j 1 yk+1j ymj T
ξ = [− − … − − … − ]
ykj ykj ykj ykj ykj ykj
−1 −1
B{R} → EB{R}

6. Compute the new value of xB{R} :

−1
xB{R} → B{R} b{R}

6) Big M Method + Revised Simplex Method

Let us assume that we added k artificial variables.

• Replace the objective function with the following:

z − Mxa1 − Mxa2 − ⋯ − Mxak

• Proceed as per the Revised Simplex Method assuming that M → ∞.

Convex Function
Let S ⊆ ℝn be a convex set and f: S → ℝ. Then f is called a convex
function if for all x1 , x2 ∈ S and for all 0 ≤ λ ≤ 1, we have

f(λx1 + (1 − λ)x2 ) ≤ λf(x1 ) + (1 − λ)f(x2 )

OR

A function f is convex if for any two points P and Q on the curve, the
line segment joining P and Q is always on or above the curve between P
and Q but never below the curve.

• If the inequality is strict then the function f is called a strictly convex

function.

Concave Function
Let S ⊆ ℝn be a convex set and f: S → ℝ. Then f is called a concave
function if for all x1 , x2 ∈ S and for all 0 ≤ λ ≤ 1, we have

f(λx1 + (1 − λ)x2 ) ≥ λf(x1 ) + (1 − λ)f(x2 )

OR

A function f is convex if for any two points P and Q on the curve, the
line segment joining P and Q is always on or below the curve between P
and Q but never above the curve.

• If the inequality is strict then the function f is called a strictly concave

function.

Some Facts about Convex/Concave Functions

• If a function is both convex and concave, then it must be a linear
function.
• A function may neither be convex nor concave.
• The domain of a convex/concave function must be a convex set.
• A convex/concave function need not be continuous.
• A convex/concave function need not be differentiable.

Determining Convexity using the Hessian Matrix

The Hessian matrix H(x) is defined as:

∂2 f
H(x) = [ ]
∂xi ∂xj
n×n

• H(x) is always a symmetric matrix.

• Positive semi – definite ⇔ y T H(x)y ≥ 0 ∀ y ∈ ℝn .
• Positive definite ⇔ y T H(x)y > 0 ∀ y ∈ ℝn .

Theorem: Let S ⊆ ℝn be an open convex set and f: S → ℝ be twice

differentiable. Then f is a convex function on S iff the Hessian matrix
H(x) is positive semi – definite ∀ x ∈ S.

In general, if f: ℝn → ℝ is a quadratic form, i.e. f(x) = x T Qx, where Q is

real and symmetric then ∇f(x) = 2Qx and ∇2 f(x) = H = 2Q. Therefore,
• if Q is positive definite, then f is a strictly convex function.
• if Q is positive semi – definite, then f is a convex function.
• if Q is negative definite, then f is a strictly concave function.
• if Q is negative semi – definite, then f is a concave function.
• if Q is indefinite, then f is neither a convex nor a concave function.

Result: If Q is a real and symmetric matrix, then it is

• positive definite iff all λi > 0.
• positive semi – definite iff all λi ≥ 0.
• indefinite iff some λi > 0 and some λi < 0.
Convex Programming Problems
Optimization Problem Nature of 𝐟 Nature of 𝐠
min f(x), subject to Convex Convex
g i (x) ≤ 0 (i = 1,2, … , m)
min f(x), subject to Convex Concave
g i (x) ≥ 0 (i = 1,2, … , m)
max f(x), subject to Concave Convex
g i (x) ≤ 0 (i = 1,2, … , m)
max f(x), subject to Concave Concave
g i (x) ≥ 0 (i = 1,2, … , m)

KKT Conditions for Constrained Optimization Problem

min f(x), subject to
g i (x) ≤ 0 (i = 1,2, … , m)

For this optimization problem, the Karush-Kuhn-Tucker (KKT) optimality

conditions are:
m

∇f(x ∗ ) + ∑ λ∗i ∇g i (x ∗ ) = 0
i=1
∗ ∗
λi g i (x ) = 0 (i = 1,2, … , m)
g i (x ∗ ) ≤ 0 (i = 1,2, … , m)
λ∗i ≥ 0 (i = 1,2, … , m)

• These are the necessary conditions for a point x ∗ to be a local

minimum.
• If the problem is convex, these conditions are also sufficient to
guarantee a global minimum.

Unconstrained Minimization Problems

Consider the unconstrained minimization problem minn f(x).
x∈ℝ
Desirable properties which we ideally expect the algorithm to possess:

• Descent Property: An algorithm is said to have the descent property

if the objective function value decreases as we go through the
sequence {x k }, i.e. f(x k+1 ) < f(x k ) for all k.

• Quadratic Termination Property (QTP): An algorithm is said to have

quadratic termination property if the minimum of a positive definite
quadratic form in n variables is reached in at most n iterations.

• Globally Convergent: An algorithm is said to be globally convergent if

starting from any point x 0 ∈ ℝn , the sequence {x k } always converges
to a point of the solution set.

• Order of Convergence: Let the sequence {x k } converge to a point x̅

and let x k ≠ x̅ for sufficiently large k. The quantity ‖x k − x̅‖ is called
the error of the k th iterate x k . Suppose that there exists p and
0 < a < ∞ such that

‖x k+1 − x̅‖
lim =a
k→∞ ‖x k − x
̅‖p

then p is called the order of convergence of the sequence {x k }.

Unimodal Function
The function f: [a, b] → ℝ is said to be a unimodal min function if it has
only one mode, i.e. it has a single relative min, i.e. ∃ a ≤ α ≤ b such
that
• f is strictly decreasing in [a, α).
• f is strictly increasing in [α, b].

Similarly, a unimodal max function is defined.

Line Search Methods for Unimodal Functions
Basic Strategy: Let us choose two distinct points (say) x1 and x2 in [a, b]
i.e. a ≤ x1 < x2 ≤ b. Let xmin denote the point giving the minimum
value of f(x) over [a, b]. Since f is unimodal min function, it is clear that
• f(x1 ) ≤ f(x2 ) ⇒ xmin ∈ [a, x2 ].
• f(x1 ) > f(x2 ) ⇒ xmin ∈ [x1 , b].

In the following two methods, these notations are used:

• xL,k = lower limit of the search interval at the k th iteration
• xU,k = upper limit of the search interval at the k th iteration
• xp,k = 1st trial point at the k th iteration
• xq,k = 2nd trial point at the k th iteration
• Ep,k = f(xp,k ) = value of the function at the 1st trial point
• Eq,k = f(xq,k ) = value of the function at the 2nd trial point
• Ik = xU,k − xL,k = length of the search interval at the k th iteration
• IkL = length of the left part of the search interval Ik
• IkR = length of the right part of the search interval Ik

1) The Golden Section Rule

• IkL = IkR for all k.
• xp,k+1 = xq,k and xq,k+1 is computed afresh for the interval Ik+1
R
(or
L
xq,k+1 = xp,k and xp,k+1 is computed afresh for the interval Ik+1 ). In
view of this we have
Ik = Ik+1 + Ik+2 for all k.

• The ratio of the lengths of two successive search intervals is a

constant, i.e.
Ik Ik+1 1 + √5
= =c= = 1.618
Ik+1 Ik+2 2

On solving we obtain:
xp,k = xU,k − 0.618Ik
xq,k = xL,k + 0.618Ik

Example: min x 2 over [−5,15].

𝐤 𝐱 𝐋,𝐤 𝐱 𝐔,𝐤 𝐱 𝐩,𝐤 𝐱 𝐪,𝐤 𝐄𝐩,𝐤 𝐄𝐪,𝐤 𝐋/𝐑
1 −5.00 15.00 2.64 7.36 6.96 54.1 L
2 −5.00 7.36 −0.27 2.64 0.08 6.96 L
3 −5.0 2.64 −2.08 −0.27 4.33 0.08 R
4 −2.08 2.64 −0.27 0.84 0.08 0.71 L
5 −2.08 0.84 −0.96 −0.27 0.92 0.08 R
6 −0.96 0.84 −0.27 0.15 0.08 0.02 R
7 −0.27 0.84

We stop the process at the k th step iff Ik ≤ ε.

2) The Fibonacci Search Method

Fibonacci Sequence: F0 = 1, F1 = 1, Fi = Fi−1 + Fi−2 (i ≥ 2).

• IkL = IkR for all k.

• Ik = Ik+1 + Ik+2 for all k.

• The ratio of the lengths of two successive search intervals depends
on the number of iterations to be performed (n), i.e.
Ik Fn−k+1
=
Ik+1 Fn−k

Justification: If we wish to stop at the nth iteration, then

In+1 ≈ In
In = In
In−1 = In + In+1 = 2In
In−2 = In−1 + In = 3In
In−3 = In−2 + In−1 = 5In
…
I1 = Fn In

On solving we obtain:
Fn−k
xp,k = xU,k − ( )I
Fn−k+1 k
Fn−k
xq,k = xL,k + ( )I
Fn−k+1 k

Example: min x 2 over [−5,15].

𝐤 𝐅𝐧−𝐤 𝐱 𝐋,𝐤 𝐱 𝐔,𝐤 𝐱 𝐩,𝐤 𝐱 𝐪,𝐤 𝐄𝐩,𝐤 𝐄𝐪,𝐤 𝐋/𝐑
𝐅𝐧−𝐤+𝟏
1 13/21 −5.00 15.00 2.64 7.38 6.88 54.6 L
2 8/13 −5.00 7.38 −0.24 2.64 0.06 6.88 L
3 5/8 −5.00 2.62 −2.14 −0.24 4.67 0.06 R
4 3/5 −2.14 2.62 −0.24 0.72 0.06 0.52 L
5 2/3 −2.14 0.72 −1.20 −0.24 1.44 0.06 R
6 1/2 −1.20 0.72 −0.24 −0.24 0.06 0.05 R
+ 0.01
7 1 −0.24 0.72

n can be computed by using I1 = Fn In and In ≤ ε.

Relation Between the Fibonacci Search Method and the Golden
Section Rule
Let R F and R GS denote the reduction factors for the Fibonacci Search
Method and the Golden Section Rule respectively. Then

In 1 √5
RF = = ≈ n+1 (For large n)
I1 Fn c

In 1
R GS = = n−1
I1 c

R GS c 2
⇒ = = 1.17
RF √5

Thus, for the same number of function evaluations, the final search
interval for the Fibonacci Search Method will be 17% smaller than the
one obtained by the Golden Section Rule.

Methods for Solving Unconstrained Minimization Problems

x k+1 = x k + ̅̅̅
αk dk

x k = current solution
̅̅̅
αk = step size
dk = direction of movement

1) The Steepest Descent Method

k
∇f(x k )
d =−
‖∇f(x k )‖
̅̅̅k > 0 is chosen such that h(α
α ̅̅̅k ) = min h(αk ) , h(αk ) = f(x k + αk dk ),
k
α ≥0
′ ̅̅̅k ) = 0 to find the value of ̅̅̅
i.e. solve for h (α αk .
• Has descent property.
• Does not have quadratic termination property.
• Is globally convergent.
• Has order of convergence p = 1.

Theorem: Let dk and dk+1 be two consecutive directions generated by

the Steepest Descent Method. Then

〈dk+1 , dk 〉 = 0

where 〈, 〉 denotes the standard innert product in ℝn .

A geometrical interpretation of the above theorem could be that the

vector −∇f(x k ) which is normal to the surface f(x) = constant at x k , is
tangent to the surface at the point x k+1 . Therefore in ℝ2 , the
movement of the Steepest Descent Method will be as shown in the
figure below.
Example: min 3x12 − 4x1 x2 + 2x22 + 4x1 + 6 over (x1 , x2 ) ∈ ℝ2 .
𝐤 𝐱𝐤 𝛁𝐟(𝐱 𝐤 ) 𝐝𝐤 ̅̅
𝛂̅̅𝐤
1 0 4 −1 2/3
[ ] [ ] [ ]
0 0 0
2 −2/3 0 0 2/3
[ ] [ ] [ ]
0 8/3 −1
3 −2/3 8/3 −1 1/6
[ ] [ ] [ ]
−2/3 0 0
4 −10/9 0 0 1/4
[ ] [ ] [ ]
−2/3 16/9 −1
5 −38/27 16/9 −1 1/6
[ ] [ ] [ ]
−10/9 0 0

2) Newton’s Method
−1
̅̅̅
αk dk = − (H(x k )) ∇f(x k )

• Has descent property.

• Has quadratic termination property.
• Not globally convergent.
• Has order of convergence p = 2.

Example: min 8x12 − 4x1 x2 + 5x22 over (x1 , x2 ) ∈ ℝ2 .

5
Let x 0 = [ ].
2
16x1 − 4x2 72
∇f(x 0 ) = [ ] =[ ]
−4x1 + 10x2 (x ,x )=(5,2) 0
1 2
16 −4
H(x 0 ) = [ ]
−4 10
−1 1 10 4
(H(x 0 )) = [ ]
144 4 16
−1 0
⇒ x1 = x 0 − (H(x 0 )) ∇f(x 0 ) = [ ]
0
3) Modified Newton’s Method
dk = −Mk ∇f(x k )
̅̅̅k > 0 is chosen such that h(α
α ̅̅̅k ) = min h(αk ) , h(αk ) = f(x k + αk dk ),
k
α ≥0
′ ̅̅̅k ) = 0 to find the value of ̅̅̅
i.e. solve for h (α αk .
Given the point x k , we fix a constant δ > 0 and calculate all eigenvalues
of H(x k ). Let εk be the smallest non negative constant for which all
eigenvalues of the matrix H(x k ) + εk I are greater than or equal to δ.

−1
Mk = (H(x k ) + εk I)

• Has descent property.

• Has quadratic termination property.
• Is globally convergent.
• Has order of convergence p = 2.

4) The Conjugate Gradient Method

Conjugate Set: Let Q be an n × n positive definite matrix. A set
{d0 , d1 , … , dk−1 } of k vectors in ℝn is said to be conjugate with respect
T
to Q if every two of them are so, i.e. (di ) Qdj = 0 (i ≠ j).

Consider the unconstrained minimization problem

1
minn f(x) = x T Qx − bT x
x∈ℝ 2

Result: Let {d0 , d1 , … , dn−1 } be a set of n non – zero vectors in ℝn

which are conjugate with respect to Q. Then x̅, which is the unique
minimizing point of the UMP, is given by

n−1 T
(dk ) b
x̅ = ∑ ( k T k ) dk
(d ) Qd
k=0
Formulas
g k = ∇f(x k )
d0 = −g 0 , dk+1 = −g k+1 + βk dk
k k T k+1 T k
̅̅̅ (g ) d (g ) Qd
αk = − k T k , βk =
(d ) Qd (dk )T Qdk

Example: min 3x12 − 4x1 x2 + 2x22 + 4x1 over (x1 , x2 ) ∈ ℝ2 .

6 −4 −4
Q=[ ] b=[ ]
−4 4 0
0
Let x 0 = [ ].
0
4
g 0 = Qx 0 − b = [ ]
0
−4
d0 = −g 0 = [ ]
0
( 0 )T 0
̅̅̅0 = − g d 1 ̅̅̅0 d0 = [−2/3]
α 0 T 0
= x1 = x 0 + α
(d ) Qd 6 0
0
g1 = Qx1 − b = [ ]
8/3
(g1 )T Qd0 4
0
β = 0 T 0=
(d ) Qd 9
−16/9
d1 = −g1 + β0 d0 = [ ]
−8/3
(g1 )T d1 3 −2
̅̅̅
α1 =− 1 T 1= x 2 = x1 + ̅̅̅
α1 d1 = [ ]
(d ) Qd 4 −2

• Has descent property.

• Has quadratic termination property.
• Not globally convergent.
• Converges in at most n steps.
5) Fletcher and Reeves’ Method
g k = ∇f(x k )
d0 = −g 0 , dk+1 = −g k+1 + βk dk
̅̅̅ ̅̅̅k ) = min h(αk ) , h(αk ) = f(x k + αk dk ),
αk > 0 is chosen such that h(α
αk ≥0
̅̅̅k ) = 0 to find the value of ̅̅̅
i.e. solve for h′ (α αk .
T
k
(g k+1 ) g k+1
β =
(g k )T g k
Powell’s Correction: Replace x 0 by x n and go back to Step 1.

• Has descent property.

• Has quadratic termination property.
• Becomes globally convergent after incorporating Powell’s Correction.

6) Davidon – Fletcher – Powell (DFP) Method

g k = ∇f(x k )
Choose any positive definite matrix S0 (we may take S0 = I).
dk = −S k g k
̅̅̅ ̅̅̅k ) = min h(αk ) , h(αk ) = f(x k + αk dk ),
αk > 0 is chosen such that h(α k α ≥0
̅̅̅k ) = 0 to find the value of ̅̅̅
′
i.e. solve for h (α αk .
pk = x k+1 − x k , qk = g k+1 − g k
T T
pk (pk ) S k qk (qk ) S k
S k+1 k
=S + k T k− k T k k
= S k + Ak − Bk
(p ) q (q ) S q
Powell’s Correction: Replace x 0 by x n and go back to Step 1.

• Has descent property.

• Has quadratic termination property.
• Becomes globally convergent after incorporating Powell’s Correction.
Example: min 3x12 − 4x1 x2 + 2x22 + 4x1 over (x1 , x2 ) ∈ ℝ2 .
6 −4 −4
Q=[ ] b=[ ]
−4 4 0
0
Let x 0 = [ ] and S 0 = I.
0
4
g0 = [ ]
0
−4
d0 = −S 0 g 0 = [ ]
0
h(α0 ) = f(x 0 + α0 d0 ) = 48(α0 )2 − 16α0 ̅̅̅0 = 1/6
⇒α
̅̅̅0 d0 = [−2/3]
x1 = x 0 + α
0
0
g1 = [ ]
8/3
−2/3
p0 = x 1 − x 0 = [ ]
0
4
q0 = g 1 − g 0 = [ ]
−8/3

0
p0 (p0 )T 1/6 0
A = 0 T 0=[ ]
(p ) q 0 0

0
S 0 q0 (q0 )T S 0 9/13 −6/13
B = = [ ]
(q0 )T S 0 q0 −6/13 4/13
37/78 6/13
S1 = S 0 + A0 − B0 = [ ]
6/13 9/13
−16/13
d1 = −S1 g1 = [ ]
−24/13
384 1 2 64 1 4
h(α1 ) = f(x1 + α1 d1 ) = (α ) − α − ⇒ ̅̅̅
α1 = 13/12
169 13 3
−2
x 2 = x1 + ̅̅̅
α1 d1 = [ ]
−2
Methods for Solving Convex Quadratic Programming Problem
max c T x + x T Dx , subject to
Ax ≤ b
x≥0

1) Wolfe’s Method
min −(c T x + x T Dx), subject to
Ax − b ≤ 0
−Ix ≤ 0

Applying KKT Conditions we obtain:

x
[−2D A
T
−I 0] [ λ ] = [ c ]
A 0 0 I μ b
s
x, λ, μ, s ≥ 0
λi si (i = 1,2, … , m)
μj xj = 0 (j = 1,2, … , n)

• Solve by using Phase 1 of Simplex Method.

• Do not make λi and si as basic variables at the same time.
• Do not make μj and xj as basic variables at the same time.

Example: max x1 + x2 − x12 + 2x1 x2 − 2x22 , subject to

2x1 + x2 ≤ 1
x1 , x2 ≥ 0

Substituting values we obtain:

x1
x2
2 −2 2 −1 0 0 λ 1
1
[−2 4 1 0 −1 0] μ = [ 1]
1
2 1 0 0 0 1 μ 1
2
[ s1 ]
max −xa1 − xa2 , subject to
x1
x2
λ1
2 −2 2 −1 0 1 0 0 μ 1
1
[−2 4 1 0 −1 0 1 0 ] μ = [1 ]
2
2 1 0 0 0 0 0 1 x 1
a1
x a2
[ s1 ]
x, λ, μ, xa , s ≥ 0
λ1 s1 = μ1 x1 = μ2 x2 = 0

𝐱𝐁 𝐱𝟏 𝐱𝟐 𝛌𝟏 𝛍𝟏 𝛍𝟐 𝐱 𝐚𝟏 𝐱 𝐚𝟐 𝐬𝟏
x a1 = 1 2 −2 2 −1 0 1 0 0
x a2 = 1 −2 𝟒 1 0 −1 0 1 0
s1 = 1 2 1 0 0 0 0 0 1
−2 0 −2 −3 1 1 0 0 0

𝐱𝐁 𝐱𝟏 𝐱𝟐 𝛌𝟏 𝛍𝟏 𝛍𝟐 𝐱 𝐚𝟏 𝐱 𝐚𝟐 𝐬𝟏
3 1 0 5 −1 1 1 1 0
x a1 = −
2 2 2 2
1 1 1 1 0 1 0 1 0
x2 = − −
4 2 4 4 4
1 𝟓 0 1 0 1 0 1 1
s1 = − −
4 𝟐 4 4 4
3 −1 0 5 1 1 0 1 0
− −
2 2 2 2

𝐱𝐁 𝐱𝟏 𝐱𝟐 𝛌𝟏 𝛍𝟏 𝛍𝟐 𝐱 𝐚𝟏 𝐱 𝐚𝟐 𝐬𝟏
6 0 0 𝟏𝟑 −1 3 1 1 2
x a1 = − −
5 𝟓 5 5
2 0 1 1 0 1 0 0 1
x2 = −
5 5 5 5
3 1 0 1 0 1 0 1 2
x1 = − −
10 10 10 10 5
6 0 0 13 1 3 0 0 2
− −
5 5 5 5
 𝐱𝐁 𝐱𝟏 𝐱𝟐 𝛌𝟏 𝛍𝟏 𝛍𝟐 𝐱 𝐚𝟏 𝐱 𝐚𝟐 𝐬𝟏
6 0 0 1 5 3 5 5 2
λ1 = − − −
13 13 13 13 13 13
4 0 1 0 1 2 1 1 3
x2 = − − −
13 13 13 13 13 13
9 1 0 0 1 1 1 8 5
x1 = − −
26 26 13 26 130 13
0 0 0 0 0 0 1 1 0

Methods for Solving Constrained Optimization Problems

1) Method of Lagrange Multipliers

min f(x), subject to
g i (x) = 0 (i = 1,2, … , m)
m

L(x, λ1 , λ2 , … , λm ) = f(x) + ∑ λi g i (x)

i=1

Using ∇L = 0, we obtain:
∂L
= 0 (j = 1,2, … , n)
∂xj
∂L
= 0 (i = 1,2, … , m)
∂λi

Theorem: Let (x ∗ , λ∗ ) ∈ ℝn × ℝm exist such that

m

∇f(x ∗ ) + ∑ λ∗i ∇g i (x ∗ ) = 0
i=1
Let Z(x = {z ∈ ℝ : z ∇g(x = 0, z ≠ 0} and HL (x ∗ , λ∗ ) be the
∗) n T ∗)

Hessian of the Lagrangian at the point (x ∗ , λ∗ ).

Then the following holds true:

• z T HL (x ∗ , λ∗ )z > 0 ∀ z ∈ Z(x ∗ ) ⇒ x ∗ is a strict local min point.
• z T HL (x ∗ , λ∗ )z < 0 ∀ z ∈ Z(x ∗ ) ⇒ x ∗ is a strict local max point.
 x3 3y2
Example: min − + 2x, subject to x − y = 0.
3 2

x 3 3y 2
L(x, y, λ) = − + 2x + λ(x − y)
3 2

x2 + 2 − λ = 0
−3y − λ = 0
x−y=0

⇒ (x, y, λ) = (2,2, −6), (1,1, −3)

∂2 L ∂2 L
∂x 2 ∂x ∂y 2x 0
HL (x, y, λ) = 2 2 =[ ]
∂ L ∂ L 0 −3
[∂x ∂y ∂y 2 ]
4 0 2 0
⇒ HL (2,2, −6) = [ ], HL (1,1, −3) = [ ]
0 −3 0 −3

z T ∇g(x ∗ ) = 0 ⇒ z1 − z2 = 0
⇒ z T HL (2,2, −6)z = z12 > 0 ⇒ (2,2) is a strict local min point.
⇒ z T HL (1,1, −3)z = −z12 < 0 ⇒ (1,1) is a strict local max point.

2) Frank and Wolfe’s Method

min f(x), subject to
Ax = b
x≥0

T
f(x) ≈ fL (x) = f(x k ) + (x − x k ) ∇f(x k )
T
⇒ f(x k ) + x T ∇f(x k ) − (x k ) ∇f(x k )

min f(x) ⇒ min wk (x) = x T ∇f(x k )

min wk (x) = x T ∇f(x k ), subject to
Ax = b
x≥0

Let x̅̅̅k be an optimal solution of this problem. The following cases arise:
• wk (x̅̅̅k ) = wk (x k ) ⇒ x k is a KKT point.
• wk (x̅̅̅k ) < wk (x k ) ⇒ x k+1 = x k + ̅̅̅
αk dk , dk = x̅̅̅k − x k ,
̅̅̅
αk is chosen such that h(α ̅̅̅k ) = min h(αk ) , h(αk ) = f(x k + αk dk ).
k
α ∈(0,1]

• Valid for convex programming problems.

Example: min 2x12 + 2x1 x2 + 2x22 − 4x1 − 6x2 , subject to

x1 + 2x2 = 2
x1 , x2 ≥ 0

1/2
Let x 0 = [ ].
1/2

min w0 (x) = x T ∇f(x 0 ) = −x1 − 3x2 , subject to

x1 + 2x2 = 2
x1 , x2 ≥ 0

0
Solving graphically, we obtain ̅̅̅
x 0 = [ ].
1

w0 (x 0 ) = −2, w0 (x̅̅̅0 ) = −3 ⇒ w0 (x̅̅̅0 ) < w0 (x 0 ).

−1/2
d0 = ̅̅̅
x0 − x0 = [ ]
1/2
h(α0 ) = f(x 0 + α0 d0 ) ̅̅̅0 = 1
⇒α
̅̅̅0 d0 = [0]
x1 = x 0 + α
1
min w1 (x) = x T ∇f(x1 ) = −2x1 − 2x2 , subject to
x1 + 2x2 = 2
x1 , x2 ≥ 0

2
Solving graphically, we obtain x̅̅̅1 = [ ].
0

w1 (x1 ) = −2, w1 (x̅̅̅1 ) = −4 ⇒ w1 (x̅̅̅1 ) < w1 (x1 ).

2
d1 = x̅̅̅1 − x1 = [ ]
−1
h(α1 ) = f(x1 + α1 d1 ) ⇒ ̅̅̅
α1 = 1/6
1/3
x 2 = x1 + ̅̅̅
α1 d1 = [ ]
5/6

min w2 (x) = x T ∇f(x 2 ) = −x1 − 2x2 , subject to

x1 + 2x2 = 2x1 , x2 ≥ 0
x1 , x2 ≥ 0

0
Solving graphically, we obtain ̅̅̅
x 2 = [ ].
1

w0 (x 2 ) = −2, w0 (x̅̅̅2 ) = −2 ⇒ w0 (x̅̅̅0 ) = w0 (x 0 ).

⇒ x1 = 1/3, x2 = 5/6 is an optimal solution to the given problem.

3) The Penalty Function Method

min f(x), subject to
g i (x) ≤ 0 (i = 1,2, … , m)

min q(x) = f(x) + αP(x), over x ∈ ℝn .

m

P(x) = ∑ max(g i (x), 0)2

i=1
Let x(α) be an optimal solution of this problem. Then,
x ∗ = lim x(α)
α→∞

• Valid for non – convex programming problems as well.

Example: min x 2 , subject to x ≥ 1.

P(x) = max(1 − x, 0)2

q(x) = x 2 + αP(x) = x 2 + α max(1 − x, 0)2

dq
= 0 ⇒ 2x − 2α max(1 − x, 0) = 0
dx

Case 1: x ≥ 1
⇒x=0
Not possible.

Case 2: x ≤ 1
α
⇒ x(α) = < 1 (α > 0)
1+α
which is consistent with x ≤ 1. Thus, the optimal solution is
x ∗ = lim x(α) = 1
α→∞

Numerical Implementation
• Choose a suitable penalty function; P(x) = ∑m 2
i=1 max(g i (x), 0) .
• Choose an increasing sequence of positive real numbers which tend
to infinity; α1 = 1, α2 = 10, α3 = 100, and so on.
• Choose an arbitrary x 0 (= ̅̅̅ x 0 ) ∈ ℝn .
• Solve the following UMPs
min q(x, αk ) = f(x) + αk P(x)
with ̅̅̅̅̅̅
x k−1 as the starting point, to obtain x̅̅̅k .
• Continue till αk P(x̅̅̅k ) < ε.
4) The Barrier Function Method
min f(x), subject to
g i (x) ≤ 0 (i = 1,2, … , m)

min c(x) = f(x) + rB(x), over x ∈ ℝn .

m
1
B(x) = − ∑
g i (x)
i=1

Let x(r) be an optimal solution of this problem. Then,

x ∗ = lim x(r)
r→0

Define S = {x ∈ ℝn : g i (x) ≤ 0, i = 1,2, … , m}.

Also, int S denotes the interior of the set S.

Barrier Function: A function B: int S → ℝ is called a barrier function if

• B(x) ≥ 0 ∀ x ∈ int S
• B(x) is differentiable
• B(x) → +∞ as x approaches to boundary of S.

• Valid for non – convex programming problems as well.

• Does not work in case of equality constraints, i.e., g i (x) = 0.

Example: min x 2 , subject to −1 ≤ x ≤ 1.

g1 (x) = x − 1 ≤ 0
g 2 (x) = −1 − x ≤ 0

1 1 1 1 1 1
B(x) = − ( + ) = −( − )=− +
g1 (x) g 2 (x) x−1 x+1 x−1 x+1
r r
c(x) = f(x) + rB(x) = x 2 − +
x−1 x+1
dc 2r
= 0 ⇒ x( + 1) = 0
dx (x − 1)2 (x + 1)2
⇒ x(r) = 0
⇒ x ∗ = lim x(r) = 0
r→0

Numerical Implementation
1
• Choose a suitable penalty function; B(x) = − ∑m
i=1 .
gi (x)
• Choose a decreasing sequence of positive real numbers which tend
to 0; r1 = 1, r2 = 0.1, r3 = 0.01, and so on.
• Choose an arbitrary x 0 (= ̅̅̅ x 0 ) ∈ int S.
• Solve the following UMPs
min r(x, αk ) = f(x) + rk B(x)
with ̅̅̅̅̅̅
x k−1 as the starting point, to obtain x̅̅̅k .
• Continue till rk B(x̅̅̅k ) < ε.

Multi Objective Programming Problems

We will consider the following form of MOPP:
min f(x) = (f1 (x), f2 (x), … , fp (x))
x∈S

where S is a non – empty subset of ℝn representing the feasible set of

the problem and f: S → ℝp is a given vector function comprising of p
objective criteria to be minimized.

Goal Programming
• It consists of formulating an optimization problem in such a manner
that ensures that the objective criteria come close to the specified
aspiration levels in order of priorities set up by the decision maker.
• It aims at satisfaction of the goals rather than exact achievement of
the goals.
• The constraints are also treated as goals.
Aspiration Level: Aspiration level is the numerical value specified by the
decision maker that reflects his desire or satisfactory level regarding the
objective function under consideration.

Goal: An objective function along with its aspiration level is termed as a

goal.

Goal Deviation: The difference between what we actually achieve and

what we desire to achieve is called goal deviation. If it is positive, it
indicates overachievement of the goal. If it is negative, it indicates
underachievement of the goal.

Rigid Goals/Hard Goals: Actual constraints.

Soft Goals: Goals representing the original objective functions.

Formulation of Goal Programming Problem

• Specify the aspiration level for each objective function.
• Set up the goals and convert them into equations by using negative
and positive deviational variables.
• Treat the constraints present in the problem as goals and convert
them into equations by using negative and positive deviational
variables.
• Assign first priority to the hard constraints and rank all other goals
according to their importance specified by the decision maker.
• Identify the appropriate undesirable deviational variables or
expressions involving deviational variable and try to minimize them
in order of the specified priorities.

Example: min(2x1 − x2 , 4x1 − 5x2 , −x1 ), subject to

4x1 + 5x2 ≤ 20
3x1 + 2x2 ≤ 12
x1 , x2 ≥ 0
Suppose the decision maker gives the following order of priorities of the
objective functions: p2 > p3 > p1 .

He wishes to keep the first, second, and third priorities objective values
below 8, −2,1, respetively. So, the goals in order of their importance are
given by
4x1 + 5x2 ≤ 20
3x1 + 2x2 ≤ 12
4x1 − 5x2 ≤ 8
x1 ≥ 2
2x1 − x2 ≤ 1

Introducing deviational variables, we obtain:

4x1 + 5x2 + d1− − d1+ = 20
+
3x1 + 2x2 + d− 2 − d2 = 12
+
4x1 − 5x2 + d− 3 − d3 = 8
x1 + d− +
4 − d4 = 2
+
2x1 − x2 + d− 5 − d5 = 1
x j , d− +
i , di ≥ 0 (j = 1,2) (i = 1,2, … ,5)

+ − +
The undesirable variables are d1+ , d+
2 , d3 , d4 , d5 , respectively.

+ − +
Now we try to minmize (d1+ + d+
2 , d3 , d4 , d5 ) in this order only.

General Model of Goal Programming Problem

𝐥𝐞𝐱𝐢 − 𝐦𝐢𝐧(F1 (d− , d+ ), F2 (d− , d+ ), … , FK (d− , d+ )), subject to

fi (x) + d− +
i − di = vi (i = 1,2, … , p)
g j (x) + d− +
j − dj = 0 (j = 1,2, … , m)
d− + − +
i , di , dj , dj ≥ 0 (i = 1,2, … , p) (j = 1,2, … , m)

If all the functions fi and g j are linear functions of the decision variable
x then the GPP is called Linear Goal Programming Problem.
Methods for Solving Linear Goal Programming Problem

1) Graphical Method
• Used if the decision variable x belongs to ℝ2 .

+ − +
Example: lexi − min(d1+ + d+ 2 , d3 , d4 , d5 ), subject to
G1 : 4x1 + 5x2 + d1− − d1+ = 20
+
G2 : 3x1 + 2x2 + d− 2 − d2 = 12
+
G3 : 4x1 − 5x2 + d− 3 − d3 = 8
G4 : x1 + d− +
4 − d4 = 2
+
G5 : 2x1 − x2 + d− 5 − d5 = 1
x j , d− +
i , di ≥ 0 (j = 1,2) (i = 1,2, … ,5)
The final priority goal G5 is outside the current feasible region, we move
the objective line G5 parallel to itself to meet the feasible region.

The lexicographic optimal solution is x ∗ = (2,12/5) and the optimal

achievement value of the GPP is (0,0,0,0.6).

2) Sequential Goal Programming Technique

• Used in case of higher dimension set up.

Column Drop Rule: Any non basic variable that has a negative
opportunity cost zj − cj in the optimal table of a LPP can be assigned
zero value in the subseqent LPPs and therefore the column
corresponding to this variable can be dropped from the subsequent
LPPs.

Implicitly the rule states that if a non basic variable with negative
opportunity cost zj − cj is introduced in the basis at the later stages of
the algorithm it will degrade the solution in the lexi − min order.

Example: Same as the previous one.

min d1+ + d+
2 , subject to
4x1 + 5x2 + d1− − d1+ = 20
+
3x1 + 2x2 + d− 2 − d2 = 12
x i , d− +
i , di ≥ 0 (i = 1,2)

𝐱𝐁 𝐱𝟏 𝐱𝟐 𝐝𝟏+ 𝐝𝟐+
−
d1 = 20 4 5 −1 0
d−
2 = 12 3 2 0 −1
z(xB ) = 0 0 0 −1 −1

Using the column drop rule, the columns corresponding to d1+ and d+
2
are dropped from the subsequent iterations.
min d+
3 , subject to
4x1 + 5x2 + d1− − d1+ = 20
+
3x1 + 2x2 + d− 2 − d2 = 12
+
4x1 − 5x2 + d− 3 − d3 = 8
x j , d− +
i , di ≥ 0 (j = 1,2) (i = 1,2,3)

𝐱𝐁 𝐱𝟏 𝐱𝟐 𝐝+𝟑
d1− = 20 4 5 0
d−2 = 12 3 2 0
d−3 =8 4 −5 −1
z(xB ) = 0 0 0 −1

min d−
4 , subject to
4x1 + 5x2 + d1− − d1+ = 20
+
3x1 + 2x2 + d− 2 − d2 = 12
+
4x1 − 5x2 + d− 3 − d3 = 8
x1 + d − +
4 − d4 = 2
x j , d− +
i , di ≥ 0 (j = 1,2) (i = 1,2,3,4)

𝐱𝐁 𝐱𝟐 𝐝−𝟒 𝐝+𝟒
−
d1 = 12 5 −4 4
d−2 =6 2 −3 3
d−3 =0 −5 −4 4
x1 = 2 0 1 −1
z(xB ) = 0 0 −1 0

min d+
5 , subject to
4x1 + 5x2 + d1− − d1+ = 20
+
3x1 + 2x2 + d− 2 − d2 = 12
+
4x1 − 5x2 + d− 3 − d3 = 8
x1 + d− +
4 − d4 = 2
+
2x1 + d− 5 − d5 = 1
x j , d− +
i , di ≥ 0 (j = 1,2) (i = 1,2,3,4,5)
 𝐱𝐁 𝐝𝟏− 𝐝−𝟓 𝐝+𝟒
x2 = 12/5 1/5 0 4/5
d−2 = 6/5 −2/7 0 7/5
d−3 = 12 1 0 8
+
d5 = 3/5 −1/5 −1 −14/5
x1 = 2 0 0 −1
z(xB ) = 3/5 −1/5 −1 −14/5

The lexicographic optimal solution is x ∗ = (2,12/5) and the optimal

achievement value of the GPP is (0,0,0,0.6).

Evolutionary Methods and Global Optimization

• Though gradient-based methods are computationally efficient, they
typically provide a locally optimal solution and in general fail to
provide the globally optimal solution to the problem.

• In many real-life problems like combinatorial optimization,

scheduling, reliability design, resource allocation, or portfolio
optimization, one is ideally interested in finding the globally optimal
solution.

• If the problem in hand is a convex optimization problem, a local min

point is guaranteed to be the global min point. Otherwise, for non-
convex problems, a local min point need not be the global min point
of the problem.

• If one can not solve a problem or if it takes considerable time to

solve it, the problem is termed as difficult.

• The worst-case complexity of an algorithmic problem is the time

taken by the fastest algorithm that can solve the problem, as a
function of the problem size N.
• The actual time taken to solve an optimization problem depends on
the machine being isued, the quality of the code, and many such
factors. These factors constitute implementation details.

• To allow a comparison that does not depend on implementation

details, the notion of asymptotic analysis is used.

• O(N p ) ⇒ ∃ a positive integer N0 and a real number K such that the

worst-case run time of the algorithm ≤ KN p , ∀ N ≥ N0 .

• Deterministic approaches used for finding the globally optimal

solutions may take a very long time particularly for real world
problems that involve several variables.

• Most global optimization methods that are widely applied therefore

involve some randomized search steps, or are heuristic in nature.

• Algorithms that can be used to define heuristic methods, and are

applicable to a wide set of different optimization problems are called
metaheuristics.

• Examples of metaheuristics include simulated annealing, genetic

algorithms, ant colony optimization, particle swarm optimization,
and tabu search.

1) Simulated Annealing
• It is a randomized search technique based on the principles of
thermodynamics.

• Annealing, in metallurgy, is a technique that involves extensive

heating followed by a controlled cooling of a metal.
• At high temperature, the atoms in the metal are in a highly
disorganized liquid state.

• When the temperature is gradually lowered, according to a cooling

schedule, the atoms in a metal organize themselves into a highly
ordered solid state.

• The solid structure is a stable configuration corresponding to a lower

energy state. The structural properties of the solid depend on the
rate of cooling.

• SA exploits this analogy between the way in which a metal cools and
freezes into a minimum energy crystalline structure.

• While solving an optimization problem, it is experienced that the

search tends to get stuck in a local min point from which there is no
escape route, and very often this point is not a global min point of
the problem.

• SA heps us to design a possible technique to overcome this hurdle by

allowing the point to bounce over mountains and traverse new
valleys.
• Suppose the current state of an optimization problem is x (k) . If the
new state is s1 , then it is accepted; but if the new state is s2 , it is only
accepted with a certain probability.

• The acceptance probability is based on the chance of obtaining the

new state with cost (energy) f(x (k+1) ) relative to the previous state
with cost (energy) f(x (k) ), and is described by the Boltzmann
probability distribution function, i.e.

(−∆f)k
p((∆f)k ) ≈ exp ( )
bT

where (∆f)k = f(x (k+1) ) − f(x (k) ), T is the temperature, and b is

the Boltzmann’s constant that is used to normalize the energy
function values. For practical purpoes, b is taken to be 1.

Procedure
• The search is initiated at a random feasible point x (0) .
Set xmin = x (0) , fmin = f(x (0) ).
We start with an initial temperature T = T 0 which is set to a high
level.

• The change in a state from the present state to the candidate new
state is accepted if
o (∆f)k < 0.
o (∆f)k > 0, but p((∆f)k ) > r, where r ∈ (0,1) is a randomly
generated number.

• The temperature is usually lowered in a geometric progression, i.e.

T (k+1) = β(T (k) )T (k) . Here, β(T (k) ) is a function that indicates the
rate of change, and is termed as the cooling schedule.
2) Genetic Algorithms
• GA find their motivation from Charles Darwin’s famous theory of
natural selection.

• Population: The set of several alternate solutions of the given

optimization problem.

• Chromosome: Each individual in the population.

• Encoding Schemes: To apply GA to solve a given optimization

problem, one needs to encode the chromose appropriately.

• The most popular and widely used way of encoding is binary coding.
In this coding, every chromosome is a string of bits (0 or 1).

• The following linear mapping is usually used to code values of x in

(x L , x U ) as a substring s of length l bits:

L
xU − xL
x = x + (decoded value of s) ( l )
2 −1

• Gene: The chromosomes are coded as a string, with each character

or symbol in the string being called gene.

• Fitness Function: An evolution function used for determining the

fitness of each chromosome.

• The fitness function F(x) is generally derived from the objective

function f(x). The following fitness functions are often used:
For maximization problem, F(x) = f(x).
1
For minimization problem, F(x) = .
1+f(x)
• Numerical experiments have indicated that a population size of 20 −
30 is reasonable for many optimization problems.

• The selection operator selects good strings in a population and

forms a mating pool. The basic idea is that the above-average strings
are picked from the current population and their multiple copies are
inserted in the mating pool with a probability proportional to its
fitness. If n is the population size, then the probability for selecting
the ith string is string is

Fi
pi =
∑ni=1 Fi

The string with a higher fitness value has a higher probability of

being copied in the mating pool.

• The crossover operator combines a part of the chromosome of one

parent with a part of the chromosome of the other parent. A simple
way is to split each parents’ chromosome at a cut point and choose
the left half of the chromosome of the first parent and the second
half of the other parent.

Chromosome 1: 110 00100

Chromosome 2: 010 11000

Offspring 1: 110 11000

Offspring 2: 010 00100

• The crossover probability is the ratio of the number of offspring to

the population size.

• The population size is maintained constant throughout the algorithm.

• The mutation operator randomly interchanges genes, i.e. for binary
coding, switch few randomly chosen bits from 1 to 0 or from 0 to 1.

• The mutation probability is the ratio of the number of random

geene changs to the total number of genes in a population.

• Empirical studies suggest that the crossover probability should be

high, somewhere between 80% − 95%, while the mutation
probability should be low, about 0.5% − 1%.

Procedure
• Choose a coding to represent problem parameters; a selection
operator; a crossover operator; a mutation operator; population size;
crossover probability; mutation probability; maximum allowable
generation number k max .

• Set the counter k = 0. Evaluate the fitness of each string in the

population.

• If k > k max , terminate the algorithm.

• Perform selection, following by crossover, thereafter mutation.

• Evaluate the fitness of the newly generated chromosomes.

• Determine the next generation. This could be composed of newly

generated chromosomes or a mix of individuals from the previous
set and the newly generated set. The fitness value of a chromosome
is used to select it from the entire population set.