MINISTRY OF INDUSTRY & TRADE

INDUSTRIAL UNIVERSITY OF HO CHI MINH CITY

Engineering Optimization
Chapter 8 : Constrained Problems -
Optimality Criteria

Lecturer : Dr. Vo Quoc Thang

 Summary optimality conditions

● Conditions for local minimum of unconstrained problem:

– First Order Necessity Condition: f = 0

– Second Order Sufficiency Condition: H positive definite

● For convex f in convex feasible domain:

condition for global minimum:
– Sufficiency Condition: f = 0

Engineering Optimization – Constrained Problems : Optimality Criteria 2

 Boundary optima

● Today’s topic:
optimality conditions for constrained problems
g2 f
f Interior
x2 optimum

min f ( x) g1
x

s. t. g ( x)  0
h ( x) = 0
Boundary
optima
x1
Engineering Optimization – Constrained Problems : Optimality Criteria 3
 Feasible perturbations / directions

● Consider feasible space X

● Feasible perturbation: x | x + x  X , x  X

● Feasible direction s: g2
x2
line in direction s remains y X g1
in X for some finite length: x

s | y + s  X
y  X ,   0, ,   0
x1

Engineering Optimization – Constrained Problems : Optimality Criteria 4

 Boundary optimum

● Necessary condition for local boundary optimum 𝒙∗ : 𝑓(𝒙∗ ) cannot

decrease further in any feasible direction 𝒔
f
∇𝑓 𝒙∗ 𝐬 ≥ 0 ∀𝐬 g2
x2
(no feasible direction exists for g1
s
which f decreases)

● Approach for numerical algorithms:

If 𝒙𝑘 is a nonoptimal point, a move along feasible directions 𝒔𝑘 x1

should be made so that ∇𝑓 𝒙𝑘 𝐬𝑘 < 0. The step length 𝛼𝑘 is found
from solving the problem min 𝑓 𝒙𝑘 + 𝛼𝑘 𝒔𝑘 , s.t 𝒙𝑘 + 𝛼𝑘 𝒔𝑘 ∈ 𝑋

Engineering Optimization – Constrained Problems : Optimality Criteria 5

 Equality constrained problem

● First, only equality constraints considered:

– Simplest case!

– Active inequality constraints can be identified (by e.g.

Monotonicity Analysis) and be treated as equality
constraints

min f ( x) x  n
x

s. t. hi (x) = 0 i = 1 m

Engineering Optimization – Constrained Problems : Optimality Criteria 6

 Equality constrained problem (2)

● Each (functionally independent) equality constraint

would be used to eliminate 1 variable or 1 constraint
from the problem, thus dimension of the problem is
reduced :
min f ( x) x  n
x
Problem dimension
s. t. hi (x) = 0 = n−m i = 1 m
● Solutions can only exist in the feasible subspace X
of dimension n – m (hypersurface)

● Examples: n = 3, m = 2: X = line (1-D)

n = 3, m = 1: X = surface (2-D)
Engineering Optimization – Constrained Problems : Optimality Criteria 7
 Optimality conditions

● Simplest approach:

– Eliminate variables using equality constraints

– Result: unconstrained problem of dimension n – m

– Apply unconstrained optimality conditions

● But often not possible / practical:

– Elimination fails when variables cannot be explicitly

solved from equality constraints
– Some other method must be used

Engineering Optimization – Constrained Problems : Optimality Criteria 11

 Contents

● Constrained Optimization: Optimality Criteria

– Graphical solution

– Direct substitution

– Reduced gradient

– Lagrangian multipliers

– Sufficiency conditions

– Inequality constraints

– Karush-Kuhn-Tucker (KKT) conditions

– Linear programming
Engineering Optimization – Constrained Problems : Optimality Criteria 12
 Graphical solution

● In 2D problem, we can plot the contours of the objective function and

the constraint lines.
● We can see the feasible region defined by the constraints. The
approximate location of the minimum is evident by inspection of the
contours.
● If the functions were more expensive, it would become difficult or
impossible to visualize the functions and feasible space fully.

Engineering Optimization – Constrained Problems : Optimality Criteria 13

 Contents

● Constrained Optimization: Optimality Criteria

– Graphical solution

– Direct substitution

– Reduced gradient

– Lagrangian multipliers

– Sufficiency conditions

– Inequality constraints

– Karush-Kuhn-Tucker (KKT) conditions

– Linear programming
Engineering Optimization – Constrained Problems : Optimality Criteria 14
 Direct substitution: Example

Find the dimensions of a box of largest volume that can be inscribed in a

sphere of unit radius.

Solution:

● Cartesian coordinate system 𝑥1 , 𝑥2 , 𝑥3 be at the center of the sphere

● Sides of the box be 2𝑥1 , 2𝑥2 , and 2𝑥3 .

● Volume of the box: 𝑓 𝑥1 , 𝑥2 , 𝑥3 = 8𝑥1 𝑥2 𝑥3

● Corners of the box lie on the surface of the sphere of unit radius:

𝑥12 + 𝑥22 + 𝑥32 = 1

● Problem: Max 𝑓 𝑥1 , 𝑥2 , 𝑥3 = 8𝑥1 𝑥2 𝑥3

s. t 𝑥12 + 𝑥22 + 𝑥32 = 1

Engineering Optimization – Constrained Problems : Optimality Criteria 15

 Direct substitution: Example

● If we choose to eliminate 𝑥3 : 𝑥12 + 𝑥22 + 𝑥32 = 1 → 𝑥3 = 1 − 𝑥12 − 𝑥22

● Thus the objective function becomes: 𝑓 𝑥1 , 𝑥2 = 8𝑥1 𝑥2 1 − 𝑥12 − 𝑥22
which can be maximized as an unconstrained function in two variables.
● Gradient: Necessity Condition
𝜕𝑓 8𝑥2 (2𝑥12 + 𝑥22 − 1)
−
𝜕𝑥1 1 − 𝑥12 − 𝑥22 0 2𝑥12 + 𝑥22 − 1 = 0
∇𝑓 = = = →൝ 2
𝜕𝑓 8𝑥1 (𝑥12 + 2𝑥22 − 1) 0 𝑥1 + 2𝑥22 − 1 = 0
−
𝜕𝑥2 1 − 𝑥2 − 𝑥2 1 2
8
→ 𝑥1∗ = 𝑥2∗ = 1Τ 3 → 𝑥3∗ = 1Τ 3 → 𝑓𝑚𝑎𝑥 =
3 3
● Hessian matrix: Sufficiency Condition
𝜕2𝑓 𝜕2𝑓
32
𝜕𝑥1 2 𝜕𝑥1 𝜕𝑥2 −32/ 3 −16/ 3 𝐻1 = − <0
𝐇= = →൞ 3
𝜕2𝑓 𝜕2𝑓 −16/ 3 −32/ 3 𝐻2 = 256 > 0
𝜕𝑥2 𝜕𝑥1 𝜕𝑥2 2 (𝑥1∗ ,𝑥2∗ )

→ 𝐇 𝑥1∗ , 𝑥2∗ is negative definite → 𝑥1∗ , 𝑥2∗ corresponds to the maximum of 𝑓.

Engineering Optimization – Constrained Problems : Optimality Criteria 16

 Contents

● Constrained Optimization: Optimality Criteria

– Graphical solution

– Direct substitution

– Reduced gradient (pp biến đổi ràng buộc)

– Lagrangian multipliers

– Sufficiency conditions

– Inequality constraints

– Karush-Kuhn-Tucker (KKT) conditions

– Linear programming
Engineering Optimization – Constrained Problems : Optimality Criteria 17
 Reduced gradient

● Approach: build local approximation for constrained case, using (very

small) feasible perturbations:
min 𝑓 𝐱 𝐱 ∈ ℜ𝑛
𝐱
s. t. ℎ𝑖 𝐱 = 0 𝑖 = 1…𝑚
● The first-order approximations of the perturbations for objective and
constraint functions are:
𝑛
𝑇 𝜕𝐱
𝜕𝑓
𝜕𝑓 𝐱 = ∇𝑓 =෍ 𝜕𝑥
𝜕𝑥𝑖 𝑖
𝑖=1
𝑛
𝑇 𝜕ℎ𝑗
𝜕ℎ𝑗 𝐱 = ∇ℎ𝑗 𝜕𝐱 = ෍ 𝜕𝑥𝑖 = 0 𝑗 = 1…𝑚
𝜕𝑥𝑖
𝑖=1

m+1 linear equations, n unknown, where n > m

 n - m = p degrees of freedom

Engineering Optimization – Constrained Problems : Optimality Criteria 18

 Reduced gradient

● We select 𝑚 out of 𝑛 independent variables, and divide

design variables in two subsets:

– p=n-m decision/control variables d s 

(tham biến cơ sở) x= 
– m state/solution variables s d 
(tham biến cần triệt tiêu)

f n
f
m p
f
f (x) =  xi = si +  d i
i =1 xi i =1 si i =1 d i
n h j m h j p
h j
h j (x) =  xi = 0 =  si +  d i
i =1 xi i =1 si i =1 d i
dependent independent
Engineering Optimization – Constrained Problems : Optimality Criteria 19
 Reduced gradient : Optimality condition

Optimality condition for equality-constrained problem:

−1
f f h h
− = 0T
d s s d
Reduced gradient zero.

Engineering Optimization – Constrained Problems : Optimality Criteria 24

 Reduced gradient : Example

min f = mgx2 , h = L(1 − cos x1 ) − x2 = 0

x1 , x2 x1
L
 x1  d 
Take x =   =   min f = mgs x2
 x2   s  d ,s
m
s.t . h = L(1 − cos d ) − s = 0
f f
= mg , =0
s d
h h
= −1, = L sin d 
s d
𝜕𝑓 𝜕𝑓 𝜕ℎ−1 𝜕ℎ −1 𝐿 sin 𝑑
∇𝜁 = − = 0 − 𝑚𝑔 −1 = 𝑚𝑔𝐿 sin 𝑑
𝜕𝑑 𝜕𝑠 𝜕𝑠 𝜕𝑑
Equilibrium:  = 0  mgLsin d = 0  d = 0,  ,2 ,
Engineering Optimization – Constrained Problems : Optimality Criteria 25
 Contents

● Constrained Optimization: Optimality Criteria

– Graphical solution

– Direct substitution

– Reduced gradient

– Lagrangian multipliers (pp nhân tử Lagrange)

– Sufficiency conditions

– Inequality constraints

– Karush-Kuhn-Tucker (KKT) conditions

– Linear programming
Engineering Optimization – Constrained Problems : Optimality Criteria 26
 Lagrange approach

● Alternative way to formulate optimality

conditions: formulate Lagrangian:
L(x,  ) = f (x) + λ T h(x) x  n , λ  m
● Consider stationary point of L: Lagrange multipliers

  x L   x f + λ T  x h  T
L =  = =0
 λ L   ( λ λ )h 
T

−1
 f T h f h
 s + λ = 0T
 λ T
=−
s s s
 −1
 f + λ T h = 0T  f − f h h = 0T
 d d d s s d
Engineering Optimization – Constrained Problems : Optimality Criteria 27
 Example

min f = mgx2 , h = L(1 − cos x1 ) − x2 = 0

x1 , x2 x1
L
𝐿(𝐱, 𝛌) = 𝑓(𝐱) + 𝛌𝑇 𝐡(𝐱)
= 𝑚𝑔𝑥2 + 𝜆 𝐿(1 − cos 𝑥1 ) − 𝑥2 x2
m
𝑇 𝜆𝐿 sin 𝑥1
𝜕𝐿 𝜕𝐿 𝜕𝐿
𝜕𝐿 = 𝟎 = = 𝑚𝑔 − 𝜆   = mg
𝜕𝑥1 𝜕𝑥2 𝜕𝜆
𝐿(1 − cos 𝑥1 ) − 𝑥2

 mgL sin x1 = 0

Equilibrium: x1 = 0,  , 2 ,
Engineering Optimization – Constrained Problems : Optimality Criteria 28
 Geometrical interpretation

● For single equality constraint: simple geometrical

interpretation of Lagrange optimality condition:

f h x2
f
+ = 0T
x x h
Meaning: h
f h
//
x x f
x1
Gradients parallel 
tangents parallel  h tangent to isolines

Engineering Optimization – Constrained Problems : Optimality Criteria 30

 Summary

x2
● First order optimality f
condition for equality h
constrained problem: h
– Zero reduced gradient:

f f h
−1
h f
− = 0T x1
d s s d
– Equivalent: stationary
Lagrangian: 𝜕𝐿
= 𝟎𝑇
L = f (x) + λ T h(x)  𝜕𝐱
𝜕𝐿
= 𝟎𝑇
𝜕𝛌
Engineering Optimization – Constrained Problems : Optimality Criteria 31
 Contents

● Constrained Optimization: Optimality Criteria

– Graphical solution

– Direct substitution

– Reduced gradient

– Lagrangian multipliers

– Sufficiency conditions

– Inequality constraints

– Karush-Kuhn-Tucker (KKT) conditions

– Linear programming
Engineering Optimization – Constrained Problems : Optimality Criteria 32
 Sufficiency?

● Until now, only stationary points considered. Does not

guarantee minimum!
● Lagrange
condition:

f + T h = 0

Two points satisfy the first order optimality

conditions; one is a constrained minimum,
and the other is a constrained maximum

Engineering Optimization – Constrained Problems : Optimality Criteria 33

 Sufficiency conditions

● Optimality conditions for equality constrained problem:

1. Necessary condition: stationary point when:

L = 0 (1)

2. Sufficient condition: minimum when (1) and:

 2
(x ) 2 x  0
T L
(2)
x
on tangent subspace.

Engineering Optimization – Constrained Problems : Optimality Criteria 36

 Example
x2
f
min f (x) = x1 + x2
2 2
x1 , x2
h

s.t. h(x) = x1 + 2 x2 − 1 = 0
2 2
x1

 L(x) = x + x2 +  (x + 2 x2 −1)
2 2 2 2
1 1

dL  2 x1 (1 +  ) 
T
dL
 = = + 2 −1
2 2
 , x 2 x
dx 2 x2 (1 + 2 ) d
1

1. Necessary condition: stationary point when L = 0

𝑥1 = 0, 𝑥2 = ± 1/2, 𝜆𝐴 = −1/2
𝑥2 = 0, 𝑥1 = ±1, 𝜆𝐵 = −1
Engineering Optimization – Constrained Problems : Optimality Criteria 37
 Example

2. Sufficient condition: Hessian should be positive definite

𝜕2𝐿 𝜕2𝐿
𝜕𝑥1 2 𝜕𝑥1 𝜕𝑥2 2 + 2𝜆 0
𝐇= =
𝜕2𝐿 𝜕2𝐿 0 2 + 4𝜆
𝜕𝑥2 𝜕𝑥1 𝜕𝑥2 2 (𝑥1∗ ,𝑥2∗ )

The Hessian is positive semi-definite for the case where 𝜆𝐴 =

− 1/2, and therefore there are 2 points where 𝑓 is a minimum.

Engineering Optimization – Constrained Problems : Optimality Criteria 38

 Example

● The Lagrangian for this problem is:

1 2
𝐿 𝐱, 𝛌 = 𝑓 𝐱 + 𝛌 𝐡 𝐱 = 𝑥1 + 2𝑥2 + 𝜆 𝑥1 + 𝑥22 − 1
𝑇
4
● Differentiating this to get the first-order optimality conditions:
𝜕𝐿 1 − 2
= 1 + 𝜆𝑥1 = 0 𝑥1
𝜕𝑥1 2 𝑥𝐴 = 𝑥 = 2 , 𝜆𝐴 = 2
2 −
𝜕𝐿 2
= 2 + 2𝜆𝑥2 = 0 →
𝜕𝑥2 2
𝑥1
𝜕𝐿 1 2 𝑥𝐵 = 𝑥 = 2 , 𝜆𝐵 = − 2
= 𝑥1 + 𝑥22 − 1 = 0 2
𝜕𝜆 4 2
● To determine if either of these points is a minimum, we need to check the
second-order conditions by evaluating the Hessian of the Lagrangian.

Engineering Optimization – Constrained Problems : Optimality Criteria 39

 Example

● Hessian matrix of the Lagrangian:

𝜕2𝐿 𝜕2𝐿
1
𝜕𝑥1 2 𝜕𝑥1 𝜕𝑥2
𝐇= = 2𝜆 0
𝜕2𝐿 𝜕2𝐿
0 2𝜆
𝜕𝑥2 𝜕𝑥1 𝜕𝑥2 2 (𝑥1∗ ,𝑥2∗ )

● The Hessian is positive definite for the case where 𝜆𝐴 = 2, and therefore 𝑥𝐴
is a minimum. The Hessian is negative definite for 𝜆𝐵 = − 2 so this is not a
minimum; instead, it is a maximum.

Engineering Optimization – Constrained Problems : Optimality Criteria 40

 Contents

● Constrained Optimization: Optimality Criteria

– Graphical solution

– Direct substitution

– Reduced gradient

– Lagrangian multipliers

– Sufficiency conditions

– Inequality constraints

– Karush-Kuhn-Tucker (KKT) conditions

– Linear programming
Engineering Optimization – Constrained Problems : Optimality Criteria 41
 Inequality constrained problems

● Consider problem with only inequality constraints:

min f ( x) x  n
x

s. t. g i ( x)  0 i = 1 m
● At optimum, only active constraints matter:

min f ( x) x  n
x

s. t. g j ( x) = 0 j = 1 k  m

● Optimality conditions similar to equality constrained

problem

Engineering Optimization – Constrained Problems : Optimality Criteria 42

 Optimality condition

L (x) = f (x) + μT g (x)

● Active constraints: g j = 0, j  0
Inactive constraints: gi  0
● Formulation including all inequality constraints:

𝐿(𝐱) = 𝑓(𝐱) + 𝛍𝑇 𝐠(𝐱)

L f T g ii ggii = 0 i = 1 m
= +μ = 0T and Complementarity
x x x μi 0 0
condition (if 𝑔𝑖 < 0,
i =𝜇𝑖1=
0mand
then
vice versa)

Engineering Optimization – Constrained Problems : Optimality Criteria 47

 Contents

● Constrained Optimization: Optimality Criteria

– Graphical solution

– Direct substitution

– Reduced gradient

– Lagrangian multipliers

– Sufficiency conditions

– Inequality constraints

– Karush-Kuhn-Tucker (KKT) conditions

– Linear programming
Engineering Optimization – Constrained Problems : Optimality Criteria 49
 Karush-Kuhn-Tucker conditions

● Combining Lagrange conditions for equality and

inequality constraints yields KKT conditions for general
problem: min f ( x)
x
Lagrangian:
s. t. g ( x)  0
L = f ( x) + μ T g ( x) + λ T h ( x)
h ( x) = 0
L f g i hi
 = +  i +  i =0 (optimality)
x x x x
and g  0, h = 0 (feasibility)
λ  0, μ  0, i gi = 0 (complementarity)

Engineering Optimization – Constrained Problems : Optimality Criteria 50

 Sufficiency

● KKT conditions are necessary conditions for local

constrained minima
● For sufficiency, consider the sufficiency conditions
based on the active constraints:
2𝐿
𝜕
𝜕𝐱 𝑇 2 𝜕𝐱 > 0
𝜕𝐱
on tangent subspace of h and active g.

● Interpretation: objective and feasible domain

locally convex

Engineering Optimization – Constrained Problems : Optimality Criteria 51

 Additional remarks

● Global optimality:

– Globally convex objective function?

– And convex feasible domain?

Then KKT point gives global optimum

● Pitfall:

– Sign conventions for Lagrange multipliers in

KKT condition depend on standard form!
– Presented theory valid for negative null form

Engineering Optimization – Constrained Problems : Optimality Criteria 52

 Additional remarks

Engineering Optimization – Constrained Problems : Optimality Criteria 53

 Example

Problem with 1 inequality constraint:

minimize 𝑓 𝑥1 , 𝑥2 = 𝑥1 + 2𝑥2
𝑥1 ,𝑥2
1 2
subject to 𝑔 𝑥1 , 𝑥2 = 𝑥1 + 𝑥22 − 1 ≤ 0
4
Solution:
The Lagrangian for this problem is:
1 2
𝐿 𝐱 =𝑓 𝐱 + 𝛍𝑇 𝐠 𝐱 = 𝑥1 + 2𝑥2 + 𝜇 𝑥1 + 𝑥22 − 1
4
The KKT condition (necessary conditions for local constrained minima):
𝜕𝐿 𝜕𝑓 𝑇
𝜕𝐠 𝜇𝑖 𝑔𝑖 = 0
= +𝛍 = 𝟎𝑇 and
𝜕𝐱 𝜕𝐱 𝜕𝐱 𝛍≥𝟎
1
1 + 𝜇 𝑥1 = 0
2
2 + 𝜇 2𝑥2 = 0
1 2 1 2
𝜇 𝑥1 + 𝑥22 − 1 = 0, 𝜇 ≥ 0, 𝑥1 + 𝑥22 − 1 ≤ 0
4 4
Engineering Optimization – Constrained Problems : Optimality Criteria 54
 Example (2)

The possibilities are as follows:

1. 𝜇 = 0:
1=0
2=0
1 2 → has no solution
𝑥1 + 𝑥22 − 1 ≤ 0
4
2. 𝜇 ≠ 0:
𝑥1 = −2/μ 𝑥1 = −2/μ
𝑥2 = −1/μ 𝑥2 = −1/μ 𝑥1 = − 2
1 2 1 1 𝑥2 = −1/ 2
𝑥1 + 𝑥22 − 1 = 0 2
+ 2
−1=0
4 𝜇 𝜇 𝜇= 2
𝜇>0 𝜇>0
1
→ 𝒙∗ , 𝜇∗ = − 2, − , 2
2

Engineering Optimization – Constrained Problems : Optimality Criteria 55

 Example (3)

Differential quadratic form (sufficient condition):

𝜕𝐱 𝑇 𝐇𝜕𝐱 > 0 on tangent subspace of h and active g
Hessian matrix of the Lagrangian:
𝜕2𝐿 𝜕2𝐿
𝜇 2
𝜕𝑥12 𝜕𝑥1 𝜕𝑥2 0 0
𝐇= = 2 = 2
𝜕2𝐿 𝜕2𝐿 0 2𝜇 𝒙∗,𝜇∗ 0 2 2
𝜕𝑥2 𝜕𝑥1 𝜕𝑥22 𝒙∗,𝜇∗
2 2 2
𝑇 0 𝜕𝑥1
𝜕𝐱 𝐇𝜕𝐱 = 𝜕𝑥1 𝜕𝑥2 2 = 𝜕 𝑥1 + 2 2𝜕 2 𝑥2 > 0
𝜕𝑥2 2
0 2 2
which means that 𝐇, on the plane ∇ℎ𝜕𝒙 = 0, is positive-definite and
1 𝑇
the point − 2, − is a local minimum.
2

Engineering Optimization – Constrained Problems : Optimality Criteria 56

 Example (4)

Problem with 2 inequality constraints:

minimize 𝑓 𝑥1 , 𝑥2 = 𝑥1 + 2𝑥2
𝑥1 ,𝑥2
1 2
subject to 𝑔1 𝑥1 , 𝑥2 = 𝑥1 + 𝑥22 − 1 ≤ 0, 𝑔2 𝑥2 = −𝑥2 ≤ 0
4
Solution:
The Lagrangian for this problem is:
1 2
𝐿 𝐱 = 𝑓 𝐱 + 𝛍𝑇 𝐠 𝐱 = 𝑥1 + 2𝑥2 + 𝜇1 𝑥1 + 𝑥22 − 1 + 𝜇2 (−𝑥2 )
4
The KKT condition (necessary conditions for local constrained minima):
1
1 + 𝜇1 𝑥1 = 0
2
2 + 𝜇1 2𝑥2 − 𝜇2 = 0
1 1 2
𝜇1 𝑥12 + 𝑥22 − 1 = 0, 𝜇1 ≥ 0, 𝑥1 + 𝑥22 − 1 ≤ 0
4 4
𝜇2 −𝑥2 = 0, 𝜇2 ≥ 0, −𝑥2 ≤ 0

Engineering Optimization – Constrained Problems : Optimality Criteria 57

 Example (5)

The possibilities are as follows:

1. 𝜇1 ≠ 0, 𝜇2 = 0: → 𝑔1 = 0
1
1 + 𝜇1 𝑥1 = 0 𝑥1 = − 2
2
2 + 𝜇1 2𝑥2 = 0 𝑥2 = −1/ 2
→ has no solution
1 2 𝜇1 = 2
𝑥1 + 𝑥22 − 1 = 0
4 𝑥2 ≥ 0
−𝑥2 ≤ 0
2. 𝜇1 = 0, 𝜇2 = 0:
1=0
2=0
1 2 → has no solution
𝑥1 + 𝑥22 − 1 ≤ 0
4
−𝑥2 ≤ 0

Engineering Optimization – Constrained Problems : Optimality Criteria 58

 Example (6)

The possibilities are as follows:

3. 𝜇1 ≠ 0, 𝜇2 ≠ 0: → 𝑔1 = 0, 𝑔2 = 0
1
1 + 𝜇1 𝑥1 = 0
2 𝑥1 = −2/μ1 𝑥1 = −2/μ1 𝜇1 = 1
2 + 𝜇1 2𝑥2 − 𝜇2 = 0 𝜇2 = 2 𝜇2 = 2 𝜇2 = 2
1 2 𝑥12 = 4 𝑥1 = ±2 𝑥1 = −2
𝑥1 + 𝑥22 − 1 = 0 𝑥2 = 0 𝑥2 = 0 𝑥2 = 0
4
−𝑥2 = 0
4. 𝜇1 = 0, 𝜇2 ≠ 0: → 𝑔2 = 0
1=0
2 − 𝜇2 = 0
1 2 → has no solution
𝑥1 + 𝑥22 − 1 ≤ 0
4
−𝑥2 = 0
→ 𝒙∗ , 𝜇∗ = −2,0,1,2

Engineering Optimization – Constrained Problems : Optimality Criteria 59

 Example (7)

Differential quadratic form (sufficient condition):

𝜕𝐱 𝑇 𝐇𝜕𝐱 > 0 on tangent subspace of h and active g
Hessian matrix of the Lagrangian:
𝜕2𝐿 𝜕2𝐿
𝜇1 1
𝜕𝑥12 𝜕𝑥1 𝜕𝑥2 0
𝐇= 2 2 = 2 = 2 0
𝜕 𝐿 𝜕 𝐿 0 2𝜇1 𝒙∗,𝜇∗ 0 2
𝜕𝑥2 𝜕𝑥1 2
𝜕𝑥2 𝒙∗ ,𝜇∗
1 1 2
𝑇 0 𝜕𝑥1
𝜕𝐱 𝐇𝜕𝐱 = 𝜕𝑥 1 𝜕𝑥 2 2 = 𝜕 𝑥1 + 2𝜕 2 𝑥2 > 0
𝜕𝑥2 2
0 2
which means that 𝐇, on the plane ∇ℎ𝜕𝒙 = 0, is positive-definite and
the point −2,0 𝑇 is a local minimum.

Engineering Optimization – Constrained Problems : Optimality Criteria 60

 Contents

● Constrained Optimization: Optimality Criteria

– Graphical solution

– Direct substitution

– Reduced gradient

– Lagrangian multipliers

– Sufficiency conditions

– Inequality constraints

– Karush-Kuhn-Tucker (KKT) conditions

– Linear programming
Engineering Optimization – Constrained Problems : Optimality Criteria 61
 Linear programming problem

● Linear objective and constraint functions:

min c1 x1 + c2 x2 +  + cn xn = c x T
x

s.t. hi = a i x − bi = 0 i = 1 m1
T
(𝑚1 equalities)

g j = a j x − bj  0 j = m1 + 1 m2 
T
(𝑚2 inequalities)

 a1T   b1 
min cT x  T b 
x

A1 =  a2 
, b =  2 
s.t. h = A1x − b1 = 0    1   
g = A2x − b2  0  T  
a m1  bm1 

Engineering Optimization – Constrained Problems : Optimality Criteria 62

 Feasible domain

● Linear constraints divide design space into two convex

half-spaces:
x2 X1 g1 = a1 x − b1  0
T

x1
● Feasible domain = intersection of convex half-spaces:
X = X1  X 2   X m
● Result: X = convex polyhedron

Engineering Optimization – Constrained Problems : Optimality Criteria 63

 Global optimality

● Convex objective function min cT x

x
on convex feasible domain:
s.t. h = A1x − b1 = 0
KKT point = unique global
g = A2x − b2  0
optimum
● KKT conditions: L = cT x + λ T (A1x − b1 ) + μT (A 2 x − b 2 )
A1x − b1 = 0, A 2 x − b 2  0
 T
c + λ T
A + μ T
A = 0 T

  T 1 2

μ (A 2 x − b 2 ) = 0
λ  0, μ  0


Engineering Optimization – Constrained Problems : Optimality Criteria 64

 Example

Solution:
The problem is reformulated in the standard negative null form:

1 1 −2 0
→ 𝒄 = −2, −1 𝑇 , 𝒃1 = 𝑨1 = 0, 𝒃2 = 8,1.5, −1, −1 𝑇 , 𝑨𝑇2 =
2 −1 0 −2
Engineering Optimization – Constrained Problems : Optimality Criteria 65
 Example (2)

The geometric representation of the feasible space and the contours of 𝑓 are
shown by:

It is geometrically evident that the solution will be found by moving the

line 𝑓 = const up and to the right, to the furthest point of the feasible domain,
that is, point 𝑷2 . This point is the intersection of the active constraints 𝑔1 = 0
and 𝑔2 = 0.

Engineering Optimization – Constrained Problems : Optimality Criteria 66

 Example (3)

𝒄 = −2, −1 𝑇 , 𝒃1 = 𝑨1 = 0,
1 1 −2 0
𝒃2 = 8,1.5, −1, −1 𝑇 , 𝑨𝑇2 =
2 −1 0 −2

● KKT conditions: 𝐿 = 𝐜 𝑇 𝐱 + 𝛌𝑇 𝐀1 𝐱 − 𝐛1 + 𝛍𝑇 𝐀2 𝐱 − 𝐛2
𝐀1 𝐱 − 𝐛1 = 𝟎, 𝐀2 𝐱 − 𝐛2 ≤ 𝟎
𝐜 𝑇 + 𝛌 𝑇 𝐀 1 + 𝛍𝑇 𝐀 2 = 𝟎 𝑇
𝛍𝑇 𝐀2 𝐱 − 𝐛2 = 0
𝛌 ≠ 𝟎, 𝛍 ≥ 𝟎
−2 1 1 −2 0 0
+ 𝜇1 + 𝜇2 + 𝜇3 + 𝜇4 =
−1 2 −1 0 −2 0
𝜇1 𝑥1 + 2𝑥2 − 8 = 0, 𝜇1 ≥ 0
𝑥1∗ = 3.67
𝜇2 𝑥1 − 𝑥2 − 1.5 = 0, 𝜇2 ≥ 0 ൞ ∗
𝑥2 = 2.17
𝜇3 −2𝑥1 + 1 = 0, 𝜇3 ≥ 0
𝜇4 −2𝑥2 + 1 = 0, 𝜇4 ≥ 0

Engineering Optimization – Constrained Problems : Optimality Criteria 67