Optimization 3

Course Materials
• Arora, Introduction to Optimum Design, 3e, Elsevier,
(https://www.researchgate.net/publication/273120102_Introductio
n_to_Optimum_design)
• Parkinson, Optimization Methods for Engineering Design, Brigham
Young University
(http://apmonitor.com/me575/index.php/Main/BookChapters)
• Iqbal, Fundamental Engineering Optimization Methods, BookBoon
(https://bookboon.com/en/fundamental-engineering-optimization-
methods-ebook)
The General Optimization Problem
• Let 𝒙 = 𝑥1 , 𝑥2 , … , 𝑥𝑛 denote the vector of design variables; then
the general optimization problem is mathematically defined as:
Objective: min 𝑓 𝒙 , 𝒙 ∈ ℝ𝑛
𝒙

𝑔𝑖 𝒙 ≤ 0, 𝑖 = 1, … , 𝑙;
Subject to ℎ𝑖 𝒙 = 0, 𝑖 = 𝑙 + 1, … , 𝑚;
𝑥𝑗𝐿 ≤ 𝑥𝑗 ≤ 𝑥𝑗𝑈 , 𝑗 = 1, … , 𝑛

• The feasible set for the problem is defined as:

Ω = 𝑥: ℎ𝑖 𝒙 = 0, 𝑔𝑗 𝒙 ≤ 0, 𝑥𝑖𝐿 ≤ 𝑥𝑖 ≤ 𝑥𝑖𝑈
Extreme Value Theorem
• The Extreme Value Theorem provides sufficient conditions for the
existence of minimum (or maximum) of a function defined over a
complex domain. The theorem states:
A continuous function 𝑓(𝒙) defined over a closed and bounded
set 𝛺 ⊆ 𝐷(𝑓) attains its maximum and minimum in 𝛺.
• According to this theorem, if the feasible region Ω of the problem is
closed and bounded, a minimum for the problem exists.
– A set 𝑆 is closed if it contains limit points of every sequence 𝑥𝑘 ,
i.e., for 𝑥𝑘 ∈ 𝑆, lim 𝑥𝑘 = 𝑥 ∈ 𝑆. The set 𝑆 = 𝑥: 𝑥 ≤ 𝑐 is closed.
𝑘→∞
– A set 𝑆 is bounded if for every 𝑥 ∈ 𝑆, 𝑥 < 𝑐, where ∙
represents a vector norm and c is a finite number.
Stationary Points
• A single-variable function, 𝑓(𝑥), has a stationary point at 𝑥0 if its
derivative vanishes at 𝑥0 , i.e., 𝑓 ′ 𝑥0 = 0. Graphically, the slope of
the function is zero at the stationary point.
• For a multivariable function, 𝑓 𝒙 , the set of stationary points is
given as: {𝒙: 𝛻𝑓 𝒙 = 𝟎}; the set includes the maxima, minima, and
the points of inflexion.
Local and Global Minimum
• Local Minimum. A multi-variable function, 𝑓(𝒙), has a local
minimum at 𝒙∗ if 𝑓(𝒙∗ ) ≤ 𝑓(𝒙) in a small neighborhood of 𝒙∗
defined by 𝒙 − 𝒙∗ < 𝜖.
• Global Minimum. A multi-variable function 𝑓(𝒙) has a global
minimum at 𝒙∗ if 𝑓(𝒙∗ ) ≤ 𝑓(𝒙) for all 𝒙 in a the domain 𝑓(𝒙), or in
the feasible region defined by the optimization problem.
Optimality Criteria: Unconstrained Problems
• Consider the unconstrained optimization problem:
min 𝑓 𝒙 , 𝒙 ∈ ℝ𝑛
𝒙
Let 𝒙∗ is a candidate for the optimum point; define 𝛿𝒙 = 𝒙 − 𝒙∗ , and
consider the first-order Taylor series expansion of 𝑓 𝒙 around 𝒙∗
𝑓 𝒙∗ ± 𝛿𝒙 ≅ 𝑓 𝒙∗ ± 𝛻𝑓 𝒙∗ 𝑇 𝛿𝒙
• A local minimum at 𝒙∗ exists if 𝑓 𝒙∗ ≥ 𝑓 𝒙 , or if
𝛿𝑓 = 𝛻𝑓 𝒙∗ 𝑇 𝛿𝒙 ≥ 0
Since 𝛿𝒙 are arbitrary, it implies that 𝛻𝑓 𝒙∗ = 𝟎 at the optimum
First Order Necessary Condition
• FONC: If a multivariable function, 𝑓 𝒙 has a local minimum at 𝒙∗ ,
∗ 𝜕𝑓 𝒙∗
then 𝛻𝑓 𝒙 = 0; equivalently, = 0, 𝑗 = 1, … , 𝑛.
𝜕𝑥𝑗
• Note,
– All stationary points of 𝑓 𝒙 satisfy the necessary condition.
Besides minima, these include the maxima and the points of
inflexion.
– If the optimization problem is convex, then FONC are both
necessary and sufficient, i.e., a point satisfying FONCE is an
optimum.
Example – Polynomial Data Fitting

Problem: Fit an 𝑛th order polynomial: 𝑝 𝑥 = 𝑛𝑗=0 𝑝𝑗 𝑥 𝑗 to data points:

𝑥𝑖 , 𝑦𝑖 , 𝑖 = 1, … , 𝑁 > 𝑛, to minimize the mean square error
1
min 𝑓 𝑝𝑗 = 𝑁
𝑖=1 𝑦𝑖 − 𝑝0 + 𝑝1 𝑥𝑖 + ⋯ + 𝑝𝑛 𝑥𝑖𝑛 2
𝑝𝑗 2𝑁

𝜕𝑓 1 𝑗
FONC: = 𝑁
𝑖=1 𝑦𝑖 − 𝑝0 + 𝑝1 𝑥𝑖 + ⋯ + 𝑝𝑛 𝑥𝑖𝑛 (−𝑥𝑖 ) = 0
𝜕𝑝𝑗 𝑁

1 1
1 𝑖 𝑥𝑖 𝑝0 𝑦
𝑁 𝑁 𝑖 𝑖
In particular, for 𝑛 = 1, 1 1 2 𝑝1 = 1
𝑁 𝑖 𝑥𝑖 𝑁 𝑖 𝑥𝑖 𝑥𝑦
𝑁 𝑖 𝑖 𝑖

Note, these equations can be solved for 𝑝𝑗 , 𝑗 = 0, … , 𝑛.

Finally, since this problem is convex, FONC are both necessary and
sufficient for a minimum.
Second Order Conditions
• Assuming that FONC are satisfied at 𝒙∗ , i.e., 𝛻𝑓 𝒙∗ = 0, the
candidate point 𝒙∗ is further categorized using the second-order
Taylor series expansion of 𝑓(𝒙), which gives
𝛿𝑓 = 𝛿𝒙𝑇 𝛻 2 𝑓 𝒙∗ 𝛿𝒙 ≥ 0, 𝛿𝒙 = 𝒙 − 𝒙∗
• Note,
– The above quadratic form is positive (semi) definite if and only if
the Hessian matrix, 𝛻 2 𝑓 𝒙∗ , is positive (semi) definite.
– A matrix is positive definite if all its eigenvalues are positive
– A matrix is positive semidefinite if all its eigenvalues are non-
negative
Second Order Conditions
• SONC: If 𝒙∗ is a local minimizer of 𝑓 𝒙 , then 𝛻 2 𝑓 𝒙∗ ≥ 0.
• SOSC: If 𝒙∗ satisfies 𝛻 2 𝑓 𝒙∗ > 0, then 𝒙∗ is a local minimizer of
𝑓 𝒙 .
• Note
– If 𝒙∗ satisfies 𝛻 2 𝑓 𝒙∗ ≥ 0, then it is a local minimum, which may
also include other points as in the case of an interval.
– The SOSC provides a stronger condition than SONC, and can be
used to prove the existence of a minimum at 𝒙∗
– In the event that 𝛻𝑓 𝒙∗ = 𝛻 2 𝑓 𝒙∗ = 0, the lowest nonzero
derivative must be even-ordered for stationary points (necessary
condition), and be positive for local minimum (sufficient condition).
Example: Open box problem
What is the largest volume for an open box that can be constructed
from a sheet of paper (8.5”x11”) by cutting out squares at the
corners and folding the sides?
• Let 𝑥 represent the side of the squares to be cut; then the
unconstrained optimization problem is defined as:
max 𝑓(𝑥) = 𝑥(8.5 − 2𝑥)(11 − 2𝑥)
𝑥

• FONC: 𝛻𝑓 𝑥 ∗ = 0 → 𝑥 ∗ = 1.585, 4.915

• SOC: 𝛻 2 𝑓 1.585 = −39.95
𝛻 2 𝑓 4.915 = 39.95
By SOSC, 𝑥 ∗ = 4.915 is a minimum
Optimality Criteria: Equality Constraints
• Consider an optimization problem with a single equality constraint:

min 𝑓 𝒙 , 𝒙 ∈ ℝ𝑛 , subject to: ℎ 𝒙 = 0

𝒙
• Define a Lagrangian function: ℒ 𝒙, 𝜆 = 𝑓 𝒙 + 𝜆ℎ 𝒙 , then
𝜕𝑓 𝜕ℎ
FONC: 1. 𝛻ℒ 𝒙∗ = 0, or + 𝜆𝑖 𝑥 = 0; 𝑗 = 1, … , 𝑛
𝜕𝑥𝑗 𝜕𝑥𝑗 𝑗

𝜕ℒ
2. =ℎ 𝒙 =0
𝜕𝜆
Note, taken together, these are 𝑛 + 1 equations in 𝑛 + 1 unknowns:
𝑥𝑖 , 𝑖 = 1, … , 𝑛 and 𝜆; hence they can be solved to obtain 𝒙∗ and 𝜆∗
Equality Constrained Problems
• Consider a problem with multiple equality constrains:
min 𝑓 𝒙 , 𝒙 ∈ ℝ𝑛 , subject to: ℎ𝑖 𝒙 = 0, 𝑖 = 1, … , 𝑙
𝒙
• Define a Lagrangian function:
𝑙
ℒ 𝒙, 𝝀 = 𝑓 𝒙 + 𝑖=1 𝜆𝑖 ℎ𝑖 (𝒙)
𝜕ℒ 𝜕𝑓 𝑙 𝜕ℎ𝑖
FONC: = + 𝑖=1 𝑖 𝜕𝑥 𝑥𝑗
𝜆 = 0; 𝑗 = 1, … , 𝑛
𝜕𝑥𝑗 𝜕𝑥𝑗 𝑗

ℎ𝑖 𝒙 = 0, 𝑖 = 1, … , 𝑙
Note:
– A total 𝑛 + 𝑙 equations, need to be solved for 𝑥𝑗 , 𝑗 = 1, … , 𝑛 and
𝜆𝑖 , 𝑖 = 1, … , 𝑙
– The Lagrange multipliers, 𝜆𝑖 for the equality constraint can take on
both positive and negative values
Example: Soda Can Design
• Example: soda can design with one equality constraint:
Objective: min 𝑓 = 12𝜋𝑑2 + 𝜋𝑑ℎ,
𝑑,𝑙
Subject to: 1
4
𝜋𝑑 2 ℎ − 𝑉 = 0
𝜋𝑑 2 𝜋𝑑 2 ℎ
• Define ℒ 𝑑, ℎ, 𝜆 = 2
+ 𝜋𝑑ℎ + 𝜆
4
−𝑉
𝜕ℒ 𝜋𝑑ℎ
• FONC: = 𝜋𝑑 + 𝜋ℎ + 𝜆 =0
𝜕𝑑 2
𝜕ℒ 𝜋𝑑 2
= 𝜋𝑑 + 𝜆 =0
𝜕ℎ 4
𝜕ℒ 𝜋𝑑 2 ℎ
= −𝑉 =0
𝜕𝜆 4
3 4𝑉 4
• FONC are collectively solved to obtain: ℎ∗ = 𝑑∗ = , 𝜆∗ = − ∗
𝜋 𝑑
Example: Soda Can Design
• MATLAB (to rescue)
syms d h V positive
syms lam real
f=pi*d*(d/2+h);
g1=pi*d^2*h/4-V;
L=f+lam*g1;
dL1=diff(L,d);
dL2=diff(L,h);
S1=solve(dL1,dL2,g1, 'Real', true);
V=125;
eval(S1.d) % 5.4193
eval(S1.h) % 5.4193
eval(S1.lam) %-0.73811
Example: Soda Can Design
• Example: soda can design with one equality constraint
Objective: min 𝑓 = 12𝜋𝑑2 + 𝜋𝑑ℎ,
𝑑,𝑙
Subject to: 14 𝜋𝑑2 ℎ − 𝑉 = 0
4𝑉
• Alternatively, use the equality constraint to solve for ℎ =
𝜋𝑑 2
4𝑉
• Define the unconstrained problem: min 𝑓 = 12𝜋𝑑 2 +
𝑑 𝑑
𝜕𝑓 4𝑉
• FONC: = 𝜋𝑑 − =0
𝜕𝑑 𝑑2

3 4𝑉 6𝑉
• FONC are solved to get: 𝑑∗ = ℎ∗ = ; 𝑓∗ =
𝜋 𝑑∗
𝜕2 𝑓(𝑑 ∗ ,ℎ∗ )
• SONC: = 3𝜋 > 0
𝜕𝑑 2
Example: Soda Can Design
• Example: soda can design with two equality constraint:
Objective: min 𝑓 = 12𝜋𝑑 2 + 𝜋𝑑ℎ,
𝑑,𝑙
Subject to: 1
4
𝜋𝑑 2 ℎ − 𝑉 = 0, 2𝑑 − ℎ = 0
𝜋𝑑 2 𝜋𝑑 2 ℎ
• Define ℒ 𝑑, ℎ, 𝜆 = + 𝜋𝑑ℎ + 𝜆1 −𝑉 + 𝜆2 2𝑑 − ℎ
2 4
𝜕ℒ 𝜋𝑑ℎ
• FONC: = 𝜋𝑑 + 𝜋ℎ + 𝜆1 + 2𝜆2 = 0
𝜕𝑑 2
𝜕ℒ 𝜋𝑑 2
= 𝜋𝑑 + 𝜆1 − 𝜆2 = 0
𝜕ℎ 4
𝜕ℒ 𝜋𝑑 2 ℎ
= −𝑉 =0
𝜕𝜆1 4
𝜕ℒ
= 2𝑑 − ℎ = 0
𝜕𝜆2
3 2𝑉
• The last two equations are solved to obtain: ℎ∗ = 2𝑑 ∗ = 2 𝜋
10 𝜋𝑑 ∗
• The first two equations are then solved to obtain: 𝜆1∗ = − 3𝑑∗ , 𝜆∗2 =
6
Inequality Constrained Problems

Consider the inequality constrained optimization problem:

min 𝑓 𝒙 , 𝒙 ∈ ℝ𝑛 , subject to: 𝑔𝑖 𝒙 ≤ 0, 𝑖 = 1, … , 𝑚
𝒙
Introduce slack variables to convert inequality constraints to equality:
𝑔𝑖 𝒙 + 𝑠𝑖2 = 0, 𝑖 = 1, … , 𝑚
Define the Lagrangian: ℒ 𝒙, 𝜆, 𝑠 = 𝑓 𝒙 + 𝑖 𝜆𝑖 𝑔𝑖 𝒙 + 𝑠𝑖2
𝜕ℒ 𝜕𝑓 𝑚 𝜕𝑔𝑖
FONC: = + 𝑖=1 𝑖 𝜕𝑥 𝑥𝑗
𝜆 = 0; 𝑗 = 1, … , 𝑛
𝜕𝑥𝑗 𝜕𝑥𝑗 𝑗

𝑔𝑖 𝒙 + 𝑠𝑖2 = 0,
𝜆𝑖 𝑠𝑖 = 0, 𝑖 = 1, … , 𝑚
Note,
𝜆𝑖 𝑠𝑖 = 0, 𝑖 = 1, … , 𝑚 define switching conditions, either 𝜆𝑖 = 0 or 𝑠𝑖 = 0.
They require a total of 2𝑚 cases to be explored for feasibility and optimality.
Example: Soda Can Design
• Example: soda can design with one inequality constraint
𝜋𝑑 2 ℎ
min 𝑓 = 𝜋𝑑 2
1
2
+ 𝜋𝑑ℎ, subject to: 𝑉 − ≤0
𝑑,𝑙 4
𝜋𝑑 2 ℎ
• Use slack variable to write: 𝑉 − + 𝑠2 = 0
4
𝜋𝑑 2 𝜋𝑑 2 ℎ
• Define ℒ 𝑑, ℎ, 𝜆 = + 𝜋𝑑ℎ + 𝜆 𝑉 − + 𝑠2
2 4
𝜕ℒ 𝜋𝑑ℎ
• FONC: = 𝜋𝑑 + 𝜋ℎ + 𝜆 =0
𝜕𝑑 2
𝜕ℒ 𝜋𝑑 2
= 𝜋𝑑 + 𝜆 =0
𝜕ℎ 4
𝜕ℒ 𝜋𝑑 2 ℎ
= 𝑉− + 𝑠2 = 0
𝜕𝜆 4
𝜕ℒ
= 2𝜆𝑠 = 0
𝜕𝑠
Then,
3 4𝑉 4
For 𝑠 = 0 we obtain ℎ∗ = 𝑑 ∗ = ; 𝜆∗ = −
𝜋 𝑑∗
No feasible solution for 𝜆 = 0
Optimality Criteria: General Optimization Problem
• Consider the general optimization problem:
min 𝑓 𝒙 , subject to: ℎ𝑖 𝒙 = 0, 𝑖 = 1, … , 𝑙; 𝑔𝑗 𝒙 ≤ 0, 𝑗 = 𝑖, … , 𝑚
𝒙
• Add slack variables and define the Lagrangian function:
𝑙 𝑚
ℒ 𝒙, 𝒖, 𝒗, 𝒔 = 𝑓 𝒙 + 𝑖=1 𝑣𝑖 ℎ𝑖 (𝒙) + 𝑗=1 𝑢𝑗 (𝑔𝑗 (𝒙) + 𝑠𝑗2 )
FONC (KKT):
𝜕ℒ 𝜕𝑓 𝑙 ∗ 𝜕ℎ𝑖 𝑚 ∗ 𝜕𝑔𝑗
Gradient: = + 𝑣
𝑖=1 𝑖 𝜕𝑥 + 𝑢
𝑗=1 𝑗 𝜕𝑥 = 0; 𝑘 = 1, … , 𝑛
𝜕𝑥𝑘 𝜕𝑥𝑘 𝑘 𝑘

Feasibility: 𝑔𝑗 𝒙∗ ≤ 0, 𝑗 = 1, … , 𝑚; ℎ𝑖 𝒙 = 0, 𝑖 = 1, … , 𝑙
Switching: 𝑢𝑗∗ 𝑠𝑗 = 0, 𝑗 = 1, … , 𝑚
Non-negativity: 𝑢𝑗∗ ≥ 0, 𝑗 = 1, … , 𝑚 (𝑣𝑖∗ are not restricted in sign)
Regularity : for those 𝑢𝑗∗ > 0, 𝛻𝑔𝑗 𝒙∗ are linearly independent
Note, we need to solve a total of 2𝑚 cases, each involving 𝑛 + 2𝑚 + 𝑙 variables
Example: Soda Can Design
• Example: soda can design with both equality and inequality constraints
1 2 𝜋𝑑 2 ℎ
min 𝑓 = 𝜋𝑑 + 𝜋𝑑ℎ, subject to: 𝑉 −
2
= 0, 2𝑑 − ℎ ≤ 0
𝑑,𝑙 4
• Use slack variable to write: 2𝑑 − ℎ + 𝑠 2 = 0
𝜋𝑑 2 2 𝜋𝑑 2 ℎ
• Define ℒ 𝑑, ℎ, 𝑢, 𝑣 = + 𝜋𝑑ℎ + 𝑢 2𝑑 − ℎ + 𝑠 +𝑣 𝑉−
2 4
𝜕ℒ 𝜋𝑑ℎ
• FONC (KKT): = 𝜋𝑑 + 𝜋ℎ + 2𝑢 − 𝑣 =0
𝜕𝑑 2
𝜕ℒ 𝜋𝑑 2
= 𝜋𝑑 − 𝑢 − 𝑣 4 =0
𝜕ℎ
𝜕ℒ 2
= 2𝑑 − ℎ + 𝑠 = 0
𝜕𝑢
𝜕ℒ 𝜋𝑑 2 ℎ
𝜕𝑣
=𝑉− 4
=0
𝜕ℒ
= 2𝑢𝑠 = 0
𝜕𝑠
3 2𝑉 𝜋𝑑 ∗ 10 5
• For 𝑠 = 0 we obtain: 𝑑 =∗
, ∗ ∗
ℎ = 2𝑑 ; 𝑢 = ∗
, 𝑣∗ = ∗ ; 𝑓∗ = 𝜋𝑑 ∗ 2
𝜋 6 3𝑑 2
• No feasible solution for 𝑢 = 0
Example: Soda Can Design
• Example: soda can design with equality and inequality constraints
1 2 𝜋𝑑 2 ℎ
min 𝑓 = 2𝜋𝑑 + 𝜋𝑑ℎ, subject to: 𝑉 − = 0, 2𝑑 − ℎ ≤ 0
𝑑,𝑙 4
4𝑉
• Use equality constraint to solve for ℎ = 𝜋𝑑2 (not always easy)
• Redefine the problem
4𝑉 4𝑉
min 𝑓 = 12𝜋𝑑 2 + 𝑑
, subject to: 2𝑑 − 𝜋𝑑2 ≤ 0
𝑑,𝑙
𝜋𝑑 2 4𝑉 4𝑉
• Define ℒ 𝑑, ℎ, 𝑢, 𝑣 = + + 𝑢 2𝑑 − 𝜋𝑑2 + 𝑠 2
2 𝑑
𝜕ℒ 4𝑉 4𝑉
• FONC/KKT = 𝜋𝑑 − + 2𝑢 1 + 𝜋𝑑3 = 0
𝜕𝑑 𝑑2
𝜕ℒ 4𝑉 2
𝜕𝑢
= 2𝑑 − 𝜋𝑑 2 + 𝑠 =0
𝜕ℒ
𝜕𝑠
= 2𝑢𝑠 = 0
3 2𝑉 𝜋𝑑 ∗ 5
• For 𝑠 = 0 we obtain: 𝑑 = ∗
, ∗
ℎ = 2𝑑 ; 𝑢 =∗ ∗
; 𝑓 ∗ = 2 𝜋𝑑 ∗ 2
𝜋 6
• No feasible solution for 𝑢 = 0
Second Order Conditions for Constrained Problems
• Assume that 𝒙∗ satisfies the FONC (KKT) conditions,
• Hessian of the Lagrangian:
𝑙 𝑚
𝛻2ℒ 𝒙 = 𝛻2𝑓 𝒙 + 2
𝑖=1 𝑣𝑖 𝛻 ℎ𝑖 (𝒙) +
2
𝑗=1 𝑢𝑗 𝛻 𝑔𝑗 (𝒙)

• Let the set of active constraints: ℐ = 𝑖 ∪ 𝑗: ℎ𝑖 𝒙 = 0, 𝑔𝑗 𝒙 = 0

• Define active constraint tangent hyperplane by vectors:
𝒅: 𝛻ℎ𝑖 𝒙 𝑇 𝒅 = 0, 𝛻𝑔𝑗 𝒙 𝑇 𝒅 = 0, 𝑗 ∈ ℐ
SONC: If 𝒙∗ is a local minimizer of 𝑓, then 𝒅𝑇 𝛻 2 ℒ 𝒙∗ 𝒅 ≥ 0
SOSC: For every vector 𝒅: 𝛻ℎ𝑖 𝒙∗ 𝑇 𝒅 = 0, 𝛻𝑔𝑗 𝒙∗ 𝑇 𝒅 = 0, 𝑢𝑗∗ > 0 ,
if 𝒙∗ satisfies 𝒅𝑇 𝛻 2 ℒ 𝒙∗ 𝒅 > 0, then 𝒙∗ is a local minimizer of 𝑓(𝒙)
Note, if 𝛻 2 ℒ 𝒙∗ > 0 (uniformly), then 𝒙∗ is a local minimizer of 𝑓(𝒙)
Example: Soda Can Design
• Example: soda can design with both equality and inequality constraints
1 2 𝜋𝑑 2 ℎ
min 𝑓 = 𝜋𝑑 + 𝜋𝑑ℎ, subject to:
2
− 𝑉 = 0, 2𝑑 − ℎ ≤ 0
𝑑,𝑙 4
𝜋𝑑 2 2 𝜋𝑑 2 ℎ
• Lagrangian: ℒ 𝑑, ℎ, 𝑢, 𝑣 = + 𝜋𝑑ℎ + 𝑢 2𝑑 − ℎ + 𝑠 +𝑣 −𝑉
2 4
3 2𝑉 𝜋𝑑 ∗ −10 5
• Optimum: 𝑑 = ∗
, ∗
ℎ = 2𝑑 ; 𝑢 =∗ ∗
, 𝑣∗ = ∗ ; 𝑓∗ = 𝜋𝑑 ∗ 2 ; 𝑠 = 0
𝜋 6 3𝑑 2
𝜕ℒ 𝜋𝑑ℎ 𝜕2 ℒ ℎ𝑣
• SOC: = 𝜋𝑑 + 𝜋ℎ + 2𝑢 + 𝑣 = 0; =𝜋 1+
𝜕𝑑 2 𝜕𝑑 2 2
𝜕ℒ 𝜋𝑑 2 𝜕ℒ 𝑑𝑣 𝜕2 ℒ
= 𝜋𝑑 − 𝑢 + 𝑣 = 0; =𝜋 1+ ; =0
𝜕ℎ 4 𝜕𝑑𝜕ℎ 2 𝜕ℎ2
2 ∗ ∗ 1 + 𝑑∗ 𝑣 ∗ 1 + 𝑑 ∗ 𝑣 ∗ /2 𝜋 −7 −2
𝛻 ℒ 𝑑 ,ℎ =𝜋 = ;
1 + 𝑑 ∗ 𝑣 ∗ /2 0 3 −2 0
• Active constraint gradients: 𝛻𝑔1 = 𝜋𝑑 ∗ 2 1 0.25 ; 𝛻𝑔2 = 2 − 1
• There is no vector in the tangent plane, i.e., 𝑑 ∗ , ℎ∗ is the optimum point
Post Optimality Analysis
• The post optimality analysis addresses the following question: how
does constraint relaxation affect the objective function?
• Consider the perturbed optimization problem: min 𝑓 𝒙 ,
𝒙
Subject to ℎ𝑖 𝒙 = 𝑏𝑖 , 𝑖 = 1, … , 𝑙; 𝑔𝑗 𝒙 ≤ 𝑒𝑗 , 𝑗 = 𝑖, … , 𝑚
• The optimum solution for the perturbed problem is 𝒙∗ 𝒃, 𝒆 , with
𝜕𝑓 𝒙∗ ∗ 𝜕𝑓 𝒙
∗
the optimal cost: 𝑓 (𝒃, 𝒆); then 𝜕𝑏 = −𝑣𝑖 , 𝜕𝑒 = −𝑢𝑗∗ ;
∗
𝑖 𝑗

∗ ∗
𝛿𝑓 𝒙∗ = − 𝑖 𝑣𝑖 𝑏𝑖 − 𝑗 𝑢𝑗 𝑒𝑗
– The non-zero Lagrange multipliers accompanying active constraints
determine cost function sensitivity to constraint relaxation.
– Inactive constraints have zero Lagrange multipliers, and hence, do
not affect the solution.
Example
• Example: soda can design
𝜋𝑑 2 ℎ
min 𝑓 = 1
2
𝜋𝑑 2 + 𝜋𝑑ℎ, subject to: 𝑉 − = 0, 2𝑑 − ℎ ≤ 0
𝑑,𝑙 4
3 2𝑉 5
• The optimum solution is : 𝑑 ∗ = , ℎ∗ = 2𝑑 ∗ ; 𝑓 ∗ = 𝜋𝑑 ∗ 2 with
𝜋 2
𝜋𝑑 ∗ 10
𝑢∗ = , 𝑣∗ =
6 3𝑑∗
• Define the perturbed problem
𝜋𝑑 2 ℎ
min 𝑓 = 1
2
𝜋𝑑 2 + 𝜋𝑑ℎ, subject to: 𝑉 − = 𝑏, 2𝑑 − ℎ ≤ 𝑒
𝑑,𝑙 4
• Variation in the cost function
∗ ∗ 𝜋𝑑 ∗ 10
𝛿𝑓(𝑏, 𝑒) = −𝑢 𝑒 − 𝑣 𝑏 = − 𝑒 − 𝑏
6 3𝑑 ∗
𝜋𝑑 ∗ 1
For example, let 𝑏 = 𝑒 = 0.1; then 𝛿𝑓 𝑏, 𝑒 = −
60
−
3𝑑 ∗
Example
• Example: min 𝑓 𝑥1 , 𝑥2 = −𝑥1 𝑥2
𝑥1 ,𝑥2
Subject to: 𝑔 𝑥1 , 𝑥2 : 𝑥12 + 𝑥22 − 1 ≤ 0; ℎ 𝑥1 , 𝑥2 : 𝑥12 − 𝑥2 = 0
• A local minimum for the problem exists at: 𝑥1∗ , 𝑥2∗ = ±0.786,0.618 ,
𝑢∗ = 0.527, 𝑣 ∗ = −0.134, 𝑓 ∗ = −0.486.
• Define the perturbed optimization problem:
min 𝑓 𝑥1 , 𝑥2 = −𝑥1 𝑥2
𝑥1 ,𝑥2
Subject to: 𝑔 𝑥1 , 𝑥2 : 𝑥12 + 𝑥22 − 1 ≤ 𝑒; ℎ 𝑥1 , 𝑥2 : 𝑥12 − 𝑥2 = 𝑏
• The variation in the optimal solution is given as:
𝛿𝑓 = 𝑓 ∗ 𝑒, 𝑏 − 𝑓 ∗ 0,0 = −𝑢∗ 𝑒 − 𝑣 ∗ 𝑏.
• Then, for 𝑒 = 0.1, the new optimum is: 𝑓 ∗ = −0.54. Similarly, for
𝑏 = 0.1, the new optimum is: 𝑓 ∗ = −0.35.
The Dual Optimization Problem
• Every optimization problem, termed as primal, has an associated dual
problem defined in terms of dual variables (the Lagrange multipliers).
• Let the primal problem be: min 𝑓 𝒙
𝒙
Subject to: ℎ𝑖 𝒙 = 0, 𝑖 = 1, … , 𝑙; 𝑔𝑗 𝒙 ≤ 0, 𝑗 = 𝑖, … , 𝑚
𝑙 𝑚
Lagrangian: ℒ 𝒙, 𝒖, 𝒗 = 𝑓 𝒙 + 𝑖=1 𝑣𝑖 ℎ𝑖 (𝒙) + 𝑗=1 𝑢𝑗 𝑔𝑗 (𝒙)
• Then, the dual problem is defined as:
max ℒ 𝒙, 𝒖, 𝒗 , Subject to 𝛻ℒ 𝒙, 𝒖, 𝒗 = 0 (optimality constraint)
𝒖≥𝟎,𝒗
• Define a dual function: 𝜑 𝒖, 𝒗 = min ℒ 𝒙, 𝒖, 𝒗 = ℒ 𝒙∗ , 𝒖, 𝒗 ; then,
𝒙∈𝒳
the dual problem is described as: max 𝜑 𝒖, 𝒗
𝒖≥𝟎,𝒗
• For non-optimal 𝒙, 𝒖, 𝒗 , a duality gap exists and is given as:
max ℒ 𝒙∗ , 𝒖, 𝒗 ≤ min ℒ 𝒙, 𝒖∗ , 𝒗∗
𝒖≥𝟎,𝒗 𝒙∈Ω
Example: Quadratic Optimization Problem
1
• Minimize 𝑞 𝒙 = 2 𝒙𝑇 𝑸𝒙 + 𝒄𝑇 𝒙 , Subject to: 𝑨𝒙 ≥ 𝒃
1
• Lagrangian: ℒ 𝒙, 𝝀 = 2 𝒙𝑇 𝑸𝒙 + 𝒄𝑇 𝒙 − 𝝀𝑇 𝑨𝒙 − 𝒃
1
• Dual problem: max ℒ 𝒙, 𝝀 = 2 𝒙𝑇 𝑸𝒙 + 𝒄𝑇 𝒙 − 𝝀𝑇 𝑨𝒙 − 𝒃
𝒙,𝝀≥𝟎
Subject to 𝛻𝑞 𝒙 = 𝑸𝒙 + 𝒄 − 𝑨𝑇 𝝀 = 𝟎
• Using the constraint to solve, 𝒙 𝝀 = 𝑸−1 (𝑨𝑇 𝝀 − 𝒄)
1 1
max 𝜑 𝝀 = 𝝀𝑇 (𝑨𝑸−1 𝑨𝑇 )𝝀 + 𝑨𝑸−1 𝒄 + 𝒃 𝑇 𝝀 − 𝒄𝑇 𝑸−1 𝒄
𝝀≥𝟎 2 2
• Dual optimal solution:
𝝀 = 𝑨𝑸−1 𝑨𝑇 −1 𝑨𝑸−1 𝒄 + 𝒃
𝒙 = 𝑸−1 𝑨𝑇 𝑨𝑸−1 𝑨𝑇 −1 𝑨𝑇 𝑸−1 𝒄 + 𝒃 −𝑸−1 𝒄

View publication stats