0% found this document useful (0 votes)
26 views

Week 5

The document discusses the Karush-Kuhn-Tucker (KKT) optimality conditions for problems with inequality constraints. It explains that the KKT conditions provide necessary conditions for a solution to be a local minimum. It also discusses using Lagrange multipliers and defines the Lagrangian function and its properties at a solution. The document notes some key points about applying and interpreting the KKT conditions.

Uploaded by

Mohmmad Break
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
26 views

Week 5

The document discusses the Karush-Kuhn-Tucker (KKT) optimality conditions for problems with inequality constraints. It explains that the KKT conditions provide necessary conditions for a solution to be a local minimum. It also discusses using Lagrange multipliers and defines the Lagrangian function and its properties at a solution. The document notes some key points about applying and interpreting the KKT conditions.

Uploaded by

Mohmmad Break
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 14

4.

6 General Constrained Problems


Problems with inequality constraints:
𝑔𝑖 𝒙 ≤ 0; 𝑖 = 1 𝑡𝑜 𝑚

Solution via Lagrange multipliers requires active constraints:


ℎ𝑖 𝒙 = 0

If we don’t know before solving a problem which constraints are


active at the optimum point, how do we solve?

Answer: Transform an inequality constraint into an equality


constraint by adding a slack variable (s).

1
Karush-Kuhn-Tucker (KKT) Optimality Conditions
Let x* be a regular point of the feasible set that is a local
minimum for f(x), subject to
ℎ𝑖 𝒙 = 0; 𝑖 = 1 𝑡𝑜 𝑝
𝑔𝑗 𝒙 ≤ 0; 𝑗 = 1 𝑡𝑜 𝑚
Then there exist Lagrange multipliers v* (p-vector) and u* (m-
vector) such that the Lagrangian function is stationary with
respect to each element of x, v, u, and s at the point x*.

1. Lagrangian Function in the Standard Form


𝑝 𝑚

𝐿 𝒙, 𝒗, 𝒖, 𝒔 = 𝑓 𝒙 + ෍ 𝑣𝑖 ℎ𝑖 𝒙 + ෍ 𝑢𝑗 𝑔𝑗 𝒙 + 𝑠𝑗2
𝑖=1 𝑗=1
= 𝑓 𝒙 + 𝑣 ℎ 𝒙 + 𝑢𝑇 𝑔 𝒙 + 𝑠 2
𝑇

(continued…) 2
Karush-Kuhn-Tucker (KKT) Optimality Conditions
2. Gradient Conditions
𝑝 𝑚
𝜕𝐿 𝜕𝑓 ∗
𝜕ℎ𝑖 ∗
𝜕𝑔𝑖
= + ෍ 𝑣𝑖 + ෍ 𝑢𝑗 = 0; 𝑘 = 1 𝑡𝑜 𝑛
𝜕𝑥𝑘 𝜕𝑥𝑘 𝜕𝑥𝑘 𝜕𝑥𝑘
𝑖=1 𝑗=1
𝜕𝐿
=0 → ℎ𝑖 𝒙∗ = 0; 𝑖 = 1 𝑡𝑜 𝑝
𝜕𝑣𝑖
𝜕𝐿
=0 → 𝑔𝑗 𝒙∗ + 𝑠𝑗2 = 0; 𝑗 = 1 𝑡𝑜 𝑚
𝜕𝑢𝑗

3. Feasibility of Inequalities
𝑠𝑗2 ≥ 0 ↔ 𝑔𝑗 ≤ 0; 𝑗 = 1 𝑡𝑜 𝑚
4. Switching Conditions
𝜕𝐿
=0 ↔ 2𝑢𝑗 ∗ 𝑠𝑗 = 0; 𝑗 = 1 𝑡𝑜 𝑚
𝜕𝑠𝑗

(continued…) 3
Karush-Kuhn-Tucker (KKT) Optimality Conditions
5. Non-Negativity of Lagrange Multipliers for Inequalities
𝑢𝑗 ≥ 0; 𝑗 = 1 𝑡𝑜 𝑚
6. Regularity
Gradients of all active constraints must be linearly independent and the
Lagrange multipliers of all constraints must be unique.

4
Items to Note about KKT Conditions
• KKT conditions are only applicable at regular points
– Recall: a point x* is a regular point of the feasible set S if f(x*) is
differentiable and gradient vectors of all active constraints at x* are
linearly independent (no gradient can be expressed as a linear
combination of the others)
• Points that satisfy the KKT conditions are called KKT points
• Any non-KKT point cannot be a local minimum point (unless it is an
irregular point and the KKT conditions are not applicable)
• KKT points can be unconstrained or constrained
– Unconstrained if there are no equality constraints and all inequality
constraints are inactive
– Constrained otherwise
• If a KKT point is unconstrained, the 2nd order necessary/sufficient
conditions can determine whether it is a minimum, maximum, or
inflection point

(continued…) 5
Items to Note about KKT Conditions
At a constrained KKT point
• If there are only equality constraints and no active
inequality constraints (u = 0), then the KKT point is
stationary.
– It can be a minimum, maximum, or inflection point.
• If there are some active inequality constraints and
their Lagrange multipliers are strictly positive, then
the KKT point cannot be a local maximum.
– It can be a minimum or inflection point.

(continued…) 6
Items to Note about KKT Conditions
• The Lagrange multiplier for each constraint depends on the form of the
constraint

For example, the constraints


2𝑥1 + 3𝑥2 ≤ 0
2 𝑥1
+1≤0
3 𝑥2
0.2𝑥1 + 0.3𝑥2 ≤ 0
will have different Lagrange multipliers, but the optimum solution (𝑥1∗ , 𝑥2∗ ) will
be the same.

• The switching conditions have two cases (si = 0 and ui = 0), each of which
can have multiple solutions.
– Each inequality in the problem formulation doubles the number of KKT
solution cases (1 inequality = 2 KKT cases, 2 inequalities = 4 KKT cases, 3
inequalities = 8 cases, etc.)

7
Minimize
𝑓 𝒙 = 𝑥1 2 + 𝑥2 2 − 3𝑥1 𝑥2
Subject to
𝑔 = 𝑥1 2 + 𝑥2 2 − 6 ≤ 0
1. Write equations for the
KKT conditions
2. Solve for candidate
points for each case of
switching conditions

Figure 4.17 Graphical solution for Example 4.31. Local minimum points, A and B (the vectors are not to scale).

8
Introduction to Optimum Design 3e. Copyright © 2012 Elsevier, Inc. All rights reserved.
5.3: Second Order Necessary Conditions
for General Constrained Problems
Recall: Unconstrained Problems

1st order NC:


𝜕𝑓 𝒙∗
= 0 for 𝑖 = 1 𝑡𝑜 𝑛
𝜕𝑥𝑖

2nd order NC:


𝐻 𝒙∗ positive semidefinite (NC)
or positive definite (SC)

Now: Constrained Problems


Consider feasible changes (d) about active constraints at 𝒙∗

9
Constraint Tangent Hyperplane

Any feasible moves away from 𝒙∗ must be along 𝒅


and 𝛻𝑔𝑖 𝒙∗ ∙ 𝒅 = 0

So the feasible region around 𝒙∗ is defined by


𝛻ℎ𝑖 𝑇 𝒅 = 0
𝛻𝑔𝑖 𝑇 𝒅 = 0 for all active constraints at 𝒙∗

10
2nd Order NC for General Constrained Problems
Let x* be a candidate point satisfying the KKT conditions.
Define the Hessian of the Lagrange Function
𝑝 𝑚
∗ ∗
𝛻 2𝐿 = 𝛻2𝑓 + ෍ 𝑣𝑖 𝛻 2ℎ 𝑖 + ෍ 𝑢𝑖 𝛻 2𝑔
𝑖
𝑗=1 𝑗=1
Let there be nonzero feasible directions 𝒅 ≠ 0 satisfying
𝛻ℎ𝑖 𝑇 𝒙∗ 𝒅 = 0 for 𝑖 = 1 𝑡𝑜 p
𝛻𝑔𝑗 𝑇 𝒙∗ 𝒅 = 0 for all active inequalities (where 𝑔𝑗 𝑇 (𝒙∗ ) = 0)
Then for 𝒙∗ to be a local minimum, it must be true that
𝒅𝑇 𝛻 2 𝐿 𝒙 ∗ 𝒅 ≥ 0

𝑄≥0
Note: This is just the Necessary Condition. Does not guarantee
optimality.

11
Sufficient Conditions for General Constrained Problems
Let x* be a candidate point satisfying the KKT conditions.
Define the Hessian of the Lagrange Function
𝑝 𝑚
∗ ∗
𝛻2𝐿 = 𝛻2𝑓 + ෍ 𝑣𝑖 𝛻2ℎ 𝑖 + ෍ 𝑢𝑖 𝛻 2 𝑔𝑖
𝑗=1 𝑗=1
Let there be nonzero feasible directions 𝒅 ≠ 0 satisfying
𝛻ℎ𝑖 𝑇 𝒙∗ 𝒅 = 0 for 𝑖 = 1 𝑡𝑜 p
𝛻𝑔𝑗 𝑇 𝒙∗ 𝒅 = 0 for all active inequalities with 𝑢𝑖 ∗ > 0
and let
𝛻𝑔𝑗 𝑇 𝒙∗ 𝒅 ≤ 0 for all active inequalities with 𝑢𝑖 ∗ = 0

If Q > 0 then 𝒙∗ is a strict (isolated) local minimum 12


Notes on NCs and SCs for General Constrained Problems

• If 𝛻 2 𝐿 𝒙∗ is negative definite or negative


semidefinite, the 2nd order NC is violated
– 𝒙∗ cannot be a local minimum
• If 𝛻 2 𝐿 𝒙∗ is positive definite for any 𝒅 ≠ 0,
the SC is satisfied
– No further checks are needed
• If 𝛻 2 𝐿 𝒙∗ is positive semidefinite or
indefinite, we must calculate 𝒅 and check the
SCs

13
Minimize
𝑓 𝒙 = 𝑥1 2 + 𝑥2 2 − 3𝑥1 𝑥2
Subject to
𝑔 = 𝑥1 2 + 𝑥2 2 − 6 ≤ 0

Candidate Points
Point 𝑥1 ∗ 𝑥2 ∗ 𝑢∗

O 0 0 0
1
A 3 3
2 Figure 4.17 Graphical solution for Example 4.31. Local minimum
points, A and B (the vectors are not to scale).
1
B − 3 − 3
2

14
Introduction to Optimum Design 3e. Copyright © 2012 Elsevier, Inc. All rights reserved.

You might also like