Engineering Optimization

Concepts and Applications

WB 1440

Fred van Keulen

Matthijs Langelaar
CLA H21.1
[email protected]

WB1440 Engineering Optimization – Concepts and Applications

 Geometrical interpretation
● For single equality constraint: simple geometrical
interpretation of Lagrange optimality condition:

f h x2
f
  0T
x x h
Meaning: h
f h
//
x x f
x1
Gradients parallel 
tangents parallel  h tangent to isolines
WB1440 Engineering Optimization – Concepts and Applications
 Summary
x2
● First order optimality f
condition for equality h
constrained problem: h
– Zero reduced gradient:
1 f
f f h h x1
  0T
d s s d
– Equivalent: stationary
Lagrangian:  L T
 x  0
L  f ( x)  λ T h ( x)   L
  0T
 λ
WB1440 Engineering Optimization – Concepts and Applications
 Contents
● Constrained Optimization: Optimality Criteria

– Reduced gradient

– Lagrangian

– Sufficiency conditions

– Inequality constraints

– Karush-Kuhn-Tucker (KKT) conditions

– Interpretation of Lagrange multipliers

● Constrained Optimization: Algorithms

WB1440 Engineering Optimization – Concepts and Applications

 Sufficiency?
● Until now, only stationary points considered. Does not
guarantee minimum!
f h
● Lagrange
h
f h f
condition: f
maximum minimum
f  T h  0 h
h
h f f h
f
minimum no extremum
h
f
WB1440 Engineering Optimization – Concepts and Applications
 Constrained Hessian
● Sufficiency conditions follow from 2nd order Taylor
approximation
● Second order information required: constrained Hessian:
T T
 2  2 f   2 f  s   s    2 f   s   2 f  s   f   2s 
2
 2              2      2 
d d  ds  d   d   sd   d   s  d   s  d 
with
 h   h   h  s   s    h   s    h  s 

1 T T T
 s
2 2 2 2 2
 2      2              2  
 d   s   d  ds  d   d   sd   d   s  d 

obtained by differentiation of the constrained gradient, and

T
second-order constraint perturbation:   h 
  0
d  d 
WB1440 Engineering Optimization – Concepts and Applications
 Sufficiency conditions
● Via 2nd order Taylor approximation, it follows that at a
minimum the following must hold:

  2 (Constrained Hessian
 0T and d  T
d  0 positive definite)
d d 2

● Lagrangian approach also yields:

 2  2
L 2L 2 f  2
h
d  T
2
d  x 
T
2
x with
x 2
 2 λ
x
T

x 2
d x
Perturbations only in
tangent subspace of h!
WB1440 Engineering Optimization – Concepts and Applications
 Summary
● Optimality conditions for equality constrained problem:

1. Necessary condition: stationary point when:

L  0

2. Sufficient condition: minimum when (1) and:

2
 L
x  2 x  0
T

x
on tangent subspace.

WB1440 Engineering Optimization – Concepts and Applications

 Example x2
f
2 2
min f (x)  x1  x2 h
x1 , x2
2 2
s.t. h(x)  x1  2 x2  1  0 x1

2 2
 L(x)  x1  x2   x1  2 x2  1  2 2

T
dL  2 x1 (1   )  dL 2 2
   ,  x1  2 x 2 1
dx 2 x2 (1  2 ) d
1. Necessary condition: stationary point when L  0
x1  0, x2   1 / 2 ,   1 / 2
x2  0, x1  1,   1

WB1440 Engineering Optimization – Concepts and Applications

 Contents
● Constrained Optimization: Optimality Criteria

– Reduced gradient

– Lagrangian

– Sufficiency conditions

– Inequality constraints

– Karush-Kuhn-Tucker (KKT) conditions

– Interpretation of Lagrange multipliers

● Constrained Optimization: Algorithms

WB1440 Engineering Optimization – Concepts and Applications

 Inequality constrained problems
● Consider problem with only inequality constraints:

min f ( x) x  n
x

s. t. g i ( x)  0 i  1 m
● At optimum, only active constraints matter:

min f ( x) x  n
x

s. t. g j ( x)  0 j  1 k  m

● Optimality conditions similar to equality constrained

problem
WB1440 Engineering Optimization – Concepts and Applications
 Inequality constraints
● First order optimality: g0
1
T f  g 
μ   
L ( x)  f ( x)  μ T g ( x)  x  x 
f T g
μ  0T
x x
● Consider feasible local variation around optimum:

f T g f T g
μ  0T  x  μ x  0
x x x x
0 0
(boundary optimum) (feasible perturbation)
WB1440 Engineering Optimization – Concepts and Applications
 Optimality condition
● Multipliers must be non-negative:

f T g g2
x  μ x  0 x2
x x f g1

0 0
0
k m
 f    j g j  0
j 1
g 2 -f
g1 x1
 f    j g j
● Interpretation: negative gradient (descent direction) -f
lies in cone spanned by positive constraint gradients
WB1440 Engineering Optimization – Concepts and Applications
 Optimality condition (2)
f     g  j
g
j
2

x2 Feasible cone
● Feasible direction:
f  g g1
g  s  0 j  1 k  m
j
T  g1 2

  g  s  0
T
j

● Descent direction: g 2 -f

g1 x1
f  s  0
T

  f  s  0
T

● Equivalent interpretation: no descent direction exists

within the cone of feasible directions
WB1440 Engineering Optimization – Concepts and Applications
 Examples
f    j g j  0  f    j g j

g 2 g 2
g 2
f -f -f
g1
g1 g1

-f 1  0 f 1  0 1  0
f
2  0 2  0 2  0

WB1440 Engineering Optimization – Concepts and Applications

 Optimality condition (3)
L ( x )  f ( x )  μ T g ( x)

● Active constraints: g j  0, j  0
Inactive constraints: gi  0
● Formulation including all inequality constraints:

L ( x)  f ( x)  μ T g ( x)

L f T g ii ggii  00 i  1 m
Complementarity
 μ  0T and
x x x μi 0 0
condition
i  1 m

WB1440 Engineering Optimization – Concepts and Applications

 Example
x11
min f  m g x22, gh  L(1  cos x1 )  x2 
0
xx11,,xx22 L
L m
L( x1 , x2 )  f ( x1 , x2 )  g ( x1 , x2 ) x2

L( x1 , x2 )  m g x2   L(1  cos x1 )  x2  m x2

T
L  L sin x1  T
   0
x  m g   
   mg
m gL sin x1  0  x1  0
x2  0
WB1440 Engineering Optimization – Concepts and Applications
 Mechanical application: contact
● Lagrange multipliers also used in:

– Contact in multibody dynamics

– Contact in finite elements

WB1440 Engineering Optimization – Concepts and Applications

 Contents
● Constrained Optimization: Optimality Criteria

– Reduced gradient

– Lagrangian

– Sufficiency conditions

– Inequality constraints

– Karush-Kuhn-Tucker (KKT) conditions

– Interpretation of Lagrange multipliers

● Constrained Optimization: Algorithms

WB1440 Engineering Optimization – Concepts and Applications

 Karush-Kuhn-Tucker conditions
● Combining Lagrange conditions for equality and
inequality constraints yields KKT conditions for general
problem: min f ( x)
x
Lagrangian:
s. t. g ( x)  0
L  f ( x)  μ T g ( x)  λ T h ( x)
h ( x)  0
L f g i hi
    i   i 0 (optimality)
x x x x
and g  0, h  0 (feasibility)
λ  0, μ  0, i gi  0 (complementarity)

WB1440 Engineering Optimization – Concepts and Applications

 Sufficiency
● KKT conditions are necessary conditions for local
constrained minima
● For sufficiency, consider the sufficiency conditions
based on the active constraints:
2L
x  T
2
x  0
x
on tangent subspace of h and active g.

● Interpretation: objective and feasible domain

locally convex
WB1440 Engineering Optimization – Concepts and Applications
 Additional remarks
● Global optimality:

– Globally convex objective function?

– And convex feasible domain?

Then KKT point gives global optimum

● Pitfall:

– Sign conventions for Lagrange multipliers in

KKT condition depend on standard form!
– Presented theory valid for negative null form

WB1440 Engineering Optimization – Concepts and Applications

 Contents
● Constrained Optimization: Optimality Criteria

– Reduced gradient

– Lagrangian

– Sufficiency conditions

– Inequality constraints

– Karush-Kuhn-Tucker (KKT) conditions

– Interpretation of Lagrange multipliers

● Constrained Optimization: Algorithms

WB1440 Engineering Optimization – Concepts and Applications

 Significance of multipliers
● Consider case where optimization problem depends on
parameter a: min f (x; a )
x

s. t. h(x; a )  0

Lagrangian: L(x, λ; a )  f ( x; a)   T h( x; a)
T T
L f T h f h
KKT:   0   
x x x x x
T
df f f dx
Looking for:  
da a x da
WB1440 Engineering Optimization – Concepts and Applications
 Significance of multipliers (2)
T T T
df f f dx f h
Looking for:   KKT:  
da a x da x x

T
df f h dx
  
da a x da
T
h dx h h h dx
h ( x, a )  0   0  
x da a a x da

df f h
 
da a a
WB1440 Engineering Optimization – Concepts and Applications
 Significance of multipliers (3)
● Lagrange multipliers describe the sensitivity of the
objective to changes in the constraints:
df f h
 
da a a
● Similar equations can be derived for multiple
constraints and inequalities

● Multipliers give “price of raising the constraint”

● Note, this makes it logical that at an optimum,

multipliers of inequality constraints must be positive!
WB1440 Engineering Optimization – Concepts and Applications
 Example
A, y
N Minimize mass (volume):
f ( A)  V  Al
l Stress constraint:
N
N  g ( A; )    y  0
L  Al      y  A
A 
A2 l Nl V
 N   
l   A2  0  y   y
2
N

 N    0 N Nl
 A y
 A , V
 y  y

WB1440 Engineering Optimization – Concepts and Applications

 Example (2)
Stress constraint:
N Nl V
g ( A; )    y  0 V 
A  y  y
Constraint sensitivity:
f V V g V
    y   
  g  
    y
Check:
Nl Nl 1 V Nl 1 Nl 1 V
V    2
 
 y  y   y   y  

WB1440 Engineering Optimization – Concepts and Applications