MECE6388 Optimal Control Theory, 2024 Fall

Dr. Zheng Chen

Chapter 2: Optimal Control of Discrete Time System

2.1 Solution of the General Discrete-Time Optimization problem

Let the plant be

𝑥𝑥𝑘𝑘+1 = 𝑓𝑓 𝑘𝑘 (𝑥𝑥𝑘𝑘 , 𝑢𝑢𝑘𝑘 )
with initial condition 𝑥𝑥0 . The superscript on the function 𝑓𝑓 indicates that,
in general, the system and thus its model can have time-varying
dynamics. Let the state 𝑥𝑥𝑘𝑘 , be a vector of size 𝑛𝑛 and the control input 𝑢𝑢𝑘𝑘
be a vector of size 𝑚𝑚.

Let an associated scalar performance index

𝑁𝑁−1

𝐽𝐽𝑖𝑖 = 𝜙𝜙(𝑁𝑁, 𝑥𝑥𝑁𝑁 ) + � 𝐿𝐿𝑘𝑘 (𝑥𝑥𝑘𝑘 , 𝑢𝑢𝑘𝑘 )

𝑘𝑘=𝑖𝑖
where [𝑖𝑖, 𝑁𝑁] is the time interval, on a discrete time scale with a fixed
sample step, 𝜙𝜙(𝑁𝑁, 𝑥𝑥𝑁𝑁 ) is a function of the final time 𝑁𝑁 and the state at
the final time, and 𝐿𝐿𝑘𝑘 (𝑥𝑥𝑘𝑘 , 𝑢𝑢𝑘𝑘 ) is a general time-varying function of the
state and control input.

The optimal control problem is to find the control 𝑢𝑢∗ on the interval
[𝑖𝑖, 𝑁𝑁] that drives the system along a trajectory 𝑥𝑥𝑘𝑘∗ such that the value of
the performance index is minimized (optimized).

Example 2:1-1
A. Minimum-time Problems
Suppose we want to find the control 𝑢𝑢𝑘𝑘 to drive the system from the
given initial state 𝑥𝑥0 to a desired final state 𝑥𝑥 ∈ ℝn in minimum time,
then we could select the performance index
𝑁𝑁−1

𝐽𝐽 = 𝑁𝑁 = � 1
𝑘𝑘=0
and specify the boundary condition
MECE6388 Optimal Control Theory, 2024 Fall
Dr. Zheng Chen
𝑥𝑥𝑁𝑁 = 𝑥𝑥

In this case, one can consider either 𝜙𝜙 = 𝑁𝑁 and 𝐿𝐿 = 0. Or equivalently

𝜙𝜙 = 0 and 𝐿𝐿 = 1.

B. Minimum- fuel Problems

To find the scalar control 𝑢𝑢𝑘𝑘 to drive the system from 𝑥𝑥0 to a desired
final state 𝑥𝑥 at a fixed time 𝑁𝑁 using minimum fuel, we could use
𝑁𝑁−1

𝐽𝐽 = � |𝑢𝑢𝑘𝑘 |
𝑘𝑘=0
Since the fuel burned is proportional to the magnitude of final state and
all intermediate state, we define a new cost
𝑁𝑁−1
1 1
𝐽𝐽 = 𝑠𝑠𝑥𝑥𝑁𝑁𝑇𝑇 𝑥𝑥𝑁𝑁 + �(𝑞𝑞𝑥𝑥𝑘𝑘𝑇𝑇 𝑥𝑥𝑘𝑘 + 𝑟𝑟𝑢𝑢𝑘𝑘𝑇𝑇 𝑢𝑢𝑘𝑘 )
2 2
𝑘𝑘=0
where 𝑞𝑞, 𝑟𝑟, and 𝑠𝑠 are scalar weighting factors.
For more generality, we could select weighting matrices 𝑄𝑄, 𝑅𝑅, and
𝑆𝑆 instead of scalars.
𝑁𝑁−1
1 1
𝐽𝐽 = 𝑥𝑥𝑁𝑁𝑇𝑇 𝑆𝑆𝑥𝑥𝑁𝑁 + � (𝑥𝑥𝑘𝑘𝑇𝑇 𝑄𝑄𝑥𝑥𝑘𝑘 + 𝑢𝑢𝑘𝑘𝑇𝑇 𝑅𝑅𝑢𝑢𝑘𝑘 )
2 2
𝑘𝑘=0

Problem Solution:
Let 𝜆𝜆 ∈ ℝ𝑛𝑛 and append the constraints to the performance index
𝑁𝑁−1

𝐽𝐽′ = 𝜙𝜙(𝑁𝑁, 𝑥𝑥𝑁𝑁 ) + �[𝐿𝐿𝑘𝑘 (𝑥𝑥𝑘𝑘 , 𝑢𝑢𝑘𝑘 ) + 𝜆𝜆𝑇𝑇𝑘𝑘+1 (𝑓𝑓 𝑘𝑘 (𝑥𝑥𝑘𝑘 , 𝑢𝑢𝑘𝑘 ) − 𝑥𝑥𝑘𝑘+1 )]
𝑘𝑘=𝑖𝑖
Note that we have associated with 𝑓𝑓 𝑘𝑘 the multiplier 𝜆𝜆𝑘𝑘+1 not 𝜆𝜆.

Define the Hamiltonian function as

𝐻𝐻𝑘𝑘 (𝑥𝑥𝑘𝑘 , 𝑢𝑢𝑘𝑘 ) = 𝐿𝐿𝑘𝑘 (𝑥𝑥𝑘𝑘 , 𝑢𝑢𝑘𝑘 ) + 𝜆𝜆𝑇𝑇𝑘𝑘+1 𝑓𝑓 𝑘𝑘 (𝑥𝑥𝑘𝑘 , 𝑢𝑢𝑘𝑘 )
We can write
MECE6388 Optimal Control Theory, 2024 Fall
Dr. Zheng Chen
𝑁𝑁−1

𝐽𝐽′ = 𝜙𝜙(𝑁𝑁, 𝑥𝑥𝑁𝑁 ) − 𝜆𝜆𝑇𝑇𝑁𝑁 𝑥𝑥𝑁𝑁 + 𝐻𝐻𝑖𝑖 (𝑥𝑥𝑖𝑖 , 𝑢𝑢𝑖𝑖 ) + � [𝐻𝐻𝑘𝑘 (𝑥𝑥𝑘𝑘 , 𝑢𝑢𝑘𝑘 , ) − 𝜆𝜆𝑇𝑇𝑘𝑘 𝑥𝑥𝑘𝑘 ]
𝑘𝑘=𝑖𝑖+1
where some minor manipulations with indies have been performed.

Now, we want to examine the increment 𝐽𝐽′ due to increment in all the
variables 𝑥𝑥𝑘𝑘 , 𝜆𝜆𝑘𝑘 and 𝑢𝑢𝑘𝑘 .
𝑇𝑇 𝑇𝑇 𝑇𝑇
𝑑𝑑𝐽𝐽′ = �𝜙𝜙𝑥𝑥𝑁𝑁 − 𝜆𝜆𝑁𝑁 � 𝑑𝑑𝑥𝑥𝑁𝑁 + �𝐻𝐻𝑥𝑥𝑖𝑖 𝑖𝑖 � 𝑑𝑑𝑥𝑥𝑖𝑖 + �𝐻𝐻𝑢𝑢𝑖𝑖 𝑖𝑖 � 𝑑𝑑𝑢𝑢𝑖𝑖
𝑁𝑁−1
𝑇𝑇 𝑇𝑇
+ � ��𝐻𝐻𝑥𝑥𝑘𝑘𝑘𝑘 − 𝜆𝜆𝑘𝑘 � 𝑑𝑑𝑥𝑥𝑘𝑘 + �𝐻𝐻𝑢𝑢𝑘𝑘𝑘𝑘 � 𝑑𝑑𝑢𝑢𝑘𝑘 �
𝑘𝑘=𝑖𝑖+1
𝑁𝑁
𝑇𝑇
+ � �𝐻𝐻𝜆𝜆𝑘𝑘−1
𝑘𝑘
− 𝑥𝑥𝑘𝑘 � 𝑑𝑑𝜆𝜆𝑘𝑘
𝑘𝑘=𝑖𝑖+1
where
𝜕𝜕𝐻𝐻𝑘𝑘
𝐻𝐻𝑥𝑥𝑘𝑘𝑘𝑘
≜
𝜕𝜕𝑥𝑥𝑘𝑘
Necessary conditions for a constrained minimum are thus given by
𝜕𝜕𝐻𝐻𝑘𝑘
𝑥𝑥𝑘𝑘+1 = , 𝑘𝑘 = 𝑖𝑖, … , 𝑁𝑁 − 1 , (1)
𝜕𝜕𝜆𝜆𝑘𝑘+1
𝜕𝜕𝐻𝐻𝑘𝑘
𝜆𝜆𝑘𝑘 = , 𝑘𝑘 = 𝑖𝑖 + 1, … , 𝑁𝑁 − 1 , (2)
𝜕𝜕𝑥𝑥𝑘𝑘
𝜕𝜕𝐻𝐻 𝑘𝑘
0= , 𝑘𝑘 = 𝑖𝑖, … , 𝑁𝑁 − 1 , (3)
𝜕𝜕𝑢𝑢𝑘𝑘
which arise from the terms inside the summations and the coefficient of
𝑑𝑑𝑢𝑢𝑖𝑖 and
𝑇𝑇
𝜕𝜕𝜕𝜕
� − 𝜆𝜆𝑁𝑁 � 𝑑𝑑𝑥𝑥𝑁𝑁 = 0, (4)
𝜕𝜕𝑥𝑥𝑁𝑁
𝑇𝑇
𝜕𝜕𝐻𝐻𝑖𝑖
� � 𝑑𝑑𝑥𝑥𝑖𝑖 = 0, (5)
𝜕𝜕𝑥𝑥𝑖𝑖
The stationary condition allows the optimal control 𝑢𝑢𝑘𝑘 to be expressed
in terms of the costate 𝜆𝜆𝑘𝑘 .
MECE6388 Optimal Control Theory, 2024 Fall
Dr. Zheng Chen

If the initial state 𝑥𝑥𝑖𝑖 is fixed, then 𝑑𝑑𝑥𝑥𝑖𝑖 = 0 . Then (5) holds
regardless of the value of 𝐻𝐻𝑥𝑥𝑖𝑖 𝑖𝑖 .

In the case of free initial state, 𝑑𝑑𝑥𝑥𝑖𝑖 is not zero, then from (5)
𝜕𝜕𝐻𝐻𝑖𝑖
=0
𝜕𝜕𝑥𝑥𝑖𝑖

In our applications, the system starts at a known initial state 𝑥𝑥𝑖𝑖 .

Thus, the case (5) holds and there are no constraints on the value of 𝐻𝐻𝑥𝑥𝑖𝑖 𝑖𝑖 .
In the case of a fixed final state, we use the desired value of 𝑥𝑥𝑁𝑁 as
the terminal condition. Since 𝑥𝑥𝑁𝑁 is not free to be varied, 𝑑𝑑𝑥𝑥𝑁𝑁 = 0 and
(4) can always be satisfied.
In the case of free-final-state situation, 𝑑𝑑𝑥𝑥𝑁𝑁 is not zero. Then (4)
gives us:
𝜕𝜕𝜕𝜕 𝜕𝜕𝜕𝜕
− 𝜆𝜆𝑁𝑁 = 0 ⇒ 𝜆𝜆𝑁𝑁 =
𝜕𝜕𝑥𝑥𝑁𝑁 𝜕𝜕𝑥𝑥𝑁𝑁
In a summary, the discrete Nonlinear Optimal Controller can be
found based on the following equations:

System Model:
𝑥𝑥𝑘𝑘+1 = 𝑓𝑓 𝑘𝑘 (𝑥𝑥𝑘𝑘 , 𝑢𝑢𝑘𝑘 ), 𝑘𝑘 > 𝑖𝑖
Performance Index:
𝑁𝑁−1

𝐽𝐽𝑖𝑖 = 𝜙𝜙(𝑁𝑁, 𝑥𝑥𝑁𝑁 ) + � 𝐿𝐿𝑘𝑘 (𝑥𝑥𝑘𝑘 , 𝑢𝑢𝑘𝑘 )

𝑘𝑘=𝑖𝑖
Hamiltonian:
𝐻𝐻𝑘𝑘 = 𝐿𝐿𝑘𝑘 + 𝜆𝜆𝑇𝑇𝑘𝑘+1 𝑓𝑓 𝑘𝑘 (𝑥𝑥𝑘𝑘 , 𝑢𝑢𝑘𝑘 )
Optimal controller
State equation:
𝜕𝜕𝐻𝐻𝑘𝑘
𝑥𝑥𝑘𝑘+1 = = 𝑓𝑓 𝑘𝑘 (𝑥𝑥𝑘𝑘 , 𝑢𝑢𝑘𝑘 )
𝜕𝜕𝜆𝜆𝑘𝑘+1
MECE6388 Optimal Control Theory, 2024 Fall
Dr. Zheng Chen
Costate equation:
𝑇𝑇
𝜕𝜕𝐻𝐻𝑘𝑘 𝜕𝜕𝑓𝑓 𝑘𝑘 𝜕𝜕𝐿𝐿𝑘𝑘
𝜆𝜆𝑘𝑘 = =� � 𝜆𝜆𝑘𝑘+1 +
𝜕𝜕𝑥𝑥𝑘𝑘 𝜕𝜕𝑥𝑥𝑘𝑘 𝜕𝜕𝑥𝑥𝑘𝑘
Stationary condition:
𝑇𝑇
𝜕𝜕𝐻𝐻 𝑘𝑘 𝜕𝜕𝑓𝑓 𝑘𝑘 𝜕𝜕𝐿𝐿𝑘𝑘
0= =� � 𝜆𝜆𝑘𝑘+1 +
𝜕𝜕𝑢𝑢𝑘𝑘 𝜕𝜕𝑢𝑢𝑘𝑘 𝜕𝜕𝑢𝑢𝑘𝑘
Boundary conditions:
𝑇𝑇
𝑖𝑖 𝑖𝑖 𝑇𝑇
𝜕𝜕𝐿𝐿 𝜕𝜕𝑓𝑓
� +� � 𝜆𝜆𝑖𝑖+1 � 𝑑𝑑𝑥𝑥𝑖𝑖 = 0
𝜕𝜕𝑥𝑥𝑖𝑖 𝜕𝜕𝑥𝑥𝑖𝑖
𝑇𝑇
𝜕𝜕𝜕𝜕
� − 𝜆𝜆𝑁𝑁 � 𝑑𝑑𝑥𝑥𝑁𝑁 = 0
𝜕𝜕𝑥𝑥𝑁𝑁

Example
Consider a simple linear dynamical system:
𝑥𝑥𝑘𝑘+1 = 𝑎𝑎𝑥𝑥𝑘𝑘 + 𝑏𝑏𝑢𝑢𝑘𝑘
where 𝑎𝑎, 𝑏𝑏 are scalar constant values.

Given an initial condition 𝑥𝑥0 and the interval [0, 𝑁𝑁], we need to find a
control to minimize the control energy
𝑁𝑁−1
𝑟𝑟
𝐽𝐽0 = � 𝑢𝑢𝑘𝑘2
2
𝑘𝑘=0
for some scalar weighting factor r.

Case A: Fixed final state

𝑥𝑥𝑁𝑁 = 𝑟𝑟𝑁𝑁
To find the optimal control sequence 𝑢𝑢0 , 𝑢𝑢1 , … , 𝑢𝑢𝑁𝑁−1 that drives 𝑥𝑥 from
the given initial point 𝑥𝑥0 to the desired final state 𝑥𝑥𝑁𝑁 = 𝑟𝑟𝑁𝑁 while
minimizing 𝐽𝐽0 .

Step 1: Formulate the Hamiltonian

MECE6388 Optimal Control Theory, 2024 Fall
Dr. Zheng Chen
𝑟𝑟
𝐻𝐻𝑘𝑘 = 𝐿𝐿𝑘𝑘 + 𝜆𝜆𝑇𝑇𝑘𝑘+1 𝑓𝑓 𝑘𝑘 = 𝑢𝑢𝑘𝑘2 + 𝜆𝜆𝑘𝑘+1 (𝑎𝑎𝑥𝑥𝑘𝑘 + 𝑏𝑏𝑢𝑢𝑘𝑘 )
2
And the optimal conditions are
𝜕𝜕𝐻𝐻𝑘𝑘
𝑥𝑥𝑘𝑘+1 = = 𝑎𝑎𝑥𝑥𝑘𝑘 + 𝑏𝑏𝑢𝑢𝑘𝑘 , (1)
𝜕𝜕𝜆𝜆𝑘𝑘+1
𝜕𝜕𝐻𝐻𝑘𝑘
𝜆𝜆𝑘𝑘 = = 𝑎𝑎𝜆𝜆𝑘𝑘+1 , (2)
𝜕𝜕𝑥𝑥𝑘𝑘
𝜕𝜕𝐻𝐻𝑘𝑘
0= = 𝑟𝑟𝑢𝑢𝑘𝑘 + 𝑏𝑏𝜆𝜆𝑘𝑘+1 , (3)
𝜕𝜕𝑢𝑢𝑘𝑘
Solving the stationary condition (3) yields
𝑏𝑏
𝑢𝑢𝑘𝑘 = − 𝜆𝜆𝑘𝑘+1 , (4)
𝑟𝑟
So, if we find the optimal 𝜆𝜆𝑘𝑘 , then we obtain the optimal control 𝑢𝑢𝑘𝑘 . To
find the optimal 𝜆𝜆𝑘𝑘 , we look at equation (2)
𝜆𝜆𝑘𝑘 = 𝑎𝑎𝜆𝜆𝑘𝑘+1 ⟹ 𝜆𝜆𝑘𝑘 = 𝑎𝑎𝑁𝑁−𝑘𝑘 𝜆𝜆𝑁𝑁 , (5)
If we know 𝜆𝜆𝑁𝑁 , we solve the problem. However, 𝜆𝜆𝑁𝑁 is still unknown. To
find it, we look at eq (1).
𝑥𝑥𝑘𝑘+1 = 𝑎𝑎𝑥𝑥𝑘𝑘 + 𝑏𝑏𝑢𝑢𝑘𝑘
From (4)
𝑏𝑏 2
𝑥𝑥𝑘𝑘+1 = 𝑎𝑎𝑥𝑥𝑘𝑘 − 𝜆𝜆𝑘𝑘+1
𝑟𝑟
From (5)
𝑏𝑏 2 𝑁𝑁−𝑘𝑘−1
𝑥𝑥𝑘𝑘+1 = 𝑎𝑎𝑥𝑥𝑘𝑘 − 𝑎𝑎 𝜆𝜆𝑁𝑁
𝑟𝑟
𝑏𝑏2
Define 𝛾𝛾 ≜ , we get
𝑟𝑟
𝑥𝑥𝑘𝑘+1 = 𝑎𝑎𝑥𝑥𝑘𝑘 − 𝛾𝛾𝑎𝑎𝑁𝑁−𝑘𝑘−1 𝜆𝜆𝑁𝑁
With initial state 𝑥𝑥0
𝑘𝑘−1

𝑥𝑥𝑘𝑘 = 𝑎𝑎𝑘𝑘 𝑥𝑥0 − � 𝑎𝑎𝑘𝑘−𝑖𝑖−1 �𝛾𝛾𝜆𝜆𝑁𝑁 𝑎𝑎𝑁𝑁−𝑖𝑖−1 �

𝑖𝑖=0
𝑘𝑘−1

= 𝑎𝑎𝑘𝑘 𝑥𝑥0 − 𝛾𝛾𝜆𝜆𝑁𝑁 𝑎𝑎𝑁𝑁+𝑘𝑘−2 � 𝑎𝑎−2𝑖𝑖

𝑖𝑖=0
MECE6388 Optimal Control Theory, 2024 Fall
Dr. Zheng Chen
Using the formula for the sum of a geometric series
𝑘𝑘 𝑁𝑁+𝑘𝑘−2
1 − 𝑎𝑎−2𝑘𝑘
𝑥𝑥𝑘𝑘 = 𝑎𝑎 𝑥𝑥0 − 𝛾𝛾𝜆𝜆𝑁𝑁 𝑎𝑎
1 − 𝑎𝑎−2
𝑘𝑘 𝑁𝑁−𝑘𝑘
1 − 𝑎𝑎2𝑘𝑘
= 𝑎𝑎 𝑥𝑥0 − 𝛾𝛾𝜆𝜆𝑁𝑁 𝑎𝑎
1 − 𝑎𝑎2
To find 𝜆𝜆𝑁𝑁 , since the final state is fixed (𝑥𝑥𝑁𝑁 = 𝑟𝑟𝑁𝑁 ) and
𝑁𝑁 𝑁𝑁−𝑁𝑁
1 − 𝑎𝑎2𝑁𝑁
𝑥𝑥𝑁𝑁 = 𝑎𝑎 𝑥𝑥0 − 𝛾𝛾𝜆𝜆𝑁𝑁 𝑎𝑎
1 − 𝑎𝑎2
2𝑁𝑁
(1 − 𝑎𝑎 )𝛾𝛾
= 𝑎𝑎𝑁𝑁 𝑥𝑥0 − 𝜆𝜆𝑁𝑁
1 − 𝑎𝑎2
So
1
𝜆𝜆𝑁𝑁 = − (rN − 𝑎𝑎𝑁𝑁 𝑥𝑥0 )
Λ
Where
γ(1 − a2N )
Λ≜
1 − a2
𝑏𝑏
Since 𝑢𝑢𝑘𝑘 = − 𝜆𝜆𝑘𝑘+1 and 𝜆𝜆𝑘𝑘+1 = 𝑎𝑎𝑁𝑁−𝑘𝑘−1 𝜆𝜆𝑁𝑁
𝑟𝑟
∗
𝑏𝑏 𝑁𝑁 𝑁𝑁−𝑘𝑘−1
(1 − 𝑎𝑎2 )
𝑢𝑢𝑘𝑘 = (𝑟𝑟𝑁𝑁 − 𝑎𝑎 𝑥𝑥0 )𝑎𝑎 = (𝑟𝑟𝑁𝑁 − 𝑎𝑎𝑁𝑁 𝑥𝑥0 )𝑎𝑎𝑁𝑁−𝑘𝑘−1
𝑟𝑟Λ 𝑏𝑏(1 − 𝑎𝑎 )2𝑁𝑁
∗
Note: 𝑢𝑢𝑘𝑘 does not depend on 𝑟𝑟.

To find out 𝑥𝑥𝑘𝑘∗ and 𝐽𝐽0∗ with the optimal control 𝑢𝑢∗ , we put 𝑢𝑢∗ into the
dynamical equation
∗ ∗
1 − 𝑎𝑎2
𝑥𝑥𝑘𝑘+1 = 𝑎𝑎𝑥𝑥𝑘𝑘 + 2𝑁𝑁
(𝑟𝑟𝑁𝑁 − 𝑎𝑎𝑁𝑁 𝑥𝑥0 )𝑎𝑎𝑁𝑁−𝑘𝑘−1
1 − 𝑎𝑎
The solution of this dynamical system with forcing function given by the
second term is
𝑘𝑘−1
∗ 𝑘𝑘
1 − 𝑎𝑎2
𝑥𝑥𝑘𝑘 = 𝑎𝑎 𝑥𝑥0 + 2𝑁𝑁
(𝑟𝑟𝑁𝑁 − 𝑎𝑎𝑁𝑁 𝑥𝑥0 ) � 𝑎𝑎𝑘𝑘−𝑖𝑖−1 𝑎𝑎𝑁𝑁−𝑖𝑖−1
1 − 𝑎𝑎
𝑖𝑖=0
Rewrite it as
MECE6388 Optimal Control Theory, 2024 Fall
Dr. Zheng Chen
∗
(1 − 𝑎𝑎2(𝑁𝑁−𝑘𝑘) )𝑎𝑎𝑘𝑘 𝑥𝑥0 + (1 − 𝑎𝑎2𝑘𝑘 )𝑎𝑎𝑁𝑁−𝑘𝑘 𝑟𝑟𝑁𝑁
𝑥𝑥𝑘𝑘 =
1 − 𝑎𝑎2𝑁𝑁
where 𝑘𝑘 = 0, 𝑥𝑥0 = 𝑥𝑥0 and when 𝑘𝑘 = 𝑁𝑁, 𝑥𝑥𝑁𝑁∗ = 𝑥𝑥𝑁𝑁 .
∗

The optimal performance index is

𝑁𝑁−1
𝑟𝑟 2
𝐽𝐽0∗ = � 𝑢𝑢𝑘𝑘∗
2
𝑘𝑘=0
𝑁𝑁−1 2
𝑟𝑟 1 − 𝑎𝑎2
= �� (𝑟𝑟𝑁𝑁 − 𝑎𝑎𝑁𝑁 𝑥𝑥0 )𝑎𝑎𝑁𝑁−𝑘𝑘−1 �
2 𝑏𝑏(1 − 𝑎𝑎 )
2𝑁𝑁
𝑘𝑘=0
𝑁𝑁−1
𝑟𝑟 (1 − 𝑎𝑎2 )2
= (𝑟𝑟𝑁𝑁 − 𝑎𝑎𝑁𝑁 𝑥𝑥0 )2 � 𝑎𝑎2(𝑁𝑁−𝑘𝑘−1)
2 𝑏𝑏 (1 − 𝑎𝑎 )
2 2𝑁𝑁 2
𝑘𝑘=0
1
= (𝑟𝑟𝑁𝑁 − 𝑎𝑎𝑁𝑁 𝑥𝑥0 )2
2Λ

The optimal performance index is only related to the final state and the
initial state.

Case B: Free Final State

We desire the system state variable 𝑥𝑥𝑁𝑁 → 𝑟𝑟𝑁𝑁 at times 𝑁𝑁 but not exactly
𝑥𝑥𝑁𝑁 = 𝑟𝑟𝑁𝑁 . Therefore, we make the difference 𝑥𝑥𝑁𝑁 − 𝑟𝑟𝑁𝑁 small by including
it in the performance index
𝑁𝑁−1
1 𝑟𝑟
𝐽𝐽0 = (𝑥𝑥𝑁𝑁 − 𝑟𝑟𝑁𝑁 )2 + � 𝑢𝑢𝑘𝑘2
2 2
𝑘𝑘=0
Now the optimal control will attempt to make |𝑥𝑥𝑁𝑁 − 𝑟𝑟𝑁𝑁 | small while also
using low control energy. In this case, the function 𝜙𝜙
1
𝜙𝜙 = (𝑥𝑥𝑁𝑁 − 𝑟𝑟𝑁𝑁 )2
2
But 𝑓𝑓𝑘𝑘 and 𝐿𝐿𝑘𝑘 are not changed. The Hamiltonian is still the same as we
discussed in Case A. The only change is the boundary condition. Since
𝑑𝑑𝑥𝑥𝑁𝑁 ≠ 0, we must have
MECE6388 Optimal Control Theory, 2024 Fall
Dr. Zheng Chen
𝜕𝜕𝜕𝜕
𝜆𝜆𝑁𝑁 = = 𝑥𝑥𝑁𝑁 − 𝑟𝑟𝑁𝑁
𝜕𝜕𝑥𝑥𝑁𝑁
Since
𝜆𝜆𝑘𝑘 = 𝑎𝑎𝑁𝑁−𝑘𝑘 𝜆𝜆𝑁𝑁 ⇒ 𝜆𝜆𝑘𝑘 = 𝑎𝑎𝑁𝑁−𝑘𝑘 (𝑥𝑥𝑁𝑁 − 𝑟𝑟𝑁𝑁 )
And
𝑏𝑏 𝑏𝑏
𝑢𝑢𝑘𝑘 = − 𝜆𝜆𝑘𝑘+1 ⇒ 𝑢𝑢𝑘𝑘 = − 𝑎𝑎𝑁𝑁−𝑘𝑘−1 (𝑥𝑥𝑁𝑁 − 𝑟𝑟𝑁𝑁 )
𝛾𝛾 𝑟𝑟
Since
𝑥𝑥𝑁𝑁 = 𝑎𝑎𝑁𝑁 𝑥𝑥0 − Λ𝜆𝜆𝑁𝑁
= 𝑎𝑎𝑁𝑁 𝑥𝑥0 − Λ(𝑥𝑥𝑁𝑁 − 𝑟𝑟𝑁𝑁 )
Solving for 𝑥𝑥𝑁𝑁 gives
Λ𝑟𝑟𝑁𝑁 + 𝑎𝑎𝑁𝑁 𝑥𝑥0
𝑥𝑥𝑁𝑁 =
1+Λ
With 𝑥𝑥𝑁𝑁 , we get the costate
𝑁𝑁−𝑘𝑘 (
𝑟𝑟𝑁𝑁 − 𝑎𝑎𝑁𝑁 𝑥𝑥0 𝑁𝑁−𝑘𝑘
𝜆𝜆𝑘𝑘 = 𝑎𝑎 𝑥𝑥𝑁𝑁 − 𝑟𝑟𝑁𝑁 ) = − 𝑎𝑎
1+Λ
And the optimal control
𝑏𝑏
𝑢𝑢𝑘𝑘∗ = − 𝜆𝜆𝑘𝑘+1
𝑟𝑟
𝑏𝑏
= (𝑟𝑟𝑁𝑁 − 𝑎𝑎𝑁𝑁 𝑥𝑥0 )𝑎𝑎𝑁𝑁−𝑘𝑘−1
𝑟𝑟(1 + Λ)
Note: the optimal control for free final state does depend on 𝑟𝑟,

As 𝑟𝑟 → 0, we are concerned less and less about the control energy since
𝑢𝑢𝑘𝑘2 is weighted less and less heavily in 𝐽𝐽0 .
When we look at the final state
𝑎𝑎𝑁𝑁
𝑟𝑟𝑁𝑁 + 𝑥𝑥0
𝑥𝑥𝑁𝑁 = Λ
1
1+
Λ
𝑏𝑏2 (1−𝑎𝑎2𝑁𝑁 ) 𝑟𝑟𝑁𝑁 +0
As 𝑟𝑟 → 0, Λ = (1−𝑎𝑎2 )
→ ∞, 𝑥𝑥𝑁𝑁 → = 𝑟𝑟𝑁𝑁
𝑟𝑟 1+0
MECE6388 Optimal Control Theory, 2024 Fall
Dr. Zheng Chen
In summary, if we are not concerned about energy, the final state will
reach its desired value. There is a trade-off between the energy
consumed and final state desired. The value of 𝑟𝑟 effects the tradeoff.

To find the optimal value of the performance index, use optimal control
𝑢𝑢𝑘𝑘∗ in (2) and simplify to obtain
Λ
𝐽𝐽0∗ = (𝑟𝑟𝑁𝑁 − 𝑎𝑎𝑁𝑁 𝑥𝑥0 )2
2(1 + Λ)2
1
As 𝑟𝑟 → 0, Λ → ∞, 𝐽𝐽0∗ → (𝑟𝑟𝑁𝑁 − 𝑎𝑎𝑁𝑁 𝑥𝑥0 )2 , which is the fixed final state
2Λ
cost.