Deep Reinforcement Learning

CS 285, University of California, Berkeley

Harry Zhang

December 2019
Contents

Preface v

1 Introduction 1
1.1 Important Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Value Function and Q Function . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2.1 Q Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2.2 Value Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.3 Reinforcement Learning Anatomy . . . . . . . . . . . . . . . . . . . . . . . . 3

2 Imitation Learning 4
2.1 Distribution Mismatch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.2 Dataset Aggregation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.3 When Does Imitation Learning Fail? . . . . . . . . . . . . . . . . . . . . . . 5
2.3.1 Non-Markovian Behaviors . . . . . . . . . . . . . . . . . . . . . . . . 5
2.3.2 Multimodal Behaviors . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.4 Theoretical Analysis of Imitation Learning’s Error . . . . . . . . . . . . . . . 7
2.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

3 Policy Gradient Methods 8

3.1 Policy Gradient Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
3.2 Evaluating the Policy Gradient . . . . . . . . . . . . . . . . . . . . . . . . . 9
3.3 Example: Gaussian Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
3.4 Intuition behind Policy Gradient: What are We Actually Doing? . . . . . . . 10
3.5 Partial Observability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3.6 Disadvantages of the Policy Gradient . . . . . . . . . . . . . . . . . . . . . . 11
3.7 Reducing Policy Gradients Variance using Baselines . . . . . . . . . . . . . . 11
3.7.1 Causality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3.7.2 Baselines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
3.7.3 Analyzing the Variance with Baselines . . . . . . . . . . . . . . . . . 12
3.8 On-Policy vs. Off-Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3.8.1 Off-policy Learning and Importance Sampling . . . . . . . . . . . . . 13
3.8.2 Deriving Policy Gradient with Importance Sampling . . . . . . . . . . 14
3.9 First Order Approximation for Importance Sampling . . . . . . . . . . . . . 14
3.9.1 Advanced Policy Gradients . . . . . . . . . . . . . . . . . . . . . . . . 15

i
CONTENTS ii

4 Actor-Critic Algorithms 17
4.1 Reward-to-Go . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
4.2 Using Baselines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
4.3 Value Function Fitting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
4.4 Policy Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
4.4.1 Why Do We Evaluate a Policy . . . . . . . . . . . . . . . . . . . . . . 19
4.4.2 How to Evaluate a Policy . . . . . . . . . . . . . . . . . . . . . . . . 19
4.4.3 Monte Carlo Evaluation with Function Approximation . . . . . . . . 20
4.4.4 Improving the Estimate Using Bootstrap . . . . . . . . . . . . . . . . 20
4.5 Batch Actor-Critic Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . 20
4.6 Aside: Discount Factors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
4.7 Online Actor-Critic Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . 22
4.8 Critics as State-Dependent Baselines . . . . . . . . . . . . . . . . . . . . . . 23
4.9 Eligibility Traces and n-Step Returns . . . . . . . . . . . . . . . . . . . . . . 23

5 Value Function Methods 25

5.1 An Implicit Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
5.2 Policy Iteration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
5.2.1 High Level Idea . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
5.2.2 Dynamic Programming . . . . . . . . . . . . . . . . . . . . . . . . . . 26
5.3 Fitted Value Iteration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
5.3.1 Fitted Supervised Value Iteration Algorithm . . . . . . . . . . . . . . 27
5.3.2 Fitted Q-Iteration Algorithm . . . . . . . . . . . . . . . . . . . . . . 27
5.3.3 A Closer Look at Q-Iteration Algorithm . . . . . . . . . . . . . . . . 28
5.3.4 Online Q-Iteration Algorithm . . . . . . . . . . . . . . . . . . . . . . 28
5.4 Value Function Learning Theory . . . . . . . . . . . . . . . . . . . . . . . . . 28

6 Q-Function Methods 30
6.1 Replay Buffers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
6.2 Target Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
6.3 Inaccuracy in Q-Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
6.3.1 Double Q-Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
6.3.2 N-Step Return Estimator . . . . . . . . . . . . . . . . . . . . . . . . . 33
6.3.3 Q-Learning with Continuous Actions . . . . . . . . . . . . . . . . . . 33

7 Policy Gradients Theory and Advanced Policy Gradients 35

7.1 Policy Gradient as Policy Iteration . . . . . . . . . . . . . . . . . . . . . . . 35
7.2 Distribution Mismatch Bound . . . . . . . . . . . . . . . . . . . . . . . . . . 36
7.2.1 A Simple  Bound . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
7.2.2 A More Convenient Bound - KL Divergence . . . . . . . . . . . . . . 38
7.2.3 Enforcing the Distribution Mismatch Constraint . . . . . . . . . . . . 38
7.2.4 Other Optimization Techniques . . . . . . . . . . . . . . . . . . . . . 38
CONTENTS iii

8 Model-Based Reinforcement Learning 41

8.1 Optimal Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
8.2 Open-loop Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
8.2.1 Random Shooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
8.2.2 Cross Entropy Method . . . . . . . . . . . . . . . . . . . . . . . . . . 43
8.2.3 Monte Carlo Tree Search . . . . . . . . . . . . . . . . . . . . . . . . . 43
8.2.4 UCT Tree Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
8.2.5 Using Derivatives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
8.2.6 Shooting Methods and Collocation Methods . . . . . . . . . . . . . . 44
8.2.7 Linear Quadratic Regulator (LQR) . . . . . . . . . . . . . . . . . . . 45
8.2.8 Iterative LQR (iLQR) . . . . . . . . . . . . . . . . . . . . . . . . . . 47
8.3 Model-based RL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
8.3.1 Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
8.3.2 Performance Gaps in Model-based RL . . . . . . . . . . . . . . . . . 50
8.3.3 Uncertainty-aware Models . . . . . . . . . . . . . . . . . . . . . . . . 50
8.3.4 Latent Space Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

9 Model-based Policy Learning 54

9.1 Back-propagate into the Policy . . . . . . . . . . . . . . . . . . . . . . . . . 54
9.1.1 Vanishing and Exploding Gradients . . . . . . . . . . . . . . . . . . . 54
9.2 Model-free Optimization with a Model . . . . . . . . . . . . . . . . . . . . . 55
9.2.1 Dyna . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
9.3 Local and Global Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
9.3.1 Local Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
9.3.2 Guided Policy Search . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
9.3.3 Distillation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

10 Variational Inference and Generative Models 60

10.1 Training Latent Variable Models . . . . . . . . . . . . . . . . . . . . . . . . . 60
10.1.1 Variational Approximation . . . . . . . . . . . . . . . . . . . . . . . . 61
10.1.2 Amortized Variational Inference . . . . . . . . . . . . . . . . . . . . . 62
10.1.3 The Reparameterization Trick . . . . . . . . . . . . . . . . . . . . . . 62
10.2 Variational Autoencoder (VAE) . . . . . . . . . . . . . . . . . . . . . . . . . 63

11 Control as Inference 64
11.1 Probabilistic Graphical Model of Decision Making . . . . . . . . . . . . . . . 64
11.1.1 Inference in the Optimality Model . . . . . . . . . . . . . . . . . . . . 65
11.1.2 Inferring the Backward Messages . . . . . . . . . . . . . . . . . . . . 65
11.1.3 A Closer Look . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
11.1.4 Aside: The Action Prior . . . . . . . . . . . . . . . . . . . . . . . . . 66
11.1.5 Inferring the Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
11.1.6 Inferring the Forward Messages . . . . . . . . . . . . . . . . . . . . . 67
11.2 The Optimism Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
CONTENTS iv

12 Inverse Reinforcement Learning 70

12.1 Feature Matching Inverse RL . . . . . . . . . . . . . . . . . . . . . . . . . . 70
12.2 Learning the Optimality Variable . . . . . . . . . . . . . . . . . . . . . . . . 71
12.2.1 Inverse RL Partition Function . . . . . . . . . . . . . . . . . . . . . . 71
12.2.2 Estimating the Expectation . . . . . . . . . . . . . . . . . . . . . . . 72
12.3 Unknown Dynamics and Large State/Action Spaces . . . . . . . . . . . . . . 73
12.3.1 More Efficient Updates . . . . . . . . . . . . . . . . . . . . . . . . . . 73
12.4 Inverse RL as a Generative Adversarial Network . . . . . . . . . . . . . . . . 74

13 Transfer Learning 76

14 Exploration 77
14.1 Multi-arm Bandits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
14.1.1 Defining a Bandit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
14.1.2 Optimistic Exploration . . . . . . . . . . . . . . . . . . . . . . . . . . 78
14.1.3 Probability Matching . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
14.1.4 Information Gain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
14.2 Exploration in MDPs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
14.2.1 Counting the Exploration Bonus . . . . . . . . . . . . . . . . . . . . . 80
14.3 Exploration with Q-functions . . . . . . . . . . . . . . . . . . . . . . . . . . 81
14.4 Revisiting Information Gain in MDP Exploration . . . . . . . . . . . . . . . 81
14.4.1 Prediction Gain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
14.4.2 Variational Information Maximization for Exploration (VIME) . . . . 82
14.5 Improving RL with Imitation . . . . . . . . . . . . . . . . . . . . . . . . . . 83
14.5.1 Pretrain and Finetune . . . . . . . . . . . . . . . . . . . . . . . . . . 83
14.5.2 Off-policy RL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
14.5.3 Q-learning with Demonstrations . . . . . . . . . . . . . . . . . . . . . 84
14.5.4 Imitation as an Auxiliary Loss Function . . . . . . . . . . . . . . . . 84

15 Offline RL 85
15.1 Offline RL Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
Preface

This is a compilation of notes from UC Berkeley’s CS 285 (formerly CS 294-112) taught by

Professor Sergey Levine. The notes are based on Fall 2018, Fall 2019, and Fall 2020 offerings
of the course. The author is currently a EECS student at UC Berkeley, and the readers are
welcome to contact the author via [email protected].
The notes assume the reader have some familiarity with key concepts such as machine
learning, neural networks, Markov Decision Process (MDP), and optimal control.

Rest in Peace, Andy.

v
Chapter 1: Introduction

Here we review some of the terminologies that frequently appear in the field of Reinforcement
Learning.

1.1 Important Concepts

Imagine a discrete-time system (environment), meaning that the system is discretized into
time steps. At time step t, we define the state of the system to be st . The state of a
system/agent could be some intrinsic data of it. For example, for a car, the state of a car at
a given time step could be the car’s angular velocity, acceleration, and mass. We also define
an agent’s action at , which is equivalent to the notion of input in control theory. A policy
is a function that takes in a state and outputs an action, determining what the agent should
do given current time step’s state.
A policy function could be deterministic such that at = πθ (st ). The policy function
could also be a distribution, which we can define as πθ (at |st ). Note that in many cases, a
state is not fully observable, so we might only partially observe the state of the agent via
an observation ot . In this case, the policy function should condition on the observation ot .
We should also define a transition function of the environment, which is also called the
model of the system. In the most general case, the transition should be stochastic, meaning
that a state could evolve into a number of other states potentially. Therefore, this function
should be a distribution, defined as p(st+1 |st , at ). Note that this distribution is conditioned
on both the state and the action at time step t. If you are familiar with control theory, you
would probably notice the resemblance of the transition distribution with a discrete-time
system’s dynamics function. A sequence of states and actions become a trajectory, which
we call τ .
Meanwhile, we also define a reward function of the environment. For example, in
Pacman, the player gains one point after eating a dot, and loses 100 points after being eaten
by a ghost. Formally, the reward function should be a function of both state and action, so
we denote the reward function as r(s, a). The rewards could be a hand-engineered function,
or it could be implicit that the agent actually needs to learn the rewards first, as seen in the
case of Inverse RL (IRL).
In many reinforcement learning problems, we assume the states transitions are Marko-
vian, meaning that the state at time step t is only responsible for contributing to the state
at time step t + 1. Fig. 1.1 an illustration of a Markov chain, and the direction of the arrow
means causality.
In control theory, such as the LQR optimization problem, we aim to minimize the cost
function of the system. What would be an equivalent notion in reinforcement learning?
Recall we defined a reward function, r(s, a), so naturally we want to collect as much reward

1
CHAPTER 1. INTRODUCTION 2

Figure 1.1: A simple Markov chain.

as possible in the environment. Without loss of generality, we assume that the environment
is stochastic. Define a trajectory distribution pθ (τ ) according to Bayes’ Rule:
T
Y
p(τ ) := p(s1 , a1 , ..., sT , aT ) = p(s1 ) πθ (at |st )p(st+1 |st , at )
t=1

where T is the length of episode horizon. Therefore, given this trajectory distribution, we can
calculate thePexpected value of the total reward function induced by following this trajectory
as Eτ ∼pθ (τ ) [ t r(st , at )]. Therefore, to optimize this objective, we want to find a parameter
θ, such that θ maximizes the above expectation:
" #
X
θ = arg max Eτ ∼pθ (τ ) r(st , at )
θ t

1.2 Value Function and Q Function

To facilitate our calculation of the above expectation, and to simplify the notations, we intro-
duce two important types of functions: Q function and value function. In most cases these
two functions are not given to us in closed form, and one needs to approximate and improve
the functions using some deep neural net, hence the notion of “deep” in deep reinforcement
learning.
1.2.1 Q Function
Q function is a function of both state and action, so it is denoted as Q(st , at ) and it quantita-
tively measures the quality of taking action at at state st . Mathematically, it is the expected
sum of reward from the current time step given state st and action at . This is very similar to
the “cost-to-go” function in control theory, especially Model Predictive Control. We define
Q(s, a) as:
T
X
π
Q (st , at ) = Eπθ [r(s0t , a0t )|st , at ]
t0 =t

1.2.2 Value Function

Unlike the Q function, value function is only a function of state, so intuitively it quanti-
tatively
PT measures the value of being in state st . Mathematically, it is defined as V π (st ) =
0 0
t0 =t Eπθ [r(st , at )|st ]. Again by Bayes’ rule we can obtain the relation between Value func-
tion and Q function: V π (st ) = Eat ∼π(at |st ) [Qπ (st , at )].
CHAPTER 1. INTRODUCTION 3

Figure 1.2: Three steps of reinforcement learning.

Furthermore, if we sum the value function over all possible initial states, we essentially
recovered the objective of reinforcement learning: Es1 ∼p(s1 ) [V π (s1 )], where p(s1 ) is a known
distribution of all possible initial states.

1.3 Reinforcement Learning Anatomy

In RL, we usually have three parts in the whole pipeline. We need to keep generating data
for the agent to learn, and use the data to generate samples in order to fit and regress onto
a model. Then with this model, we estimate the reward and based on the reward we update
our policy to maximize the reward, and we go back to step 1. Therefore, our primal concern
is to efficiently run the three parts so that the agent can learn optimally with less data and
computation. Here is an illustration of the three steps in Fig. 1.2.
Chapter 2: Imitation Learning

Imitation learning is also called behavioral cloning. The basic idea of imitation learning is
“train to fit the expert behavior”. In other words, given a demonstration, we want to make
the agent follow the demonstration as closely as possible, to best imitate the demonstration’s
behaviors.

2.1 Distribution Mismatch

However, a big problem of such type of approach is that it does not generalize well, if at all.
For example, one can imagine that the agent makes a small mistake and ends up being in
a slightly different state from what it has seen (trained) before, but since the state is novel,
the agent does not know how to act, thus behaving randomly, diverging from the learned
trajectory. An illustration of such mismatches is shown in Fig. 2.1.

2.2 Dataset Aggregation

The aggregation of mistakes that the agent makes often makes imitation learning not feasible.
But imitation learning does work in some cases. Intuitively, if the agent could somehow learn
from the mistakes, and we keep appending data to the agent’s dataset so that the agent is
exposed to a variety of states, then the expected trajectory would get closer to the training
trajectory. If the actions applied to those states are correct, then eventually, the agent can,
ideally, converge to an optimal trajectory. This is essentially the idea behind an imitation
learning algorithm called Dataset Aggregation (DAgger) [1]. DAgger essentially makes the
training trajectory close to the expected trajectory by constantly appending test data to
training data. In step 3 of Algorithm 1, what we are doing is basically discarding the actions
from running trained policy πθ (at |ot ). Instead, we ask a human in the loop to label what they

Figure 2.1: Mistakes aggregate in behavior cloning.

4
CHAPTER 2. IMITATION LEARNING 5

Algorithm 1 Dataset Aggregation (DAgger)

Require: Human data D = {o1 , a1 , ..., oN , aN }
1: while true do
2: Train πθ (at |ot ) from human data D = {o1 , a1 , ..., oN , aN }.
3: Run πθ (at |ot ) to get dataset Dπ = {o1 , ..., M }
4: Ask human to label Dπ with actions at
5: Aggregate D ← D ∪ Dπ
6: return optimal imitation-learned trajectory as τ return

would have done based on the observations in Dπ . Take an autonomous car for example, the
training data would be images labeled with steering commands, and we let the car collect
more data, which are only images. Then we give those images to a human expert, and let
the human determine, based on each image, what action (steer left, right, or go straight)
that the human would have applied if he observed such an image.
It can be proven that DAgger resolves the distribution “drift” issue. However, one
problem with DAgger is that human might be error-prone, so the human labelled data
might be flawed to use. Furthermore, more subtly, human, in most cases, does not make
decisions based on a Markovian process. Therefore, the current time step’s action might be
dependent on a state/observation some number of time steps ago.

2.3 When Does Imitation Learning Fail?

In general, there are some cases where one may fail to fit the expert data, which lead to
undesirable outcomes of imitation learning.
2.3.1 Non-Markovian Behaviors
First, the data is Non-Markovian, as mentioned above. Actually Non-Markovian process is
more natural and intuitive for human in that human learns from their past mistakes. Why
is this wrong? Essentially, we are fitting the wrong distribution. Since we are fitting a
policy distribution based on a Markovian process, our goal is fitting π(at |ot ). However, if the
expert data is not Markovian, then we are trying to fit π(at |ot ) from another distribution
π(at |o + 1, ..., ot ). One solution is to use a lot of previous memory frames, and concatenate
them as one huge frame, effectively augmenting the state space. However, this solution might
require too many weights in the neural net encoder, significantly increasing the computational
complexity. Another solution is that one can fit the expert data using an RNN with shared
weights in the convolutional encoder, and one possible implementation is shown in Fig. 2.2.
Usually, an LSTM cell works well.
The underlying reason why having a full history makes imitation learning difficult is
that history data tends to exacerbate the causality misclassification, which is also called
causal confusion. Having more history might make the agent learn the wrong direction
of causality, and in many cases, the wrong direction is actually easier to learn. This can
be illustrated in an autonomous car example. Consider a car that has a brake light on the
dashboard that lights up every time the brake pedal is pressed. When the brake pedal is
pressed due to the red light/obstacle in front of the vehicle, it is probably easier for the agent
CHAPTER 2. IMITATION LEARNING 6

Figure 2.2: Using RNN to address non-Markovian expert data.

Figure 2.3: Multimodal behaviors.

to associate the brake action with the brake light rather than with the red light/obstacle in
front of the car. The causal confusion issue can be alleviated with the use of DAgger because
the human annotator is able to provide the correct causal relation. For more information,
please refer to this paper [2].
2.3.2 Multimodal Behaviors
Another scenario where fitting expert might fail is that the expert has multi-modal behav-
iors. An example of this is that when you are controlling a drone to dodge a tree ahead,
you either steer left or steer right. However, if you choose the wrong parametric form of
the distribution (e.g. a simple Gaussian) of the actions, the distribution might average out
left and right and choose to go straight, as shown in figure 2.3. Some methods to mitigate
this issue include: first, one can use a mixture of different Gaussian distributions, instead of
just one. Second, construct a latent space variables model, which we will talk more about in
variational inference. Third, we can use autogregressive discretization. Specifically, a mix-
ture of Gaussians means that the policy distribution should be a weighted sum of different
Gaussians with different means and variances.
CHAPTER 2. IMITATION LEARNING 7

2.4 Theoretical Analysis of Imitation Learning’s Error

First we define two different reward functions for imitation learning. To make our analysis
easier, we assume the policy function is deterministic. First, we can define the reward
function as r(s, a) = log p(a = π ∗ (s)|s). This function measures the log likelihood of the
action equal to expert policy’s action. Another (simpler) choice can just be a counter to
count the number of mistakes. Specifically,
(
0 if a = π ∗ (s)
c(s, a) =
1 o.w.

To analyze this, let’s introduce a lower bound on the probability of making mistakes: πθ (a 6=
π ∗ (s)|s) ≤  for all s ∼ ptrain (s), where ptrain is training data distribution. The fit distribution
of states pθ (s) is consisted of two parts: the first part is the probability of no mistakes made,
and the second part is the probability of making some mistakes. Using Bayes’ rule, we can
calculate pθ (s) as follows:
pθ (st ) = (1 − )t ptrain (st ) + (1 − (1 − )t )pmistake (st )
so to measure the divergence of pθ from ptrain , we take the difference of the two distributions
(naive, total variation divergence):
|pθ (st ) − ptrain (st )| = (1 − (1 − )t )|pmistake − ptrain | ≤ 2(1 − (1 − )t )
≤ 2t
where we used the identity that (1 − )t ≥ 1 − t for  ∈ [0, 1]. Thus, we can calculate the
expected number of mistakes the agent makes using this scheme by:
X XX XX
Epθ (st ) [ct ] = pθ (st )ct (st ) ≤ ptrain (st )ct (st ) + |pθ (st ) − ptrain (st )|cmax
t t st t st
X
≤  + 2t
t

∈ O(T 2 )
Also note that with DAgger ptrain (s) → pθ (s). So we no longer have the second item inside
the summation for DAgger. Thus for DAgger, the expected value should be in O(T ).
As we see, when we have longer horizon length, the errors are going to be aggregated,
thus making more mistakes, and this is one of the most fundamental disadvantages of imi-
tation learning, as discussed in [1].

2.5 Summary
Overall, what are some disadvantages of imitation learning? We have a human factor to
provide data in the entire loop, which is potentially finite, and to generate a good policy,
one need to learn from a lot of data. Moreover, human cannot provide all kinds of data.
Specifically, a human may have trouble with providing data such as the joint angle/torque
of a robotic arm. Therefore, we wish that machines can learn automatically, from unlimited
data.
Chapter 3: Policy Gradient Methods

Recall the objective of Reinforcement Learning:

" #
X
θ = arg max Eτ ∼pθ (τ ) r(st , at )
θ t

This is actually framed as an optimization problem. Therefore, we can use a variety of

optimization techniques, such as gradient descent, to optimize this objective. To be more
concrete, let us define a function J(θ):

J(θ) = Eτ ∼πθ (τ ) [r(τ )]

PT r(τ ) is the sum of reward incurred in this trajectory, so it can be equivalently

By definition,
defined as t=1 r(st , at ), and by definition of expectation, we can more conveniently express
the J function as an integral of the product of policy and reward:
Z
J(θ) = πθ r(τ )dτ

. With this integral, we can easily take the gradient to perform gradient descent/ascent. A
convenient expression of the gradient of J(θ) is shown below.

3.1 Policy Gradient Theorem

In this section, we will derive the mathematical expression of the policy gradient theorem.
Recall a convenient identity:

∇θ πθ (τ )
πθ (τ )∇θ log πθ (τ ) = πθ (τ ) = ∇θ πθ (τ )
πθ (τ )

Using this identity, we can take the gradient of J(θ) in a cleaner fashion:
Z
∇θ J(θ) = ∇θ πθ r(τ )dτ
Z
= πθ (τ )∇θ log πθ (τ )r(τ )dτ

= Eτ ∼πθ (τ ) [∇θ log πθ (τ )r(τ )]

Now, we want to get rid of the huge log πθ (τ ) from our equation.QRecall that a trajectory
τ is a list of states and actions, so πθ (s1 , a1 , ..., sT , aT ) = p(s1 ) Tt=1 πθ (at |st )p(st+1 |st , at )

8
CHAPTER 3. POLICY GRADIENT METHODS 9

Algorithm 2 REINFORCE Algorithm

Require: Base policy πθ (at |st ), sample trajectories τ i
1: while true do
2: Sample {τ i } P
fromPπθ (at |st ) (run it on a P
robot).
3: ∇θ J(θ) ' N1 i ( t ∇θ log πθ (ai,t |si,t )) ( t r(si,t , ai,t ))
4: Improve policy by θ ← θ + α∇θ J(θ)
5: return optimal trajectory from gradient ascent as τ return

PT Then we take the log on both sides, and we end up getting log πθ (τ ) =
by Bayes’ rule.
log p(s1 ) + t=1 log πθ (at |st ) + log p(st+1 |st , at ). Plugging into our original gradient:
" T
! #
X
∇θ J(θ) = Eτ ∼πθ (τ ) ∇θ log p(s1 ) + log πθ (at |st ) + log p(st+1 |st , at ) r(τ )
t=1
" T
! T
!# (3.1)
X X
= Eτ ∼πθ (τ ) ∇θ log πθ (at |st ) r(st , at )
t=1 t=1

Note that in the above calculation, we cancel out log p(s1 ) and log p(st+1 |st , at ) because we
are taking the gradient with respect to θ, but those two expressions do not depend on θ.
The first item in the final expectation is similar to maximum likelihood.

3.2 Evaluating the Policy Gradient

In our derivation, we mathematically derived an expression for policy gradient, which involves
calculating an expectation. However, in most cases we cannot easily obtain this expectation
easily because it is highly possible that it involves a huge, intractable integral. Therefore,
what are we going to do if the expectation (integral) is hard to evaluate? The answer is to
approximate, and more specifically, we use Monte Carlo approximation. The idea is to take
N samples, and average them out:
N T
! T !
1 X X X
∇θ J(θ) ' ∇θ log πθ (ai,t |si,t ) r(si,t , ai,t )
N i=1 t=1 t=1

where the subscripts i, t means time step t in the i-th rollout. With the above gradient, we
can do gradient descent (ascent) on the parameter θ by:

θ ← θ + α∇θ J(θ)

Now we are ready to propose a vanilla policy gradient algorithm by direct gradient ascent
on the Monte Carlo-approximated policy gradient parameters, the REINFORCE Algorithm,
as shown in Algorithm 2.

3.3 Example: Gaussian Policy

Now let us work out a simple case of running Algorithm 2 on a simple Gaussian policy.
A Gaussian policy means that the policy function is a Gaussian distribution. Specifically,
CHAPTER 3. POLICY GRADIENT METHODS 10

Figure 3.1: More rewarding trajectories are more probable.

πθ (at |st ) = N (fneural net (st ); Σ). One advantage of using a Gaussian policy is that it is easy
to obtain a closed-form expression for the Gaussian derivative. We simply write out the
quadratic discriminant function in a multivariate Gaussian distribution:
1 1
log πθ (at |st ) = − (f (st ) − at )T Σ−1 (f (st ) − at ) + C = − kf (st ) − at k2Σ + C
2 2
Taking the derivative, we have:
1 df
∇θ log πθ (at |st ) = − Σ−1 (f (st ) − at )
2 dθ
And we use Gradient ascent as discussed above.

3.4 Intuition behind Policy Gradient: What are We Ac-

tually Doing?
Recall we mentioned that the first term inside the expectation is similar to maximum like-
lihood. Maximum Likelihood can be written as maximizing the log likelihood of an event.
Let us compare the two side by side. Recall the expression of the policy gradient is:
N
1 X
∇θ J(θ) ' ∇θ log πθ (τi )r(τi )
N i=1

And the Maximum Likelihood is defined as:

N
1 X
∇θ JM L (θ) ' ∇θ log πθ (τi )
N i=1

As we discussed before, the first term in the policy gradient is exactly the same as the
definition of maximum likelihood!
So what are we doing here when taking this gradient? Intuitively, we are assigning
more weight to more rewarding trajectories by making trajectories with higher rewards more
probable. Equivalently, higher-reward trajectories are likely to have more probability to be
chosen. This intuition is crucial to the policy gradient methods and is illustrated in Fig. 3.1.
CHAPTER 3. POLICY GRADIENT METHODS 11

3.5 Partial Observability

Can we use the policy gradient on a Partially Observed Markov Decision Process (POMDP)?
The short answer is Yes. Why? Recall (yet again) the policy gradient expression:
N T
! T !
1 X X X
∇θ J(θ) ' ∇θ log πθ (ai,t |si,t ) r(si,t , ai,t )
N i=1 t=1 t=1

In this expression, we do not even have the transition function in it. Long story short,
the Markovian property is not actually used! So we can use policy gradient on a POMDP
without any modification except instead of st , we use ot .
Note that we do not care about what the state actually is. Any Non-Markovian proces
can be made Markovian by setting the state as the whole history.

3.6 Disadvantages of the Policy Gradient

Recall the intuition behind the policy gradient update: we make trajectories with more
reward more probable. Let’s consider the following scenario: say two trajectories have similar
positive rewards, while another trajectory has a low, negative reward. Then policy gradient
is going to assign zero to low probability to the third trajectory, and high probabilities to
the other two. Now imagine we add a large constant number to our reward function, and
apparently it does not change the relative relation between different trajectories’ rewards
because we are only adding in a constant. Now the negative reward becomes positive, and
policy gradient is likely to spread out the likelihood for the three trajectories since all three
rewards are positive now. This is bad because our reward distribution does not change at
all, but after adding in a constant, the distribution of policy gradient changed substantially.
This problem shows the high variance in our naive policy gradient algorithm. Therefore,
we need to come up with some methods to reduce the variance introduced in the policy
gradients.

3.7 Reducing Policy Gradients Variance using Baselines

3.7.1 Causality
One simple fix for high variance is to use the fact of causality: policy at time t0 cannot affect
reward at t if t < t0 . This simple, commonsensical idea allows us to discard some operands
in the summation:
N T T
!
1 XX X
∇θ J(θ) ' ∇θ log πθ (ai,t |si,t ) r(si,t0 , ai,t0 )
N i=1 t=1 t0 =t

and we define the second item in the summation as the “reward-to-go”. Notice that in the
reward-to-go term, we start the summation from time t instead of 1, by causality. The idea
is that we are multiplying the likelihood by smaller numbers due to the reduction of the
summation term, so we can reduce the variance to some extent.
CHAPTER 3. POLICY GRADIENT METHODS 12

3.7.2 Baselines
Another common approach is to use baselines. By baselines, we mean that instead of making
all high-reward trajectories more likely, we only make trajectories better than average
more likely. So naturally, we define a baseline b as the average reward:
N
1 X
b= r(τ )
N i=1

Incorporating the baseline b into our original policy gradient expression:

N
1 X
∇θ J(θ) ' ∇θ log πθ (τ ) [r(τ ) − b]
N i=1

But, are we allowed to that? Yes, in fact, we can show that the expectation is the same with
baseline b. To show this, we can express the expectation of baseline as:
Z
Eπθ (τ ) [∇θ log πθ (τ )b] = πθ (τ )∇ log πθ (τ )b dτ
Z
= ∇θ πθ (τ )b dτ
Z
= b∇θ πθ (τ ) dτ

= b∇θ 1
=0

Therefore, by subtracting a baseline, our policy gradient is still unbiased in expectation!

3.7.3 Analyzing the Variance with Baselines
Let us explicitly write down the variance of the policy gradient. Recall the definition of
variance:
Var[x] = E[x2 ] − E[x]2
And the policy gradient with baselines is written as:

∇θ J(θ) ' Eτ ∼πθ (τ ) [∇θ log πθ (τ ) (r(τ ) − b)]

Therefore, the variance of the policy gradient can be written as follows:

Var = Eτ ∼πθ (τ ) (∇θ log πθ (τ ) (r(τ ) − b))2 − Eτ ∼πθ (τ ) [∇θ log πθ (τ ) (r(τ ) − b)]2
 

Note that in the second squared expectation term of variance, it can be equivalently written
as Eτ ∼πθ (τ ) [∇θ log πθ (τ )r(τ )]2 since baselines are unbiased in expectation.
CHAPTER 3. POLICY GRADIENT METHODS 13

Now we have an expression of variance with respect to baseline b, we can calculate the
optimal b that minimizes the variance by setting the gradient of variance to 0:
dVar d 
= E g(τ )2 (r(τ ) − b)2

db db
d 
= E g(τ )2 r(τ )2 − 2E g(τ )2 r(τ )b + b2 E g(τ )2
    
db 
= −2E g(τ )2 r(τ ) + 2bE g(τ )2
  

=0

Solving the equation, we will have

E [g(τ )2 r(τ )]
bopt =
E [g(τ )2 ]

where bopt is the optimal baseline value for reducing the variance.
In practice, we just use the average reward for baseline.

3.8 On-Policy vs. Off-Policy

We now introduce two concepts in RL: on-policy and off-policy. On-policy means that
we learn only from using the current policy πθ , and off-policy means we learn also from
other policies. Apparently, policy gradient is an on-policy method because ∇θ J(θ) '
Eτ ∼πθ (τ ) [∇θ log πθ (τ ) (r(τ ) − b)], and the expectation is taken under the current, known tra-
jectory of interest. Therefore, every time we have a new policy, we need to use new samples.
Since we are changing θ, πθ also changes overtime in policy gradient. One can imagine that
this is extremely inefficient in neural networks, because in a neural network, θ only changes
a little and the overhead for changing the policy is large.
One solution is to use off-policy learning.
3.8.1 Off-policy Learning and Importance Sampling
We first introduce an important technique called importance sampling. Given a distribution
p(x), how do we calculate the expectation from samples from another distribution q(x)?
This is the idea of importance sampling, by using an importance ratio, we can calculate the
expectation from another distribution, thus enabling to learn off-policy.
In importance sampling:
Z
Ex∼p(x) [f (x)] = p(x)f (x) dx
Z
q(x)
= p(x)f (x) dx
q(x)
 
p(x)
= Ex∼q(x) f (x)
q(x)

Then we can plug it into the off-policy policy gradient. Say we have a trained policy
πθ (τ ), and we have samples from another policy π̄(τ ), we can use the samples from π̄(τ ) to
CHAPTER 3. POLICY GRADIENT METHODS 14

calculate J(θ) function using importance sampling:

J(θ) = Eτ ∼πθ (τ ) [r(τ )]

 
πθ (τ )
= Eτ ∼π̄(τ ) r(τ )
π̄(τ )

Now Q we want to look closely at the importance ratio. Recall that πθ (τ ) =

p(s1 ) Tt=1 πθ (at |st )p(st+1 |st , at ). Then we can simplify the ratio in the following way:

p(s1 ) Tt=1 πθ (at |st )p(st+1 |st , at )

Q
πθ (τ )
=
π̄(τ ) p(s1 ) Tt=1 π̄(at |st )p(st+1 |st , at )
Q
QT
πθ (at |st )
= Qt=1
T
t=1 π̄(at |st )

3.8.2 Deriving Policy Gradient with Importance Sampling

It turns out that we can recover the original policy gradient theorem using off-policy learning
using importance sampling. Recall the objective of RL as defined in the first chapter:

θ∗ = arg max J(θ)

θ

and we defined J(θ) as J(θ) = Eτ ∼πθ (τ ) [r(τ )]. Now if we want to estimate J with some new
parameter θ0 , we can use importance sampling as discussed above:
 
0 πθ0 (τ )
J(θ ) = Eτ ∼πθ (τ ) r(τ )
πθ (τ )

then we take the gradient as:

   
0 ∇θ0 πθ0 (τ ) πθ0 (τ )
∇θ0 J(θ ) = Eτ ∼πθ (τ ) r(τ ) = Eτ ∼πθ (τ ) ∇θ0 log πθ0 (τ )r(τ )
πθ (τ ) πθ (τ )

Now if we estimate it locally, by setting θ = θ0 , then we will cancel out the importance
ratio, ending up with Eτ ∼πθ (τ ) [∇θ0 log πθ0 (τ )r(τ )].

3.9 First Order Approximation for Importance Sampling

Now we focus on the cases where we do not use local approximation, when θ 6= θ0 .

J(θ0 ) = Eτ ∼πθ (τ ) [r(τ )]

 
0 πθ0 (τ )
∇θ0 J(θ ) = Eτ ∼πθ (τ ) ∇θ0 log πθ0 (τ )r(τ )
πθ (τ )
" Q ! T ! T !#
T
π θ 0 (at |st ) X X
= Eτ ∼πθ (τ ) Qt=1T
∇θ0 log πθ0 (at |st ) r(st , at )
t=1 πθ (at |st ) t=1 t=1
CHAPTER 3. POLICY GRADIENT METHODS 15

Now there is a problem in the equation. Note that the ratio of the two products can be very
small or very big if T is big, thus increasing the variance. To alleviate the issue, one can
make use of causality as we discussed before:
" Q ! T ! T !#
T
t=1 πθ0 (at |st )
X X
0
∇θ0 J(θ ) = Eτ ∼πθ (τ ) QT ∇θ0 log πθ0 (at |st ) r(st , at )
t=1 πθ (at |st ) t=1 t=1
t0
" T t
! T !!#
X Y πθ0 (at0 |st0 ) X Y πθ0 (at00 |st00 )
= Eτ ∼πθ (τ ) ∇θ0 log πθ0 (at |st ) r(st0 , at0 )
t=1 t0 =1
πθ (at0 |st0 ) t0 =t t00 =t
πθ (at00 |st00 )

Here we used the fact of causality that future actions don’t affect the current weight. Also
note that the last ratio of products can be deleted, and we essentially get the policy iteration
algorithm, which we will discuss in later chapters.
So when we delete the last weight, we end up having
" T t
! T !#
X Y π θ 0 (a t 0 |s t0 ) X
∇θ0 J(θ0 ) = Eτ ∼πθ (τ ) ∇θ0 log πθ0 (at |st ) r(st0 , at0 )
t=1 0
t =1
π θ (a t 0 |s t 0)
0
t =t

The product of ratio is again exponential in T , so we may have high variance.

Recall on-policy policy gradient:
T T
1 XX
∇θ J(θ) ' ∇θ log πθ (ai,t |si,t )Q̂i,t
N i=1 t=1

Similarly in off-policy policy gradient:

T T
01 X X πθ0 (si,t , ai,t )
∇ J(θ ) =
θ0 ∇θ0 log πθ0 (ai,t |si,t )Q̂i,t
N i=1 t=1 πθ (si,t , ai,t )
T T
1 X X πθ0 (si,t ) πθ0 (si,t |ai,t )
= ∇θ0 log πθ0 (ai,t |si,t )Q̂i,t
N i=1 t=1 πθ (si,t ) πθ (si,t |ai,t )

In later chapters, we can see that we can pretty much ignore the first states priors ratio.
3.9.1 Advanced Policy Gradients
Recall the policy gradients update rule:

θ ← θ + α∇θ J(θ)

In many cases, some parameters have more impact on the outcome than others. Therefore,
intuitively, we would like to set higher learning rate for parameters with less impact and lower
learning rate for parameters with more impact. To do this, we leverage covariant/natural
policy gradient. Let us look at at the constrained view of iterative gradient descent:

θ0 ← arg max(θ0 − θ)T ∇θ J(θ)

θ0 (3.2)
s.t. ||θ0 − θ||2 ≤ 
CHAPTER 3. POLICY GRADIENT METHODS 16

where this  controls how far we should go. But this  is defined in the parameters’ space,
which means that we do not have much control over individual parameters. To resolve this,
we would like to rescale this constraint so that we can constrain the step size in terms of
the policy space, thus giving us more control on individual parameters. For example, we can
use:

θ0 ← arg max(θ0 − θ)T ∇θ J(θ)

θ0 (3.3)
s.t. D(πθ0 − πθ ) ≤ 
where D(·, ·) is a parameterization-independent divergence measure, which usually is the
KL-divergence: DKL (πθ0 − πθ ) = Eπθ0 [log πθ − log πθ0 ].
We can also estimate the KL divergence locally using second-order Taylor expansion by:

DKL (πθ0 − πθ ) ≈ (θ0 − θ)T F (θ0 − θ)

where F is the Fisher-information matrix defined as:

F = Eπθ0 [∇θ log πθ (a|s)∇θ log πθ (a|s)T ]

Thus, with F , the rescaled constraint optimization problem can be equivalently rewritten
as:
θ0 ← arg max(θ0 − θ)T ∇θ J(θ)
θ0 (3.4)
s.t. ||θ0 − θ||2F ≤ 
Using Lagrangian, one could solve this optimization problem iteeratively as follows:

θ ← θ + αF −1 ∇θ J(θ)

This is the basic idea behind TRPO (PPO).

Chapter 4: Actor-Critic Algorithms

Recall from last chapter, we derived the policy gradient theorem:

N T
! T !
1 X X X
∇θ J(θ) ' ∇θ log πθ (ai,t |si,t ) r(si,t0 , ai,t0 )
N i=1 t=1 0
t =t

where we defined the summed reward as the “reward-to-go” function Q̂i,t , and it represents
the estimate of expected reward if we take action ai,t in state si,t . We have shown that this
estimate has very high variance, and we shall see how we can improve policy gradients from
using better estimation of the reward-to-go function.

4.1 Reward-to-Go
Let us take a closer look at the reward-to-go. To improve the estimation, one way is to
get closer to the precise value of the reward-to-go. We can define the reward-to-go using
expectation:
T
X
Q(st , at ) = Ep(θ) [r(st0 , at0 )|st , at ]
t0 =t

this is the true, expected value of the reward-to-go.

Therefore, one could imagine using this true expected value, combined with our original
Monte Carlo approximation to yield a better estimate:
N T
1 XX
∇θ J(θ) ' ∇θ log πθ (ai,t |si,t )Q(si,t , ai,t )
N i=1 t=1
.

4.2 Using Baselines

As we have seen in last chapter, one can reduce the high variance of the policy gradient
using baselines. We have also seen that it is possible to calculate the optimal baseline value
to yield the minimum variance, although people often use the average reward for sake of
simplicity.
Motivated by this, let us recall the definition of the value function (defined in the
introduction section):
V (st ) = Eat ∼πθ (at |st ) [Q(st , at )]
By definition, the value function is the average of Q-function value.

17
CHAPTER 4. ACTOR-CRITIC ALGORITHMS 18

Similarly, we can use the average reward-to-go as a baseline to reduce the variance.
Specifically, we could use the value function V (st ) as the baseline, thus improving the esti-
mate of the gradient in the following way:
N T
1 XX
∇θ J(θ) ' ∇θ log πθ (ai,t |si,t ) (Q(si,t , ai,t ) − V (si,t ))
N i=1 t=1

and
1
P the value function we used is a better approximation of the baseline bt =
N i Q(si,t , ai,t ).
What have we done here? What is the intuition behind subtracting the value function
from the Q-function? Essentially, we are quantifying how much an action ai,t is better than
the average actions. In some sense, it measures the advantage of applying an action over
the average action. Therefore, to formalize our intuition, let us define the advantage as
follows:
Aπ (st , at ) = Qπ (st , at ) − V π (st )
which quantitatively measures how much better action at is.
Putting it all together, now a better baseline-backed policy gradient estimate using
Monte Carlo estimate can be written as:
N T
1 XX
∇θ J(θ) ' ∇θ log πθ (ai,t |si,t )Aπ (si,t , ai,t )
N i=1 t=1
.

4.3 Value Function Fitting

The better the estimate of the advantage function, the lower the variance, and we can have
better policy gradient. Let us massage the definition of the Q-function a little in order to
find some interesting mathematical relations between Q and V :
T
X
π
Q (st , at ) = Eπθ [r(st0 , at0 )|st , at ]
t0 =t
T
X
= r(st , at ) + Eπθ [r(st0 , at0 )|st , at ]
t0 =t+1

= r(st , at ) + V π (st+1 )
= r(st , at ) + Est+1 ∼p(st+1 |st ,at ) [V π (st+1 )]

The last expectation of the value function is used because we do not know what the next
state actually is. Note that we can be a little crude with respect to that expectation in such
a way that we just use the full value function V π (·) on one single sample of the next state,
and use the value as the expectation, ignoring the fact that there are multiple other next
states. With this estimate, we can plug into the advantage function:

Aπ (st , at ) ' r(st , at ) + V π (st+1 ) − V π (st )

CHAPTER 4. ACTOR-CRITIC ALGORITHMS 19

Figure 4.1: Fitting the value function

Therefore, it is almost enough to just approximate the value function, which solely
depends on state, to generate approximations of other functions. To achieve this, we can use
a neural network to fit our value function V (s), and use the fit value function to approximate
our policy gradient, as illustrated in Fig. 4.1

4.4 Policy Evaluation

Here in this section, we discuss the process and purpose of fitting the value function.
4.4.1 Why Do We Evaluate a Policy
Policy evaluation is a process that given a fixed policy π, we figure out how good it is by
fitting the value function V π (·) by using this expectation:
T
X
π
V (st ) = Eπθ [r(st0 , at0 )|st ]
t0 =t

Having the value function allows us to figure out how good the policy is because the rein-
forcement learning objective can be equivalently written as J(θ) = Es1 ∼p(s1 ) [V π (s1 )], where
we take the expectation of the value function value of the initial state over all possible initial
states.
4.4.2 How to Evaluate a Policy
To evaluate a policy, we can use an approach similar to the policy gradient - Monte Carlo
approximation. Specifically, we can estimate the value function by summing up the reward
collected from time step t:
XT
π
V (st ) ' r(st0 , at0 )
t0 =t

and if we are able to reset the simulator, we could indeed ameliorate this estimate by taking
multiple samples (N ) as follows:
N T
1 XX
V π (st ) ' r(si,t0 , ai,t0 )
N i=1 t0 =t

In practice, we can just use the single sample approximation.

Here is a question, if our original objective is to use V π to reduce the variance, but
we end up using a single sample estimation to estimate V π , does it actually help? The
CHAPTER 4. ACTOR-CRITIC ALGORITHMS 20

answer is yes, because we are using a neural net to fit the Monte Carlo targets from a variety
of different states, so even though we do single sample estimate, the value function does
generalize when we visit similar states.
4.4.3 Monte Carlo Evaluation with Function Approximation
To fit our value function, we could use a supervised learning approach. Essentially, we
can use our single sample estimation of the value function as our function value, and fit
a function n thatPmaps the statesoto the value function values. Therefore, our training data
will be (si,t , Tt0 =t r(si,t0 , ai,t0 )) , and we denote the function value labels as yi,t , and we
define
P aπ typical supervised regression loss function which we try to minimize as L(φ) =
1 2
2 i ||V̂φ (si ) − yi || .

4.4.4 Improving the Estimate Using Bootstrap

In fact, we can improve our training process because the original applied target yi,t is not
perfect. We could use a technique called bootstrapping. Recall the definition of our ideal
target in the supervised regression:
T
X
yi,t = Eπθ [r(st0 , at0 )|si,t ]
t0 =t
T
X
' r(si,t , ai,t ) + [r(st0 , at0 )|si,t+1 ]
t0 =t+1

' r(si,t , ai,t ) + V π (si,t+1 )

, compared with our Monte Carlo targets: yi,t = Tt0 =t r(si,t0 , ai,t0 ).
P
Bootstrapping means applying our previous estimation on our current estimation. In
our ideal targets, the last estimation is accurate if we knew the actual V π . But if the actual
value function is not known, we can just apply bootstrapping by using the current fit estimate
V̂φπ to estimate the next state’s value: V̂φπ (si,t+1 ). Such an estimate is biased, but it has low
variance.
Consequently, our training data using bootstrapping becomes:
n o
(si,t , r(si,t , ai,t ) + V̂φπ (si,t+1 ))

. Such bootstrapped targets work well with highly stochastic environments.

4.5 Batch Actor-Critic Algorithm

Now we are ready to devise our first actor-critic algorithm. The reason why we call it
actor-critic is that we use a critic (value function) to decrease the high variance of the actor
(Q-function/policy). The full algorithm is shown in Alg. 3 and we call it a batch algorithm
because it is not online. We shall see the online version later. In Algorithm 3, the way how
1 π
we fit V̂φ is by minimizing the supervised regression norm L(φ) = 2 i ||V̂φ (si ) − yi ||2 .
P
CHAPTER 4. ACTOR-CRITIC ALGORITHMS 21

Algorithm 3 Batch Actor-Critic Algorithm

Require: Base policy πθ (at |st )
1: while true do
2: Sample {si , ai } from πθ (a|s) (run it on a robot)
3: Fit V̂φ (s) to sampled reward sums
4: Evaluate Âπ (si , ai ) = r(si , ai ) + V̂φ (s0i ) − V̂φ (si )
∇θ J(θ) ' i ∇θ log πθ (ai |si )Âπ (si , ai )
P
5:
6: Improve policy by θ ← θ + α∇θ J(θ)
7: return optimal policy from gradient ascent as π return

4.6 Aside: Discount Factors

Imagine if we had an infinite horizon environment (T → ∞), then our estimated value
function V̂φπ (s) can get infinitely large in many cases. Therefore, one possible way to address
this issue is to say that it is better to get rewards sooner than later. Therefore, instead of
labeling our values as yi,t ' r(si,t , ai,t ) + V̂φπ (si,t+1 ), we can shrink the value function value
as we progress to the next time step. To achieve this, we introduce a hyperparameter called
a discount factor, denoted as γ, where γ ∈ [0, 1]:
yi,t ' r(si,t , ai,t ) + γ · V̂φπ (si,t+1 )
in most cases, γ = 0.99 works well.
Let us apply the discount factor to policy gradients. Basically, we have two options to
impose this discount factor. The first option is :
N T
! T !
1 X X X 0
∇θ J(θ) ' ∇θ log πθ (ai,t |si,t ) γ t −t r(si,t0 , ai,t0 )
N i=1 t=1 t0 =t

and the second option is:

N T
! T !
1 X X X
∇θ J(θ) ' ∇θ log πθ (ai,t |si,t ) γ t−1 r(si,t , ai,t )
N i=1 t=1 t=1
N T
! T !
1 X X X 0
' ∇θ log πθ (ai,t |si,t ) γ t −1 r(si,t0 , ai,t0 ) (causality)
N i=1 t=1 t0 =t
N T
! T
!
1 X X X 0
' γ t−1 ∇θ log πθ (ai,t |si,t ) γ t −t r(si,t0 , ai,t0 )
N i=1 t=1 t0 =t

Intuitively, the second option assigns less weight to later step’s gradient, so it essentially
means that later steps matter less in our discount.
In practice, we can show that option 1 gives us better variance, so it is actually what
we use. The full derivation can be found in this paper [3]. Now in our actor-critic algorithm,
after we impose the discount factor, we have the following gradient:
N T
!
1 X X  
∇θ J(θ) ' ∇θ log πθ (ai,t |si,t ) r(si,t , ai,t ) + γ V̂φπ (si,t+1 ) − V̂φπ (si,t )
N i=1 t=1
CHAPTER 4. ACTOR-CRITIC ALGORITHMS 22

Algorithm 4 Batch Actor-Critic Algorithm with Discount Factor

Require: Base policy πθ (at |st ), hyperparameter γ
1: while true do
2: Sample {si , ai } from πθ (a|s) (run it on a robot)
3: Fit V̂φ (s) to sampled reward sums
4: Evaluate Âπ (si , ai ) = r(si , ai ) + γ V̂φ (s0i ) − V̂φ (si )
∇θ J(θ) ' i ∇θ log πθ (ai |si )Âπ (si , ai )
P
5:
6: Improve policy by θ ← θ + α∇θ J(θ)
7: return optimal policy from gradient ascent as π return

Algorithm 5 Online Actor-Critic Algorithm

Require: Base policy πθ (at |st ), hyperparameter γ
1: while true do
2: Take action a ∼ πθ (a|s), get (s, a, s0 , r)
3: Update V̂φπ using target r + γ V̂φπ (s0 )
4: Evaluate Âπ (s, a) = r(s, a) + γ V̂φπ (s0 ) − V̂φπ (s)
5: ∇θ J(θ) ' ∇θ log πθ (a|s)Âπ (s, a)
6: Improve policy by θ ← θ + α∇θ J(θ)
7: return optimal policy from gradient ascent as π return

Now we can incorporate the discount factor with our actor-critic algorithm in Algorithm
4.

4.7 Online Actor-Critic Algorithm

Now that we have seen actor-critic algorithms with a batch of samples, we can further
improve the performance by making it fully online. Namely, we are taking the gradient step
based on the current sample so that we are not storing any large number of samples, which
is more efficient. In the online version of actor-critic, we essentially use two neural nets: one
for the policy, the other one for the value function. This is simple and stable, but as the
states dimension becomes higher, we are not giving any shared features between the actor
and the critic. Therefore, we can also make the network shared between the policy and the
value function. For example, in image-based observations scenarios, we could share the conv
layers’ weights for the two networks and only differ the two in the final fully connected layers.
In each step, we can only take one sample and gradually improve our value function using
that sample. Here is the sketch of the online version of actor-critic algorithm in Algorithm
5. Note that in steps 3-5, we are only taking a gradient step from one sample. In reality,
this works best if we use a batch of samples instead of just one, and one can use parallel
workers (simulations) either synchronously or asynchronously to achieve it, as illustrated in
Fig 4.2.
One caveat about the asynchronous version is that as the parameter server gets updated,
the data collection policy might not be updated, which means the newly collected data might
not come from the latest policy, thus making the current acting policy slightly outdated. Such
problem is less of an issue in practice because the policy only gets updated by a tiny bit
CHAPTER 4. ACTOR-CRITIC ALGORITHMS 23

Figure 4.2: Parallel simulations for online actor-critic

every time.

4.8 Critics as State-Dependent Baselines

Now let us further discuss the connection between a baseline and a critic. Recall in the
Monte Carlo version of policy gradient, the gradient is defined as:
N T
! T !
1 X X X
∇θ J(θ) ' ∇θ log πθ (ai,t |si,t ) r(si,t0 , ai,t0 ) − b
N i=1 t=1 0
t =1

and in actor-critic algorithm, we estimate the gradient by estimating the advantage function:
N T
!
1 X X  
∇θ J(θ) ' ∇θ log πθ (ai,t |si,t ) r(si,t , ai,t ) + γ V̂φπ (si,t+1 ) − V̂φπ (si,t )
N i=1 t=1

So what are the pros and cons of the two approaches? In policy gradient with baselines,
we have shown that there is no bias in our estimation, but there might be high variance
due to our single-sample estimation of the cost-to-go function. On the other hand, in the
actor-critic algorithm, we have shown that we have lower variance due to the critic, but we
end up having a biased estimation because of the possibly bad critic as we are bootstrapping.
So can we somehow keep the estimator unbiased while lowering the variance with the critic
V̂φπ ?
The solution is obvious and straightforward, we can just use V̂φπ in place of b:
N T
! T !
1 X X X
∇θ J(θ) ' ∇θ log πθ (ai,t |si,t ) r(si,t0 , ai,t0 ) − V̂φπ (si,t )
N i=1 t=1 t0 =1

In this way, we obtain an unbiased estimator with lower variance.

4.9 Eligibility Traces and n-Step Returns

In the above comparison of the two methods, we have seen that in the actor-critic advantage
function, we have lower variance but higher bias, while in the Monte Carlo policy gradient,
CHAPTER 4. ACTOR-CRITIC ALGORITHMS 24

the advantage function has lower bias but higher variance. The reason why this tradeoff
exists is that as we go further in our trajectory into the future, the variance increases due to
the fact that the current single sample approximation is not representative enough for the
future. Therefore, the Monte Carlo advantage function is good for getting accurate values
in the near term, but not the long term. In contrast, in actor-critic advantage, the bias
potentially skews the values in the near term, but the fact that the bias incorporates a lot
of states will likely make it a better approximator in the long run. Therefore, it would be
better if we could use the actor-critic based advantage for further in the future, and use the
Monte Carlo based one for the near term in order to control the bias-variance tradeoff.
As a result, we can cut the trajectory before the variance gets too big. Mathematically,
we can estimate the advantage function by combining the two approaches: use the Monte
Carlo approach only for the first n steps:
t+n
0
X
Âπn (st , at ) = γ t −t r(st0 , at0 ) − V̂φπ (st ) + γ n V̂φπ (st+n )
t0 =t

here we applied an n-step estimator, which sums the reward from now to n steps from now,
and n > 1 often gives us better performance.
Furthermore, if we don’t want to choose just one n, we can use a weighted combination
of different n-steps returns, which we can define as the General Advantage Estimation(GAE):
∞
X
ÂGAE (st , at ) = wn Âπn (st , at )
n=1

To choose the weights, we should prefer cutting earlier, so we can assign the weights accord-
ingly: wn ∝ λn−1 , where we call λ the chance of getting cut.
Chapter 5: Value Function Methods

In the last two chapters, we discussed some policy gradient-based algorithms. We have also
seen the fact that the policy gradient methods have high variance. Therefore, it would be
nice if we could completely omit the gradient step. To achieve this, we are going to talk
about the value function methods in this chapter.

5.1 An Implicit Policy

To omit the policy gradient, one still has to generate a policy function so that it takes in a
state and outputs an action. Recall in actor-critic algorithms, we use the advantage function
Aπ (st , at ) to gauge how much better is the action at than the average action according to
π. Provided that we have a somehow accurate representation of this advantage function, we
can just forget about generating a policy π, and just do this:
arg max Aπ (st , at )
at

which means we take the best action from st , if we follow π. Even though we have no
knowledge of what the policy π actually is, by doing the arg max, we can guarantee that
the action produced is at least as good as the action from the policy function that we do
not know. Therefore, as long as we have a accurate representation of the advantage function
Aπ (st , at ), we can implicitly generate a parameter-free policy function:
(
1, if at = arg maxat Aπ (st , at )
π 0 (at |st ) =
0, otherwise
and as we have shown above, this implicit policy is at least as good as the unknown policy
π.

5.2 Policy Iteration

Having omitted the policy, we can then proceed to introduce the policy iteration algorithm.
5.2.1 High Level Idea
The basic idea of policy iteration algorithms is very simple: we evaluate the advantage
function Aπ (s, a) and then update the policy using the update rule as defined above in the
implicit policy, and then we can loop to constantly improve our policy.
The problem here is how to evaluate Aπ (s, a). In other words, how does one find an
accurate representation of the advantage function in order to accurately update the policy.
As before, we have seen that the advantage function an be equivalently defined as follows:
Aπ (s, a) = r(s, a) + γE [V π (s0 )] − V π (s)

25
CHAPTER 5. VALUE FUNCTION METHODS 26

Algorithm 6 Policy Iteration via DP

1: while true do
2: Evaluate V π (s, a)
3: Set π ← π 0

Algorithm 7 Value Iteration via DP

1: while true do
2: Set Q(s, a) ← r(s, a) + γE [V (s0 )]
3: Set V (s) ← maxa Q(s, a)

Therefore, if we can evaluate the value function V π (s) then we can also evaluate Aπ (s, a).
So in the high-level policy iteration algorithm, we can just use the value function in place of
the advantage function.
5.2.2 Dynamic Programming
Now let us make a simple assumption. Suppose we know a priori the transition probabil-
ity p(s0 |s, a) and both states s and action a are discrete. Then a very natural dynamic
programming update is the bootstrapped update, as we have seen before:

V π (s) ← Ea∼π(a|s) r(s, a) + γEs0 ∼p(s0 |s,a) [V π (s0 )]

 

and we can just use the current estimate inside the nested expectation for simplicity.
According to our definition of the implicit policy function π 0 , the policy is actually
deterministic. Therefore, we can completely get rid of the outside expectation, and the value
function update can be further simplified as:

V π (s) ← r(s, π(s)) + γEs0 ∼p(s0 |s,a) [V π (s0 )]

This version of policy iteration is sketched in Algorithm 6

We can even further simplify the dynamic programming update. Note that in each
iteration, we are updating the policy first in order to update the value function. Thus, it
would be faster if we could skip the policy part and directly improve the value estimation.
Meanwhile, the arg max operation that we apply on the advantage function itself is an
implicit policy. We also know that arg maxat Aπ (s, a) = arg maxat Qπ (s, a), because the two
values only differ by the subtraction term V π , which does not depend on action:

Aπ (s, a) = r(s, a) + γE [V π (s0 )]

Having this, we can simplify the policy iteration algorithm further, as illustrated in Alg. 7.
As we skipped the policy update part, we call this new, simplified algorithm “value iteration
algorithm”.

5.3 Fitted Value Iteration

The policy iteration and value iteration algorithm we discussed above are heavily based on an
impractical assumption: the total number of states is finite and small, because we are trying
CHAPTER 5. VALUE FUNCTION METHODS 27

Algorithm 8 Fitted Value Iteration

1: while true do
2: set yi ← maxai (r(si , ai ) + γE [Vφ (s0i )])
3: set φ ← arg minφ 21 Σi ||Vφ (si ) − yi ||2

Algorithm 9 Fitted Q-Iteration Algorithm

Require: Some base policy for data collection; hyperparameter K
1: while true do
2: Collect dataset {(si , ai , s0i , ri )} using some policy
3: for K times do
4: Set yi ← r(si , ai ) + γ maxa0i Qφ (s0i , a0i )
5: Set φ ← arg minφ 12 Σi ||Qφ (si , ai ) − yi ||2

to construct a tabular expression of the value function and the Q function. Apparently, the
tables are going to explode in dimensions if there are a lot of states. We call this the Curse
of Dimensionality. To resolve this problem, we can use a neural network to approximate the
functions instead of constructing a tabular expression of the function.
5.3.1 Fitted Supervised Value Iteration Algorithm
Since we know that the value function is defined as maxa Qπ (s, a), we can use this definition
as the labels for the value function in order to define an L2 loss function:
1
L(φ) = ||Vφ (s) − max Qπ (s, a)||2
2 a

Then we can sketch out a simple fitted value iteration algorithm using this loss function
in Algorithm 8. Note that when setting the label, the ideal step to take is to enumerate
all the states and find the corresponding label. However, when it is impractical, one could
just use some samples and enumerate all the actions to find the labels. Moreover, when we
take the maximum over all the actions from a state, we implicitly assume that the transition
dynamics are known. Why? Because we want to take an action, record the value of that
action, and then roll back to the previous state in order to check the values of other actions.
Thus, without the transition dynamics, we cannot easily take the maximum.
5.3.2 Fitted Q-Iteration Algorithm
To address this problem, we can apply the same “max” trick in policy iteration. In policy
iteration, we skip the policy update and calculate the values directly. Here in fitted value
iteration, we can get around the transition dynamics by looking up the Q function table,
because Vφ (s) ' maxa Qφ (s, a), and this max operation is merely a table lookup from the Q
value table. Consequently, we are now iterating on the Q values. Such a method works for
off-policy samples (unlike actor-critic), and it only needs one network, so it does not have any
high-variance policy gradient. However, as we shall see in later sections, such methods do not
have convergence guarantees on non-linear functions, which could potentially be problematic.
The full fitted Q-iteration algorithm is shown in Algorithm 9.
CHAPTER 5. VALUE FUNCTION METHODS 28

Algorithm 10 Online Q-Iteration Algorithm

1: while true do
2: Take some action ai and observe (si , ai , s0i , ri )
3: yi = r(si , ai ) + γ maxa0 Qφ (si , ai , s0i , ri )
dQ
4: φ ← φ − α dφφ (si , ai )(Qφ (si , ai ) − yi )

5.3.3 A Closer Look at Q-Iteration Algorithm

Let us take a closer look at the fitted Q-learning algorithm. First, let us discuss why the
algorithm is fully off-policy. In step 2 of Alg. 9, we are not collecting a lot of transition
data, and we do not care about the trajectories. Furthermore, in step 4, we are taking the
step off-policy in that we do not care about which state we are going to, because we only
care about the value of the transition. In other words, the tansition we take is independent
of the unknown policy π. Therefore, the fitted Q-iteration algorithm is fully off-policy.
Another question we can ask is that what is fitted Q-iteration actually optimizing? In
step 5 of Alg. 9, we are minimizing the difference between the Q function value and the
label we approximated. In fact, we call this difference the Bellman Error, defined as follows:
 i2 
1 h
0 0
 = E(s,a)∼β Qφ (s, a) − r(s, a) + γ max Qφ (s , a )
2 a0

So in this particular step, we are optimizing the Bellman Error, and if  = 0, we have opti-
mal Q-function, corresponding to optimal policy π, which can be recovered by the arg max
operation. However, rather ironically, we do not know what we are optimizing in the pre-
vious steps, and this is a potential problem of the fitted Q-learning algorithm, and most
convergence guarantees are lost when we do not have the tabular case.
5.3.4 Online Q-Iteration Algorithm
We can also make the samples more efficient by making the Q-iteration algorithm completely
online. By online we mean that we do not store any transition. Instead, we take one transition
and immediately apply the transition to our value update. The online version of Q-Iteration
Algorithm is sketched in Alg. 10. As we see in step 2 of the algorithm, we are taking an
action off-policy, so we have a lot of choices to make.

5.4 Value Function Learning Theory

One question that one might ask after seeing the variety of algorithm as shown above is does
the value function method converge? If so, it converges to what? To take a closer look in
order to answer the question, let us define a Bellman backup operator B:
BV = max ra + γTa V
a

where ra is a stacked vector of rewards at all states for action a. Ta is a matrix of transitions
for action a such that Ta,i,j = p(s0 = i|s = j, a). We also define a fixed point of the Bellman
backup operator B, denoted as V ∗ :
V ∗ (s) = max r(s, a) + γE[V ∗ (s0 )]
a
CHAPTER 5. VALUE FUNCTION METHODS 29

Figure 5.1: Bellman Backup Projection

so it is similar to the notion of the stationary distribution in MDP, V ∗ = BV ∗ . One can

prove that such fixed point always exists, and it corresponds to the optimal policy, but the
question is: will we reach it?
In the tabular representation case, we can prove that value iteration always reaches
the fixed point V ∗ because mathematically, the Bellman backup operator is a contraction.
A contraction in our scenario is defined as follows: for any V and V̄ , we have ||BV −B V̄ ||∞ ≤
||V − V̄ ||∞ . In other words, after applying the Bellman backup operator, the gap always
gets smaller by γ with respect to the l-∞ norm.
Now let us proceed to analyze the non-tabular representation case. In this scenario,
unfortunately, we have lost a lot of convergence guarantees. Recall that in normal value
iteration (tabular case), we use the Bellman backup operator B to update V : V ← BV . In
fitted (non-tabular) value iteration, we use the Bellman backup operator,
1
P B, 0together with
another operator Π. The operator Π is defined as: ΠV = arg minV 0 ∈Ω 2 ||V (s) − V (s)||2 .
So Π is a projection onto Ω in terms of l2 norm. This projection is illustrated in Fig. 5.1. The
set that V and V 0 lie in can be thought as a representation of all value functions. Therefore,
the set Ω can be represented by neural networks.
Now we have the two operators defined, we can see that B is a contraction with respect
to the l-∞ norm, and the operator Π is a contraction with respect to the l2 norm. But
what if we impose one operator to another? Is the compound operator also a contraction?
The answer is no. Therefore, such non-tabular Q-iterations do not have any convergence
guarantee as the operator is not a contraction.
What about fitted Q-iteration? Concisely, the fitted Q-iteration algorithm can be de-
fined as Q ← ΠBQ. Therefore, the same reasoning can be applied to the fitted Q-learning:
since the compound operator is no longer a contraction, we do not have any guarantee for
convergence. We can say the same thing in online Q-iteration as well.
However, one might ask, in step 4 of Alg. 10, aren’t we just doing gradient descent,
which definitely converges? As a matter of fact, this is not real gradient descent in that the
target value is constantly changing due to the off-policy nature of this algorithm. So we have
this sad corollary: in general cases, fitted bootstrapped policy evaluation does not converge.
Chapter 6: Q-Function Methods

Recall the algorithms that we discussed in last chapter: Alg. 9 and Alg. 10, where we
devised a fitted Q-learning algorithm and a fully online version of it. We have also shown
that Q-learning is fully off-policy, meaning that we do not care about the trajectory we are
taking, we only care about the current transition and the next state we land in. So what is
the problem with the above Q-learning algorithms? To see this, let us carefully look at step
4 of Alg. 10. The gradient step that we are taking is equivalently written as:
dQφ  h
0
i
φ←φ−α (si , ai ) Qφ (si , ai ) − r(si , ai ) + γ max Qφ (s i , ai , si , ri )
dφ a0

This is not gradient descent! Because the “target” value yi is not constant and depends on
our parameter φ, and we are not taking gradient from yi . Therefore, this is not the gradient
descent step that we used to see before. Moreover, in step 2, we are only taking one sample
of transition. This sampling scheme brings us two problems: the first one is that one sample
is really hard to train the network (recall in online actor-critic, we would use parallel workers
to obtain multiple online samples), and the second problem is the samples we are drawing are
correlated in that the transitions are dependent on each other. As you may know, Stochastic
Gradient Descent converges only if we are taking the correct gradient, and when the samples
are IID. We violated both requirements, so the Q-learning algorithms in Alg. 9 and Alg. 10
do not have any convergence guarantees.

6.1 Replay Buffers

One of the solutions that we can implement to resolve the above correlation issue is to use
a replay buffer. A replay buffer B is a buffer of transition data that contains the single
samples that we have drawn so far. Therefore, in Q-learning, if we sample a batch from B,
the samples are no longer correlated, and we also keep updating the replay buffer by adding
in real-world transitions. One can view this interaction in the image illustrated in Fig. 6.1.

Figure 6.1: Q-learning with replay buffers

30
CHAPTER 6. Q-FUNCTION METHODS 31

Algorithm 11 Q-Learning with Replay Buffer

Require: Some base policy for data collection; hyperparameter K
1: while true do
2: Collect dataset {(si , ai , s0i , ri )} using some policy, add it to replay buffer B
3: for K times do
4: Sample a batch (si , ai , s0i , ri ) from B
5: Set yi ← r(si , ai ) + γ maxa0i Qφ (s0i , a0i )
dQ
6: Set φ ← φ − αΣi dφφ (si , ai )(Qφ (si , ai ) − yi )

Algorithm 12 Q-Learning with Replay Buffer and Target Network

Require: Some base policy for data collection; hyperparameter K and N
1: while true do
2: Save target network parameters: φ0 ← φ
3: for N times do
4: Collect dataset {(si , ai , s0i , ri )} using some policy, add it to replay buffer B
5: for K times do
6: Sample a batch (si , ai , s0i , ri ) from B
7: Set yi ← r(si , ai ) + γ maxa0i Qφ0 (s0i , a0i )
dQ
8: Set φ ← φ − αΣi dφφ (si , ai )(Qφ (si , ai ) − yi )

Putting it all together, we sketch out the full Q-learning algorithm with replay buffer in
Alg. 11. What have changed here in Alg. 11 compared with Alg. 9? In step 2, we are not
only collecting dataset, but also adding the dataset to replay buffer B. Inside the for loop,
we are now sampling a batch of transitions from B, which will bring us lower variance when
we take the gradient step on the batch. We also periodically update B.
The above solution solves the correlation problem, but we still need to address the wrong
gradient problem.

6.2 Target Networks

To solve the wrong gradient problem, we are essentially trying to solve the problem that
the target value yi is heavily dependent on our gradient parameter φ. Therefore, one way to
improve this is to use a separate Q-function with another parameter φ0 in order to decorrelate
the two values. We should also use a well-defined φ0 , which can be set as the φ parameter
1000 steps ago. When setting yi , we set it from the Q function with another parameter, thus
decorrelating the two values. Together with the use of replay buffer, we alleviated the wrong
policy and the correlation samples problems, as shown in Alg. 12. Here we are frequently
sampling batches from B and taking the gradient steps. We are less frequently updating
the replay buffer as discussed in Alg. 11. We are even less frequently updating the network
parameters, since we said one good choice for parameter φ0 is to use φ 1000 steps ago.
Using target networks and replay buffers, we can sketch out a classic deep Q-learning
algorithm (DQN), proposed by Minh et al. in [4]. The pseudocode is in 13. Here we choose
N to be a large number because our intention is to infrequently update the parameters.
To further optimize the algorithm, it might feel weird to abruptly update φ0 after N steps.
CHAPTER 6. Q-FUNCTION METHODS 32

Algorithm 13 Classic Deep Q-Learning Algorithm (DQN)

1: while true do
2: Take some action ai and observe (si , ai , s0i , ri ) and add it to B
3: Sample mini-batch {sj , aj , s0j , rj }
4: Compute yj = rj + γ maxa0j Qφ0 (s0j , a0j ) using target network Qφ0
dQ
5: φ ← φ − αΣj dφφ (sj , aj )(Qφ (sj , aj ) − yJ )
6: update φ0 : copy φ every N steps.

Figure 6.2: Q-learning in a more general view.

Therefore, to alleviate this “abruptness”, we can use Polyak Averaging: in step 6 of Alg.
13, instead of copying φ every N steps, we do φ0 ← τ φ0 + (1 − τ )φ. We also call such update
a damped update.
Now let us view the three different Q-learning algorithms in a more general way. As
shown in Fig. 6.2, there are three different steps in the algorithm. The first step is to collect
data, the second step is to update the target in the target network, and the third step is to
regress onto the Q-function. In the simplest, regression-based fitted Q-learning algorithm,
process 3 is in the inner loop of process 2, which is in the inner loop of process 1. In online
Q-learning, we evict the old transitions immediately, and process 1, 2, and 3 run at the same
speed. In DQN, process 1 and 3 run at the same speed, but process 2 runs slower.

6.3 Inaccuracy in Q-Learning

Q-values are not necessarily accurate. The reason lies in the target value. Recall that the
target value y is defined as yj = rj + γ maxa0j Qφ0 (s0j , a0j ). The max operation in the tar-
get is the main problem, because for two random variables X1 and X2 , E[max(X1 , X2 )] ≥
max(E[X1 ], E[X2 ]). Therefore, when Qφ0 (s0j , a0j ) is noisy, the max operation is going to over-
estimate the next Q-value.
6.3.1 Double Q-Learning
One might notice that maxa0 Qφ0 (s0 , a0 ) = Qφ0 (s0 , arg maxa0 Qφ0 (s0 , a0 )). Thus, if we somehow
managed to decorrelate the error from the selected action and the error from the Q-function,
we could eliminate the erroneous overestimation. To achieve this, we can use two different
CHAPTER 6. Q-FUNCTION METHODS 33

networks to choose actions and evaluate the Q-function values.

 
0 0 0
QφA (s, a) ← r + γQφB s , arg max QφA (s , a )
a0
 
0 0 0
QφB (s, a) ← r + γQφA s , arg max QφB (s , a )
a0

Essentially we are using one network’s parameter to update the value, while using the other’s
to select the action. Using the two separate networks, we are decorrelating the action selec-
tion and value evaluation errors, thus decreasing the overestimation in the Q-values.
In practice, we can just use the actual and target networks for the two separate networks.
Therefore, instead of setting target y as y = r + γQφ0 (s0 , arg maxa0 Qφ0 (s0 , a0 )), we use the
current network to select action, and use the target network to evaluate value: y = r +
γQφ0 (s0 , arg maxa0 Qφ (s0 , a0 )).
6.3.2 N-Step Return Estimator
In the definition of our target y, yi,t = ri,t + maxai,t+1 Qφ0 (si,t+1 , ai,t+1 ), the Q-value only
matters if it is a good estimate. If the Q-value estimate is bad, the only values that matter
are from the reward term, so we are not learning much about the Q-function. To resolve
this problem, let us recall the N-step cut trick we did in the actor-critic algorithm. In actor-
critic algorithm, to leverage the bias and variance tradeoff in policy gradient, we can end the
trajectory earlier, and only count the reward summed up to N steps from now. Specifically,
we can define the target as:
X−1
t+N
0
yi,t = γ t−t ri,t + γ N max Qφ0 (si,t+1 , ai,t+1 )
ai,t+1
t0 =t

One subtle problem with this solution is that the learning process suddenly becomes on-
policy, so we cannot efficiently make use of the off-policy data. Why is it on-policy? If we
look at the summation of the rewards, we are collecting the rewards data using a certain
trajectory, which is generated by a specific policy. Therefore, we end up having less biased
target values when the Q-values are inaccurate, and in practice, it is faster in early stages
of learning. However, it is only correct when we are learning on-policy. To fix this problem,
one could ignore this mismatch, which somehow works very well in practice. Or one could
cut the trace by dynamically adapting N to get only on-policy data. Also, one could use
importance sampling as we discussed before. For more details, please refer to this paper by
Munos et al. [5].
6.3.3 Q-Learning with Continuous Actions
Recall the implicit policy that we define using Q-learning:
(
1, if at = arg maxat Aπ (st , at )
π 0 (at |st ) =
0, otherwise

One problem with this definition is that the arg max operation cannot be easily applied if
the actions are continuous. How are we going to address such an issue?
CHAPTER 6. Q-FUNCTION METHODS 34

Algorithm 14 Deep Deterministic Policy Gradient (DDPG)

1: while true do
2: Take some action ai and observe (si , ai , s0i , ri ) and add it to B
3: Sample mini-batch {sj , aj , s0j , rj }
4: Compute yj = rj + γQφ0 (s0j , a0j ) using target networks Qφ0 and µθ0
dQ
5: φ ← φ − αΣj dφφ (sj , aj )(Qφ (sj , aj ) − yJ )
dQ
θ ← θ + β j dµ
P
6: (s ) φ (sj , a)
dθ j da
7: update φ0 and θ0

One option is to use various optimization techniques, as one may have seen in UC
Berkeley’s EE 127. Specifically, one could use SGD on the action space to produce an optimal
at by solving an optimization problem. Another simple approach is to stochastically optimize
the Q-values by using some samples of the values from some pre-defined distribution (e.g.
uniform): maxa Q(s, a) ' max{Q(s, a1 ), ..., Q(s, aN )}. One could also improve the accuracy
by using some more sophisticated optimization techniques such as Cross-Entropy Method
(CEM).
Option no. 2 is to use function classes that are easy to maximize. For example, one
could use the Normalized Advantage Functions (NAF) proposed by Gu et al. in [6].
Another rather fancier option is to learn an approximate optimizer, which was originally
proposed by Lillicrap et al. in [7]. The idea of Deep Deterministic Policy Gradient (which
is actually a Q-learning in disguise) is to train another network µθ (s) such that µθ (s) '
arg maxa Qφ (s, a). To train the network, one can see that the optimization of Q-function with
dQ dQθ
respect to θ can be solved by θ ← arg maxθ Qφ (s, µθ (s)) because by chain rule, dθφ = da
dθ da
.
Then the new target becomes:

yj = rj + γQφ0 (s0j , µθ (s0j )) ' rj + γQφ0 (s0j , arg max Qφ0 (s0j , a0j ))
a0

The sketch of DDPG is in Alg. 14. In step 5, we are updating the Q-function, and in
step 6, we are updating the argmax-er. Therefore, DDPG is essentially DQN with a learned
argmax-er.
Chapter 7: Policy Gradients Theory and Advanced Pol-
icy Gradients

Why does Policy Gradient algorithm work? Recall our generic policy gradient algorithm: we
are essentially looping to constantly estimate the advantage function Âπ (st , at ) for the current
policy π, and then we use this estimate to improve the policy by taking a gradient step on the
policy parameter θ, as shown in Alg. 2. This is very similar to the policy iteration algorithm
that we discussed in last chapter; the idea of policy iteration is to constantly evaluate the
advantage function Aπ (s, a) and update the policy accordingly using the arg max implicit
policy. In this chapter, we are going to dive deeper into the policy gradient algorithm, and we
will show that the policy gradient algorithm can be reduced to our policy iteration algorithm,
which we will prove mathematically.

7.1 Policy Gradient as Policy Iteration

The sum of rewards of RL was defined as J(θ) = Eτ ∼pθ (τ ) [ t γ t r(st , at )], which is an ex-
P
pectation taken under the trajectory distribution. We claim that given a new parameter θ0 ,
J(θ0 ) − J(θ) = Eτ ∼pθ0 (τ ) [ t γ t Aπθ (st , at )], which is an expectation taken under the new pol-
P
icy’s trajectory distribution. The difference of the two sums of rewards is the improvement of
applying the new policy compared to using the old policy. We claim that this improvement
is equal to the expected value of the old policy’s advantage function value taken under the
new policy’s trajectory distribution. The proof is as follows:

J(θ0 ) − J(θ) = J(θ0 ) − Es0 ∼p(s0 ) [V πθ (s0 )]

= J(θ0 ) − Eτ ∼pθ0 (τ ) [V πθ (s0 )]
"∞ ∞
#
X X
= J(θ0 ) − Eτ ∼pθ0 (τ ) γ t V πθ (st ) − γ t V πθ (st )
" t=0
∞
t=1
#
X
= J(θ0 ) + Eτ ∼pθ0 (τ ) γ t (γV πθ (st+1 ) − V πθ (st ))
t=0
" ∞
# " ∞
#
X X
= Eτ ∼pθ0 (τ ) γ t r(st , at ) + Eτ ∼pθ0 (τ ) γ t (γV πθ (st+1 ) − V πθ (st ))
" t=0
∞
t=0
#
X
= Eτ ∼pθ0 (τ ) γ t (r(st , at ) + γV πθ (st+1 ) − V πθ (st ))
" t=0
∞
#
X
= Eτ ∼pθ0 (τ ) γ t Aπθ (st , at )
t=0

35
CHAPTER 7. POLICY GRADIENTS THEORY AND ADVANCED POLICY GRADIENTS36

In the first two steps, we swapped out the initial states distribution in the expectation. This
might seem weird at the first sight, but the intuition is that the initial state marginal is the
same for any policy. Therefore, the expectation taken under the initial state marginal can
be equivalently written as any policy’s trajectory distribution, and for simplicity, we choose
the policy of interest π 0 , with corresponding parameter θ0 .
Now we have proved our claim, but we see the result has a distribution mismatch: the
expectation we take is under πθ0 , but the advantage function A is under πθ . It would be nice
if we could make the two distributions the same. Therefore, we make use of our powerful
statistical tool, importance sampling:

" ∞
#
X X
γ t Aπθ (st , at ) = Est ∼pθ0 (st ) Eat ∼πθ0 (at |st ) γ t Aπθ (st , at )
  
Eτ ∼pθ0 (τ )
t=0 t
  
X πθ0 (at |st ) t πθ
= Est ∼pθ0 (st ) Eat ∼πθ (at |st ) γ A (st , at )
t
πθ (at |st )

Now the outer expectation is still under θ0 state marginal. Can we sim-
ply ignore hthe distribution h mismatch and ii say that it is approximately equal to
P πθ0 (at |st ) t πθ 0
t Est ∼pθ (st ) Eat ∼πθ (at |st ) πθ (at |st ) γ A (st , at ) , which we define as Ā(θ )? We would be
all set if the approximation holds, because if so, then J(θ0 ) − J(θ) ' Ā(θ0 ), which means
we can calculate ∇θ0 Ā(θ0 ) without generating new samples and calculating any new advan-
tage functions because the only term that depends on θ0 in Ā(θ0 ) is the policy term in the
numerator of importance sampling ratio. Thus, we can just use the current samples from πθ .

7.2 Distribution Mismatch Bound

As we discussed above, if we could ignore the distribution mismatch, then we would solve a
number of problems. So when can we indeed ignore the distribution mismatch? We claim
that pθ (st ) is close to pθ0 (st ) when πθ is close πθ0 . This claim sounds rather silly, but in light
of this claim, we could quantify the mismatch and bound the distribution change.
7.2.1 A Simple  Bound
First, let us assume that πθ is deterministic, which means at = πθ (st ). Then as we have seen
in imitation learning, πθ0 is close to πθ if πθ0 (at 6= πθ (st )|st ) ≤ . Using the same probability
bound we defined in imitation learning, we have the new policy’s state marginal defined as:

pθ0 (st ) = (1 − )t pθ (st ) + (1 − (1 − )t )pmistake (st )

and we can bound the prior distribution mismatch by using:

|pθ0 (st ) − pθ (st )| = (1 − (1 − )t )|pmistake (st ) − pθ (st )|

≤ 2(1 − (1 − )t )
≤ 2t

This is not a good bound, but it is a bound.

CHAPTER 7. POLICY GRADIENTS THEORY AND ADVANCED POLICY GRADIENTS37

Now let’s focus on the more general case, that πθ is an arbitrary distribution. Then we
can try to quantify the notion of “close” by saying πθ is close to πθ0 if:
|πθ0 (at |st ) − πθ (at |st )| ≤  ∀st
Here is a useful lemma that we will use later: if |pX (x) − pY (y)| = , then there exists a
joint distribution of x, y, which we call p(x, y) such that p(x) = pX (x) and p(y) = pY (y) and
p(x = y) = 1 − . Equivalently, this means that under these circumstances, pX (x) disagrees
with pY (y) with probability . If we plug in our πθ and πθ0 , then we can show that πθ0 (at |st )
takes a different action than πθ (at |st ) with probability less than . Using this lemma, we
have the same bound as in the deterministic case:
|pθ0 (st ) − pθ (st )| = (1 − (1 − )t )|pmistake (st ) − pθ (st )|
≤ 2(1 − (1 − )t )
≤ 2t
Now let us first focus on a more general case where we have a generic function of state f (st ):
X
Ep0θ (st ) [f (st )] = pθ0 (st )f (st )
st
X
= (pθ (st ) − pθ (st ) + pθ0 (st ))f (st )
st
X
= pθ (st )f (st ) − (pθ (st ) − pθ0 (st ))f (st )
st
X
≥ pθ (st )f (st ) − |pθ (st ) − pθ0 (st )|f (st )
st
X
≥ pθ (st )f (st ) − |pθ (st ) − pθ0 (st )| ∗ max f (st )
st
st

≥ Epθ (st ) [f (st )] − 2t max f (st )

st

Now, putting it all together, let us plug in the term inside the expectation taken under the
mismatched distribution:
  
X πθ0 (at |st ) t πθ
Est ∼pθ0 (st ) Eat ∼πθ (at |st ) γ A (st , at )
t
πθ (at |st )
   X
X πθ0 (at |st ) t πθ
≥ Est ∼pθ (st ) Eat ∼πθ (at |st ) γ A (st , at ) − 2C (7.1)
t
πθ (at |st ) t

where the constant term C is a constant depending on the maximum reward, so in the finite
horizon case, it should be of O(T rmax ), and in infinite horizon case, it should be of O(rmax γ t ),
whose sum can be simplified by convergence theory to O( r1−γ max
). Therefore, for small , we
can simply ignore the mismatch.
What have we proved? We have proved that we can update the policy parameter θ0 by
  
0
X πθ0 (at |st ) t πθ
θ ← arg max Est ∼pθ (st ) Eat ∼πθ (at |st ) γ A (st , at )
θ0 t
πθ (at |st )
such that |πθ0 (at |st ) − πθ (at |st )| ≤  if  is small.
CHAPTER 7. POLICY GRADIENTS THEORY AND ADVANCED POLICY GRADIENTS38

7.2.2 A More Convenient Bound - KL Divergence

We now use a better, more convenient bound, which is the KL-divergence. We claim that we
can apply the aforementioned update if the total variational divergence (DT V ) is bounded
by the KL-divergence as follows:
r
1
|πθ0 (at |st ) − πθ (at |st )| ≤ DKL (πθ0 (at |st )||πθ (at |st ))
2
where KL-divergence is defined as:
 
p1 (x)
DKL (p1 (x)||p2 (x)) = Ex∼p1 (x) log
p2 (x)

Then the update rule of the policy parameter becomes:

  
0
X πθ0 (at |st ) t πθ
θ ← arg max Est ∼pθ (st ) Eat ∼πθ (at |st ) γ A (st , at )
θ0 t
π θ (a t |s t )

such that DKL (πθ0 (at |st )||πθ (at |st )) ≤ . We have guaranteed improvement if we have small
.
7.2.3 Enforcing the Distribution Mismatch Constraint
Now how do we incorporate the constraint on the distribution mismatch with our objective?
One way to do it is to introduce a Lagrangian because we have the following optimization
problem:
  
0
X πθ0 (at |st ) t πθ
θ ← arg max Est ∼pθ (st ) Eat ∼πθ (at |st ) γ A (st , at )
θ0 t
π θ (a t |s t ) (7.2)
s.t. DKL (πθ0 (at |st )||πθ (at |st )) ≤ 

Then we can have our Lagrangian L(θ0 , λ) as

  
0
X πθ0 (at |st ) t πθ
L(θ , λ) = Est ∼pθ (st ) Eat ∼πθ (at |st ) γ A (st , at ) −λ(DKL (πθ0 (at |st )||πθ (at |st ))−)
t
π θ (a t |s t )

Then we optimize in terms of the Lagrangian by first maximizing L(θ0 , λ) with respect to θ0 ,
which we can just do incompletely for a few gradient steps, then we update the dual variable
by λ ← λ + α(DKL (πθ0 (at |st )||πθ (at |st )) − ). This technique is an instance of dual gradient
descent, and we will talk about it more in depth in a later chapter. Essentially, the intuition
is that we raise λ if the constraint is violated too much, and else lower it. Note that one
could also solve this optimization problem by thinking of λ as a regularization term for the
original optimization program.
7.2.4 Other Optimization Techniques
There are also some other ways to optimize based on the distribution mismatch bound. One
way to do it is by using 1st order Taylor expansion. Since θ0 ← arg maxθ0 Ā(θ0 ), we can apply
CHAPTER 7. POLICY GRADIENTS THEORY AND ADVANCED POLICY GRADIENTS39

first order Taylor expansion by

θ0 ← arg max ∇θ Ā(θ)(θ0 − θ)

θ0 (7.3)
s.t. DKL (πθ0 (at |st )||πθ (at |st )) ≤ 

From what we have learned in policy gradients, we can derive the gradient of Ā as
follows:
  
X πθ0 (at |st ) t πθ
∇θ Ā(θ) = Est ∼pθ (st ) Eat ∼πθ (at |st ) γ ∇θ0 log πθ0 (at |st )A (st , at )
t
π θ (a t |s t )

and if we have πθ ' πθ0 , then we can effectively cancel out the importance ratio:
X
Est ∼pθ (st ) Eat ∼πθ (at |st ) γ t ∇θ0 log πθ0 (at |st )Aπθ (st , at ) = ∇θ J(θ)
  
∇θ Ā(θ) =
t

just like normal policy gradient.

Consequently, our original optimization problem can be equivalently written as:

θ0 ← arg max ∇θ J(θ)T (θ0 − θ)

θ0 (7.4)
s.t. DKL (πθ0 (at |st )||πθ (at |st )) ≤ 

We now have the RL objective in our optimization, then can we just use gradient ascent
just like what we did in policy gradient? Well, it turns out gradient ascent is enforcing some
other constraint:
θ0 ← arg max ∇θ J(θ)T (θ0 − θ)
θ0 (7.5)
s.t. ||θ0 − θ||2 ≤ 
whose update rule can be written as
r
0 
θ ←θ+ ∇θ J(θ)
||∇θ J(θ)||2

, and the square root term is just our learning rate, which depends on . We do not want this
constraint in that it is optimizing in the parameter θ space, which is a bounded −ball, but
we want to optimize in the policy space in an ellipsoidal shape because we want to optimize
the more important parameters with smaller step size and less important parameters with
bigger step size.
Since the two optimization problems are not the same, we will tweak the KL-divergence
constraint a little bit using Taylor expansion one more time. If the two policies are very
similar to each other, one could approximate KL-divergence by
1
DKL (πθ0 ||πθ ) ' (θ0 − θ)T F (θ0 − θ)
2
, where F is called the Fisher-information matrix, and it is defined as

F = Eπθ ∇θ log πθ (a|s)∇θ log πθ (a|s)T

 
CHAPTER 7. POLICY GRADIENTS THEORY AND ADVANCED POLICY GRADIENTS40

.
This matrix F can be estimated using samples, and it gives us a convenient quadratic
bound. Using a technique similar to Newton-Raphson, we can update the parameter by

θ0 ← θ + αF −1 ∇θ J(θ)

, and the learning rate α can be chosen as

s
2
α=
∇θ J(θ)T F ∇θ J(θ)

. Now our update rule is a lot more similar to gradient descent, except that in gradient
descent, the l2 norm constrains the update step into a circle, while in our 2nd order approx-
imation of KL-divergence, it constrains the update step into a ellipse. In practice if we want
to solve this natural gradient descent problem with Fisher information matrix efficiently,
there are some nice techniques to approximate the inverse of F , as suggested in the TRPO
paper [8].
Chapter 8: Model-Based Reinforcement Learning

What we have covered so far can be categorized as “model-free” reinforcement learning. The
reason why it is called model-free is that the transition probabilities are unknown and we
did not even attempt to learn the transition probabilities. Recall the RL objective:
πθ (τ ) = pθ (s1 , a1 , ..., sT , at )
T
Y
= p(s1 ) πθ (at |st )p(st+1 |st , at )
t=1
" #
X
θ∗ = arg max Eτ ∼pθ (τ ) r(st , at )
θ t

The transition probabilities p(st+1 |st , at ) is not known in all the model-free RL algorithms
that we have learned such as Q-learning and policy gradients. But what if we know the
transition dynamics? Recall that at the very beginning of the notes we drew an analogy
of RL and control theory; in many cases, we do know the system’s internal transition. For
example, in games, easily modeled systems, and simulated environments, the transitions are
given to us. Moreover, it is not uncommon to learn the transition models: in classic robotics,
system identification fits unknown parameters of a known model to learn how the system
evolves, and one could also imagine a deep learning approach where we could potentially fit
a general-purpose model to observed transition data for later use. In fact, the latter case is
the essence of Model-based RL, where we learn the transition dynamics first, and then figure
out how to choose actions. To learn about model-based RL, we shall start from a simpler
case, where we know the transitions and determine how we control the system optimally
based on the transitions. After this, we can apply our optimal control theory to the more
general case, where we actually learn the transitions first.

8.1 Optimal Control

Optimal control is a task that we come across when we are well aware of the transition
probabilities and we try to learn how to control the system optimally. In optimal control,
there are two different categories of controller design: the first one is open-loop control,
where we do not have any state feedbacks, and we roll out a sequence of actions based on
the current state that we observe. The second one is called closed-loop control, where we
determine the action at each time step based on the current state, and how we determine
the action to apply is based on state feedbacks. In some cases, our transition functions are
deterministic, while in others, the transition functions are stochastic.
In a open loop controller, if we have a deterministic transition in our system such that
st+1 = f (st , at ), then our action sequence should be determined by choosing those that can

41
CHAPTER 8. MODEL-BASED REINFORCEMENT LEARNING 42

return the maximum rewards:

T
X
a1 , ..., aT = arg max r(st , at )
a1...aT
t=1
s.t. st+1 = f (st , at )

In stochastic scenarios, the transition function is a probabilistic distribution, where we have

p(st+1 |st , at ), and the action sequence should be chosen based on expectation of the rewards:
T
Y
pθ (s1 , ..., sT ) = p(s1 ) p(st+1 |st , at )
t=1
" #
X
a1 , ..., aT = arg max E r(st , at )|a1 , ..., aT
a1 ,...,aT
t

Note that we roll out all actions to apply only based on the initial state marginal, so we do
not consider any state-feedback in this case.
In a closed-loop controller, however, we keep interacting with the world, so we need a
policy function that can tell us the action to apply if we input the current state: at ∼ π(at |st ),
which we call a state-feedback. We choose our policy function as follows:
T
Y
p(s1 , a1 , ..., sT , aT ) = p(s1 ) π(at |st )p(st+1 |st , at )
t=1
" #
X
π = arg max Eτ ∼p(τ ) r(st , at )
π
t

Generally, π could take many forms, such as a neural net or time-variant linear controller
K t st + kt .

8.2 Open-loop Planning

For now, let us focus on a simple, open-loop controller, and see how we choose actions using
such controller. In open-loop scenarios, we roll out a sequence of actions by doing arg max
on the sum of rewards: a1 , ..., aT = arg maxa1 ,...,aT J(a1 , ..., aT ) and compactly, we can say
that A = arg maxA J(A).
8.2.1 Random Shooting
Perhaps the easiest and most intuitive stochastic optimization method in open-loop control
is the random shooting method. In such method, we first sample some different action
sequences A1 , ..., AN from some known distribution (such as uniform), and then we choose
Ai based on arg maxi J(Ai ). This is highly inefficient in that we are not improving what
we sample, so we might get stuck in some mediocre action sequence. Therefore, we can
keep improving the samples we choose from a Gaussian distribution based on some elites
sequences. This is the basic idea of Cross Entropy Method (CEM).
CHAPTER 8. MODEL-BASED REINFORCEMENT LEARNING 43

Algorithm 15 Cross Entropy Method with Continuous-valued Input

Require: Some base distribution for action sequence p(A)
1: while true do
2: Sample A1 , ..., AN from p(A)
3: Evaluate J(A1 ), ..., J(AN )
4: Pick elites Ai1 , ..., AiM with the highest value, where M < N
5: Refit p(A) to elites Ai1 , ..., AiM

Algorithm 16 Generic Monte Carlo Tree Search (MCTS)

Require: Some base tree policy for expanding nodes, some base default policy to simulate
a trajectory from a leaf
1: while true do
2: Find a leaf sl using TreePolicy(s1 )
3: Evaluate the leaf using DefaultPolicy(sl )
4: Update all values in tree between s1 and sl
5: Take best action from s1

8.2.2 Cross Entropy Method

CEM improves upon the random shooting’s guess and check scheme by choosing some elites
sequences which give us higher rewards and refit the distribution to the high rewards. Intu-
itively, we are getting closer to higher rewards as we refit the distribution. Here is a sketch
of CEM, as shown in Alg. 15. Usually there are two different criteria for CEM termination.
The first one is simply just upper bound the maximum number of iterations. The second
one is more subtle. For example, if we model the distribution as normal, we wish to stop
if the distribution becomes “narrow” enough eventually, and we can say that we stop CEM
when the fit standard deviation is below some threshold value.
When we “fit” a distribution, what we are actually doing is to iteratively search for the
best parameters to find a distribution from which we can sample to give us the best cost-to-
go value. In a normal distribution, we are just learning µ and σ. This algorithm is extremely
simple and very fast if parallelized. However, it suffers from very harsh dimensionality limit
and it only works for open-loop scenarios.
8.2.3 Monte Carlo Tree Search
Now imagine our action space is discrete, we can apply a stochastic optimization technique
called Monte Carlo Tree Search (MCTS), which is very popular in planning in stochastic
games. The gist of this method is that in discrete action space, we are essentially expanding
out a tree. However, the tree might be too big to expand out due to computational cost.
Therefore, one way to save the computational cost is to partially expand the tree and use a
policy to simulate a trajectory from the last expanded node. A generic sketch of MCTS is
shown in Alg. 16. Note that we the tree policy is not an actual policy, because it is just a
method to traverse through our tree in order to select a node to expand. The default policy
is an actual policy that is able to simulate the system. Since simulations are involved here,
we have to be able to roll back to the original state.
CHAPTER 8. MODEL-BASED REINFORCEMENT LEARNING 44

8.2.4 UCT Tree Policy

In MCTS, how do we choose the nodes to expand? Intuitively, we need to keep choosing the
nodes with high rewards so far, and simultaneously pick the ones that have not been chosen
in order to explore. Therefore, one way to do it is to use the UCT tree policy. In this policy,
we gauge the performance of each node by assigning a score function where the score
s
Q(st ) 2 ln N (st−1 )
S(st ) = + 2C
N (st ) N (st )

. If the current node st is not fully expanded, meaning that there is action that we never
took before, then we choose new at ; else, we choose the child with best S(st+1 ).
More details about MCTS can be found in [9] and [10].
8.2.5 Using Derivatives
Let us consider the control theory counterpart of the RL objective. Essentially, we have a
constrained optimization problem defined as follows:
T
X
min c(xt , ut ) s.t. xt = f (xt−1 , ut−1 )
u1 ,...,uT
t=1

if we plug in the transition constraint, we have:

min c(x1 , u1 ) + c(f (x1 , u1 ), u2 ) + ... + c(f (f (...)...), uT )

u1 ,...,uT

which becomes an unconstrained optimization problem. Since it is unconstrained now, one

might ask, can we do gradient descent on it? The usual answer is yes, but only if we use
some more powerful optimization technique such as 2nd-order Newton method. Because
optimization problems such as shooting methods are hard and often ill-conditioned via 1st
order gradient descent.
8.2.6 Shooting Methods and Collocation Methods
There are two different classes of gradient descent based method: shooting method and
collocation method. In shooting methods, we optimize only on action sequences, so the
actions are the only optimization variables. We only optimize upon the actions and apply
the action to the state and see where it shoots to, so the states are the consequences of the
actions that we optimize. The optimization problem can be written as:

min c(x1 , u1 ) + c(f (x1 , u1 ), u2 ) + ... + c(f (f (...)...), uT )

u1 ,...,uT

However, in collocation method, we optimize upon both actions and states, with con-
straints, and the optimization problem is written as:
T
X
min c(xt , ut ) s.t. xt = f (xt−1 , ut−1 )
u1 ,...,uT ,x1 ,...,xT
t=1
CHAPTER 8. MODEL-BASED REINFORCEMENT LEARNING 45

8.2.7 Linear Quadratic Regulator (LQR)

Let us start with a simple case of shooting method, where we apply a 2nd order-style op-
timization technique to achieve optimal control. The simple case assumes that we have a
linear system, where the transition function is affine, and we have a quadratic cost function.
Thus, the transition function should be of the form:
 
x
f (xt , ut ) = Ft t + ft
ut

and the cost function should be of the form:

 T    T
1 xt x x
c(xt , ut ) = C t t + t ct
2 ut ut ut

What we are doing right now is to solve for a closed-form solution for an optimal LQR
controller. The idea is to use backward recursion. Since we are doing shooting method, we
have
min c(x1 , u1 ) + c(f (x1 , u1 ), u2 ) + ... + c(f (f (...)...), uT )
u1 ,...,uT

and the last item is the only term that depends on uT . Therefore, as a base case, we can try
to solve for uT first. In order to simplify our computation, let us define some blocks in the
matrices that we defined above. Specifically, let us assume that
 
CxT ,xT CxT ,uT
CT =
CuT ,xT CuT ,uT

and  
c
cT = xT
cuT
Since our cost function is
 T    T
1 xT x x
Q(xT , uT ) = const + C T T + T cT
2 uT uT uT

by setting gradient to 0, we will have

∇uT Q(xT , uT ) = CuT ,xT xT + CuT ,uT uT + cTuT = 0

By solving this equation, we have solved uT , in terms of known constants and xT :

uT = −Cu−1
T ,uT
(CuT ,xT xT + cuT )

and to make notations more compact, let us denote uT as uT = KT xT + kT , and KT =

−Cu−1
T ,uT
Cut ,xT , kT = −Cu−1 c .
T ,uT uT
CHAPTER 8. MODEL-BASED REINFORCEMENT LEARNING 46

Now having solved our terminal control input uT , which is fully determined by our
terminal state xT , we can eliminate uT in Q(xT , uT ). Plugging in, we have
 T    T
1 xT xT xT
V (xT ) = const + CT + cT
2 K T xT + k T KT xT + kT KT xT + kT
1 1 1 1
= xTT CxT ,xT xT + xTT CxT ,uT KT xT + xTT KTT CuT ,xT xT + xTT KTT CuT ,uT KT xT
2 2 2 2
1
+ xTT KTT CuT ,uT kT + xTT CxT ,uT kT + xTT cxT + xTT KTT cuT + const
2
1 T
= const + xT VT xT + xTT vT
2
where we define vT and VT to make the notation more compact as follows:

VT = CxT ,xT + CxT ,uT KT + KTT CuT ,xT + KTT CuT ,uT KTT
vT = cxT + CxT ,uT kT + KTT cuT + KTT CuT ,uT kT

Having solved the base case, we solve for other optimal control inputs backwards. Let
us first proceed to solve for uT −1 in terms of xT −1 . Now note that uT −1 not only affects state
xT −1 , but it also affects xT because of the system dynamics:
 
xT −1
f (xT −1 , uT −1 ) = xT = FT −1 + fT −1
uT −1
Therefore, the cost function from T − 1 can be calculated as:
 T    T
1 xT −1 xT −1 xT −1
Q(xT −1 , uT −1 ) = const + CT −1 + cT −1 + V (f (xT −1 , uT −1 ))
2 uT −1 uT −1 uT −1
and if we plug the transition dynamics function into V (xT ), we will have:
 T    T  T
1 xT −1 T xT −1 xT −1 T xT −1
V (xT ) = const + FT −1 VT FT −1 + FT −1 VT fT −1 + FTT−1 vT
2 uT −1 uT −1 uT −1 uT −1
More compactly, we write the cost function as:
     T
1 xT −1 xT −1 xT −1
Q(xT −1 , uT −1 ) = const + QT −1 + qT −1
2 uT −1 uT −1 uT −1

where QT −1 = CT −1 + FTT−1 VT FT −1 , and qT −1 = cT −1 + FTT−1 VT fT −1 + FTT−1 vT . To solve the

optimization problem, we set the gradient to 0:

∇uT −1 Q(xT −1 , uT −1 ) = QuT −1 ,xT −1 xT −1 + QuT −1 ,uT −1 uT −1 + quTT −1 = 0

solving the equation, we have the following expression for uT −1 :

uT −1 = KT −1 xT −1 + kT −1
KT −1 = −Q−1
uT −1 ,uT −1 QuT −1 ,xT −1

kT −1 = −QuT −1 ,uT −1 quT −1

CHAPTER 8. MODEL-BASED REINFORCEMENT LEARNING 47

Algorithm 17 Solving for Linear Quadratic Regulator (LQR)

1: Backward Recursion
2: for t = T to 1 do
3: Qt = Ct + FtT Vt+1 Ft
4: qt = ct + FtT VT ft + FtT vt+1
 T    T
1 xt x x
5: Q(xt , ut ) = const + 2 Qt t + t qt
ut ut ut
6: ut ← arg minut Q(xt , ut ) = Kt xt + kt
7: Kt = −Q−1 ut ,ut Qut ,xt
8: kt = −Qut ,ut qut
9: Vt = Qxt ,xt + Qxt ,ut Kt + KtT Qut ,xt + KtT Qut ,ut Kt
10: vt = qxt + Qxt ,ut kt + KtT Qut + KtT Qut ,ut kt
11: V (xt ) = const + 21 xTt Vt xt + xTt vt
12: Forward Recursion
13: for t = 1 to T do
14: ut = Kt xt + kt
15: xt+1 = f (xt , ut )

Applying the same technique backwards, we can solve for the states and inputs at each
time step, as illustrated in Alg. 17. In step 5 of Alg. 17, Q-function represents the total
cost from now until end if we take ut from state xt , and in step 11, the V-function represents
the total cost from now until end from state xt , so V (xt ) = minut Qxt ,ut , which we call
the cost-to-go function. The above derivation is one of the many derivations of the Riccati
Equation.
What we have analyzed above is based on deterministic dynamics. What if the transition
(dynamics) is stochastic? Specifically, consider the following setup:
xt+1 ∼ p(xt+1 |xt , ut )
   
xt
p(xt+1 |xt , ut ) = N Ft + f t , Σt
ut
where our transition is actually a Gaussian distribution with constant covariance. It turns
out that we can apply the exact same algorithm, choosing actions according to ut = Kt xt +kt ,
and we can ignore Σt due to symmetry of Gaussians.
8.2.8 Iterative LQR (iLQR)
In LQR, we assumed that the dynamics are linear. In non-linear cases, however, we can apply
a similar approach called iterative LQR. Specifically, we can iteratively apply Jacobian lin-
earization to locally linearize the system with respect to an equilibrium point. Consequently,
we approximate a non-linear system as a linear-quadratic system:
 
xt − x̂t
f (xt , ut ) ' f (x̂t , ût ) + ∇xt ,ut f (x̂t , ût )
ut − ût
   T  
xt − x̂t 1 xt − x̂t 2 xt − x̂t
c(xt , ut ) ' c(x̂t , ût ) + ∇xt ,ut c(x̂t , ût ) + ∇xt ,ut c(x̂t , ût )
ut − ût 2 ut − ût ut − ût
CHAPTER 8. MODEL-BASED REINFORCEMENT LEARNING 48

Algorithm 18 Iterative LQR (iLQR)

1: while until convergence do
2: Ft = ∇xt ,ut f (δxt , δut )
3: Ct = ∇2xt ,ut c(δxt , δut )
4: ct = ∇xt ,ut c(δxt , δut )
5: Run LQR backward recursion on state δxt = xt − x̂t and action δut = ut − ût
6: Run forward pass with real nonlinear dynamics and ut = Kt (xt − x̂t ) + kt + ût
7: Update x̂t and ût based on states and actions in forward pass

Now we have an LQR system with respect to the divergence from the action space and
state space’s equilibrium points:
 
¯ δxt
f (δxt , δut ) = Ft
δut
 T  T
1 δxt δxt
c̄(δxt , δut ) = Ct ct
2 δut δut
where
Ft = ∇xt ,ut f (δxt , δut )
Ct = ∇2xt ,ut c(δxt , δut )
ct = ∇xt ,ut c(δxt , δut )
Then we can iteratively run LQR with dynamics f¯, cost c̄, state δxt , and action δut .
A sketch of iLQR is shown in Alg. 18. In essence, iLQR is an approximation of Newton’s
method for solving minu1 ,...,uT c(x1 , u1 ) + c(f (x1 , u1 ), u2 ) + ... + c(f (f (...)...), uT ).

8.3 Model-based RL
In this section, we are going to cover a rather simpler case of model-based RL. Specifically,
we are going to talk about a technique to learn a model of the system first, and then use the
optimal control technique we covered last time to improve the model. Furthermore, we will
learn to address uncertainty in the model such as model mismatch and imperfection.
8.3.1 Basics
Why do we learn the model? Because when the model is unknown, we can learn the model
so that we know f (st , at ) = st+1 or p(st+1 |st , at ) in stochastic case, we could use the tools
from optimal control to maximize our rewards.
Our first attempt is naive, we learn f (st , at ) from data, and then plan through it. We call
this approach model-based RL version 0.5, or vanilla model-based RL, as shown in algorithm
19. This is essentially what people do in system identification, which is a technique used
in classic robotics, and it is effective when we can hand-engineer a dynamics representation
using our knowledge of physics, and fit just a few parameters. However, it does not work
generally because of distribution mismatch: when the model is imperfect, we might suffer
from false learning. Furthermore, since we are blindly following a trajectory, the mismatch
exacerbates as we use more expressive model classes, when pπ0 (st ) 6= pπf (st ).
CHAPTER 8. MODEL-BASED REINFORCEMENT LEARNING 49

Algorithm 19 Model-based Reinforcement Learning Version 0.5

Require: Some base policy for data collection π0
1: Run base policy π(at |st ) (e.g. random policy) to collect D = {(s, a, s0 )i }
0 2
P
2: Learn dynamics model f (s, a) to minimize i ||f (si , ai ) − si ||
3: Plan through f (s, a) to choose actions

Algorithm 20 Model-based Reinforcement Learning Version 1.0

Require: Some base policy for data collection π0
1: Run base policy π(at |st ) (e.g. random policy) to collect D = {(s, a, s0 )i }
2: while True do
Learn dynamics model f (s, a) to minimize i ||f (si , ai ) − s0i ||2
P
3:
4: Plan through f (s, a) to choose actions
5: Execute those actions and add the resulting data {(s, a, s0 )j } to D

Algorithm 21 Model-based Reinforcement Learning Version 1.5

Require: Some base policy for data collection π0 , hyperparameter N
1: Run base policy π(at |st ) (e.g. random policy) to collect D = {(s, a, s0 )i }
2: for every N steps do
3: while True do
Learn dynamics model f (s, a) to minimize i ||f (si , ai ) − s0i ||2
P
4:
5: Plan through f (s, a) to choose actions
6: Execute the first planned action, observe resulting state s0 (MPC)
7: Append (s, a, s0 ) to dataset D

Acknowledging this disadvantage, we could improve the vanilla model-based RL by

making pπ0 (st ) = pπf (st ). As we have seen in Alg. 1, we can keep aggregating data into our
dataset in order to make our model converge to demonstration model. Applying the same
approach, we keep updating the dataset by running the current model, and then update
the model accordingly. Take a look at the updated model-based RL algorithm in Alg. 20.
Version 1.0 addresses the model mismatch issue and drives the current model as close as
possible to the true dynamics model. However, we are still blindly following a trajectory in
step 5 of Alg. 20, and if we made a mistake, we would follow the wrong step which makes
the mistake exacerbate. Therefore, we need to somehow adjust our plan as time goes on.
One way to do this is to borrow some ideas from modern control theory: Model Predictive
Control (MPC).
In MPC, we are given the system’s dynamics model, and we are trying to design an
adaptive controller by solving a finite time constrained optimal control problem at each time
step, and take only the first action in the generated sequence of actions. Then we replan
based on the new state. For sake of simplicity, we will skip the discussion about safe set
and terminal set in MPC in this chapter. But the “replan” idea in MPC is exactly what
we need to improve our model-based RL version 1.0. We essentially are aiming to take
one action in the planned sequence and only observe one new state, and then append the
observed transition to our dataset D. The improvement is shown in Alg. 21. The while loop
CHAPTER 8. MODEL-BASED REINFORCEMENT LEARNING 50

Figure 8.1: False belief about the model from overfitting

in algorithm 21 refers to replanning in MPC, which is solving for an optimization problem

at each time step after we take the first action planned. The for loop, however, means that
we are periodically retraining the model in order to make it closer to the true underlying
transition model. Intuitively, the more frequently the agent replans, the less perfect each
individual plan needs to be, because since we are frequently replanning, we are able to correct
our mistakes made in previous plans more easily. Consequently, one is able to correct the
plans as one increase the replanning frequency. Therefore, if we are frequently replanning,
we could use shorter horizons in the CFTOC problem that MPC is solving.
8.3.2 Performance Gaps in Model-based RL
Believe it or not, sometimes model-based RL performs worse than model-free RL. The prob-
lem is from step 5 of algorithm 21. In this step, we plan through the model to choose actions,
which means we are solving an optimization problem based on the data we collect. One could
imagine that if we overfit the data, the agent might have some wrong belief about the model,
thus generating wrong actions. Pictorially, this phenomenon is illustrated in Fig. 8.1.
Therefore, we need to explore to get better, more representative data of the model, thus
preventing overfitting and false belief. The expected value of the reward is not the same as
optimistic or pessimistic. In step 5, when we choose actions, we only take actions for which
we think we will get high reward in expectation, with respect to uncertain dynamics, which
avoids exploiting the model too much.
8.3.3 Uncertainty-aware Models
Under imperfect models and model mismatch, one might expect wrong actions planned.
Therefore, one way to deal with this problem is to construct an uncertainty-aware model,
where we can quantitatively estimate the uncertainty in the model, so that we can assess
the accuracy of the model and the planned actions.
The first idea is to use entropy of output distribution, and as we know, higher entropy
means higher uncertainty. We can estimate the entropy of p(st+1 |st , at ). However, this is
not enough because when the model is wrong, we might still have low variance, thus low
entropy. Even though in some regions the model is highly uncertain, the output entropy is
still low.
The reason why entropy of the output distribution alone is not expressive enough is that
CHAPTER 8. MODEL-BASED REINFORCEMENT LEARNING 51

Figure 8.2: Estimating the model using a neural net

there are two types of uncertainty:

• aleatoric (statistical) uncertainty, where the data itself is noisy.
• epistemic (model) uncertainty, where the model is certain about data, but we are not
certain about model.
These two types of uncertainty are not the same. We cannot gauge the correctness of the
second model based on the output entropy, and the entropy of the first model might be
higher even though it is potentially a very “good” model.
The second idea is to estimate the model uncertainty, where we essentially estimate how
uncertainty we are about the model. Usually, we use maximum likelihood estimation, where

arg max log p(θ|D) = arg max log p(D|θ)

θ θ

Instead if we estimate the posterior of data p(θ|D) instead of argmax, the entropy of the
distribution
R gives us the model uncertainty from the data. Moreover, we can predict using
p(st+1 |st , at , θ)p(θ|D)dθ.
To learn the posterior distribution, we can apply bootstrap ensembles, where we use
multiple networks to learn the same distribution. Formally, say we have N networks, each
with a parameter θi to learn p(st+1 |st , at ), we can then estimate the posterior by:
1 X
p(θ|D) ' δ(θi )
N i

where δ(·) is the direc-delta function. To train it, we need to generate independent datasets
to get independent models. One way to do this is to train θi on Di sampled with replacement
from D. This method is simple, but it is a very crude approximation.
With this ensemble of networks, we choose actions a little differently. Before, we choose
actions by J(a1 , . . . , aH ) = H
P
t=1 r(s t , at ), where st+1 = f (st , at ), and now we average over
the ensemble by J(a1 , . . . , aH ) = N N
1
P PH
i=1 t=1 r(st,i , at,i ), where st+1,i = f (st,i , at,i )
In general, for candidate action sequence a1 , . . . , aH , we first sample θ ∼ p(θ|D), then
at
P each time step t, we sample st+1 ∼ p(st+1 |st , at , θ), then we calculate the reward R =
t r(st , at ), and we repeat the aforementioned steps and accumulate the average reward.

8.3.4 Latent Space Model

In many cases, we are given very complex observations of the states such as pixel-based
images, where we do not have full access to the states. To learn the dynamics using obser-
vations, we need to learn from the latent space and infer the states from observations. From
Fig. 8.3, we can see that we need to learn the following models:
CHAPTER 8. MODEL-BASED REINFORCEMENT LEARNING 52

Figure 8.3: Latent space model

• p(ot |st ), the observation model

• p(st+1 |st , at ), the dynamics model
• p(rt |st , at ), the reward model
Recall that in high level, model-based RL algorithms are basically doing a maximum
likelihood estimation in training given fully observed states:
N T
1 XX
max log pφ (st+1,i |st,i , at,i )
φ N
i=1 t=1

then with latent models, we are not sure about the actual state, so we take the expected
value:
N T
1 XX
max E [log pφ (st+1,i |st,i , at,i ) + log pφ (ot,i |st,i )]
φ N
i=1 t=1

where the expectation is with respect to the distribution of (st , st+1 ) ∼ p(st , st+1 |o1:T , a1:T )
However, the posterior distribution p(st , st+1 |o1:T , a1:T ) is usually intractable if we have
very complex dynamics. As a result, we could instead try to learn an approximate posterior,
which we call qψ (st |o1:t , a1;t ). We could also learn qψ (st , st+1 |o1:t , a1;t ) and qψ (st |ot ). We call
this technique learning an encoder. Learning the distribution qψ (st |ot ) is crude, but it is
the simplest to implement. If we just decide to learn this distribution for now, then the
expectation becomes:
N T
1 XX
max E [log pφ (st+1,i |st,i , at,i ) + log pφ (ot,i |st,i )]
φ N i=1 t=1

such that the expectation is with respect to st ∼ qψ (st |ot ), st+1 ∼ qψ (st+1 |ot+1 )
For now, let us focus on a simple case where q(st |ot ) is deterministic, because the stochas-
tic case requires variational inference, which will be covered in-depth in a later chapter. In
deterministic case, we are training a neural net gψ (ot ) = st using a direc-delta function such
that qψ (st |ot ) = δ(st = gψ (ot )). Then the expectation can be simplified as
N T
1 XX
max log pφ (gψ (ot+1,i )|gψ (ot,i ), at,i ) + log pφ (ot,i |gψ (ot,i ))
φ N
i=1 t=1

Now everything is differentiable, we can train using backpropagation.

Thus, we can slightly modify Alg. 21 so that we can deal with observations and latent
space. We show the sketch of this slightly modified algorithm in Alg. 22. In step 4, we are
respectively learning the dynamics, reward model, observation model, and encoder.
CHAPTER 8. MODEL-BASED REINFORCEMENT LEARNING 53

Algorithm 22 Model-based Reinforcement Learning with Latent States

Require: Some base policy for data collection π0 , hyperparameter N
1: Run base policy π(at |st ) (e.g. random policy) to collect D = {(s, a, s0 )i }
2: for every N steps do
3: while True do
4: Learn dynamics model pφ (st+1 |st , at ), pφ (rt |st ), p(ot |st ), gψ (ot )
5: Plan through f (s, a) to choose actions
6: Execute the first planned action, observe resulting state o0 (MPC)
7: Append (o, a, o0 ) to dataset D

Interested readers can refer to [11] and [12] for more information on learning from pixel-
based images as latent states.
Chapter 9: Model-based Policy Learning

So far we have covered the basics of model-based RL that we first learn a model and use a
model for control. We have seen that this approach does not work well in general because
of the effect of distributional shift in model-based RL. We have also seen the method to
quantify uncertainty in our model in order to alleviate this issue. The methods we covered so
far do not involve learning policies. In this chapter, we will cover model-based reinforcement
learning of policies. Specifically, we will learn global policies and local policies, and combine
local policies into global policies using guided policy search and policy distillation. We shall
understand how and why we should use models to learn policies, global and local policy
learning, and how local policies can be merged via supervised learning into a global policy.
We have seen the difference between a closed-loop and open-loop controller. We also
discussed why an open-loop controller is suboptimal because we are rolling out a whole
sequence of actions solely based on one state observation. Therefore, it would be more ideal
if we could design a closed-loop controller where state feedbacks can help us correct the
mistakes we make. Recall in a stochastic environment, we are optimizing over the policy as
follows:
T
Y
p(s1 , a1 , . . . , sT , aT ) = p(s1 ) π(at |st )p(st+1 |st , at )
t=1
" #
X
π = arg max Eτ ∼p(τ ) r(st , at )
π
t

and π could take several forms: π can be a neural net, which we call a global policy, and
it can also be a time-varying linear controller Kt st + kt as we saw in LQR, which we call a
local policy.

9.1 Back-propagate into the Policy

Let us start with a simple solution for model-based policy learning. Ideally, we could build a
computational graph in Tensorflow, and calculate the partial derivatives step by step so that
we can backpropage into policy and optimize the policy, illustrated in Fig. 9.1. Then we can
modify our model-based policy-free RL algorithm to accomodate this new policy learning
process in Alg. 23.
9.1.1 Vanishing and Exploding Gradients
One problem with Alg. 23, or general gradient-based optimization is that as we progress into
the time steps, we might encounter vanishing or exploding gradients. Because as we apply
chain rule, the gradients get multiplied by each other, so we the product may get extremely

54
CHAPTER 9. MODEL-BASED POLICY LEARNING 55

Figure 9.1: Back-propagate into policies

Algorithm 23 Model-based Reinforcement Learning Version 1.5

Require: Some base policy for data collection π0
1: Run base policy π0 (at |st ) (e.g. random policy) to collect D = {(s, a, s0 )i }
2: while True do
Learn dynamics model f (s, a) to minimize i ||f (si , ai ) − s0i ||2
P
3:
4: Backpropagate through f (s, a) into the policy to optimize πθ (at |st )
5: Run πθ (at |st ), appending the visited tuples (s, a, s0 ) to D.

big (exploding) or extremely small (vanishing), making optimization a lot harder. Further-
more, we have similar parameter sensitivity problems as shooting methods, but we no longer
have convenient second order LQR-like method, because the policy function is extremely
complicated and policy parameters couple all the time steps, so no dynamic programming.
So what can we do about it? First, we can use model-free RL algorithms with synthetic
samples generated by the model. Essentially, we are using models to accelerate model-free
RL. Second, we can use simpler policies than neural nets such as LQR, and train local policies
to solve simple tasks, and then combine them into global policies via supervised learning.

9.2 Model-free Optimization with a Model

Recall the equation from policy gradients:
N T
1 XX
∇θ J(θ) ' ∇θ log πθ (ai,t |si,t )Q̂πi,t
N i=1 t=1

Note that we are not doing any backprop through time in policy gradient because we are
calculating the gradient with respect to an expectation, so we can just take the derivative of
the probability of the samples instead of the actual dynamics function.
Then we look at the regular backprop (pathwise) gradient, we see a more chain rule-like
gradient:
T t
X drt Y dst0 dat0 −1
∇θ J(θ) =
t=1
dst t0 =2 dat0 −1 dst0 −1
The two gradients are different, because the policy gradient is for stochastic systems while
the backprop policy is for deterministic systems. But using variational inference, we can
prove that they are calculating the same gradient differently, thus having different tradeoffs.
We will talk about variational inference more in-depth in the next chapter.
CHAPTER 9. MODEL-BASED POLICY LEARNING 56

Algorithm 24 Dyna
Require: Some exploration policy for data collection π0
1: Given state s, pick action a using exploration policy
2: Observe s0 and r, to get transition (s, a, s0 , r)
3: Update model p̂(s0 |s, a) and r̂(s, a) using (s, a, s0 )
4: Q-update: Q(s, a) ← Q(s, a) + αEs0 ,r [r + maxa0 Q(s0 , a0 ) − Q(s, a)]
5: for K times do
6: Sample (s, a) ∼ B from buffer of past states and actions
7: Q-update: Q(s, a) ← Q(s, a) + αEs0 ,r [r + maxa0 Q(s0 , a0 ) − Q(s, a)]

Algorithm 25 General Dyna

Require: Some exploration policy for data collection π0
1: Collect some data, consisting of transitions (s, a, s0 , r)
2: Learn model p̂(s0 |s, a) (and optionally, r̂(s, a))
3: for K times do
4: Sample s ∼ B from buffer
5: Choose action a (from B, from π, or random)
6: Simulate s0 ∼ p̂(s0 |s, a) (and r = r̂(s, a))
7: Train on (s, a, s0 , r) with model-free RL
8: (optional) take N more model-based steps

Actually, given more samples to reduce variance, policy gradient is more stable because
it does not require multiplying many Jacobians. However, if our models are inaccurate,
the samples we use from the wrong model will be incorrect, and the mistakes are likely to
exacerbate as time goes on. So it would be nice to use such model-free optimizer and keep
the rolled out samples’ trajectory short. This is essentially what Dyna algorithm does.
9.2.1 Dyna
Dyna is an online Q-learning algorithm that performs model-free RL with a model. In step
3 of Alg. 24, we are updating the model and reward function using the observed transition.
Then in step 6, we will sample some old state and action pairs and apply the model onto
the sampled pair, so the s0 in step 7 are synthetic next states. Intuitively, as the models get
better, the expectation estimate in step 7 also gets more accurate. This algorithm seems
arbitrary in many aspects, but the gist is to keep improving models and use models to
improve Q-function estimation by taking expectations.
We can also generalize Dyna to see how this kind of general Dyna-style model-based
RL algorithms work. The generalized algorithm is shown in Alg. 25. As shown in Fig.
9.2, we choose some states (orange dots) from the buffer, simulate the next states using the
learned model, and then train model-free RL with synthetic data (s, a, s0 , r) where s is from
the experience buffer, s0 is from the learned model. One could also take more than one step
if one believes that the model is good enough for more steps.
This algorithm only requires very short (as few as one step) rollouts from model, so the
mistakes will not exacerbate and accumulate much. Moreover, we explore well with a lot of
samples because we still see diverse states.
CHAPTER 9. MODEL-BASED POLICY LEARNING 57

Figure 9.2: General Dyna training

9.3 Local and Global Models

Recall that in LQR, we can turn a constrianed optimization problem into an unconstrained
problem:
min c(x1 , u1 ) + c(f (x1 , u1 ), u2 ) + · · · + c(f (f (. . . ) . . . ), uT )
u1 ,...,uT

Backpropagation is indeed a possible solution to solve this optimization problem, and we

df df dc dc
need dx t
, du t
, dx t
, du t

9.3.1 Local Models

Since LQR gives us a state-feedback controller for a linear system, we can keep linearizing
df df
the system and iteratively apply LQR to generate local models. We fit dx t
, du t
around the
current trajectory or policy. Say the model is a Gaussian p(xt+1 |xt , ut ) = N (f (xt , ut ), Σ),
then we can approximate the model as a linear function f (xt , ut ) ' At xt + Bt ut , and we can
df df
use dx t
as At , and du t
as Bt .
Iterative LQR produces x̂t , ût , Kt , kt , where ut = Kt (xt − x̂t ) + kt + ût . We can execute
the controller using a Gaussian p(ut |xt ) = N (Kt (xt − x̂t ) + kt + ût , Σt ) because we can add
noise to the iLQR controller so that all samples do not look the same. Practically, we can
set Σt = Q−1ut ,ut . We can fit the model p(st+1 |st , at ) using Bayesian linear regression, and use
the global model as prior.
We also need to stay close to old controller if we go too far. If trajectory distribution
is close, then dynamics will be close too. Close here means the KL-divergence is small
DKL (p(τ )||p(τ̄ )) ≤ .
9.3.2 Guided Policy Search
The high level idea of guided policy search is to use some simpler local policy such as local
LQR controller to help and guide the learning process of more complex global policy learner.
Essentially, we would use the local models trajectories as the training data for a supervised
learning neural net that can solve all the tasks.
CHAPTER 9. MODEL-BASED POLICY LEARNING 58

Figure 9.3: Local models fitting

Algorithm 26 Guided Policy Search

1: while True do
2: Optimize each local policy πLQR,i (ut |xt ) on initial state x0,i with respect to c̃k,i (xt , ut )
3: Use samples from the previous step to train πθ (ut |xt ) to mimic each πLQR,i (ut |xt )
4: Update cost function c̃k+1,i (xt , ut ) = c(xt , ut ) + λk+1 log πθ (ut |xt )

However, one problem is that the local policies might not be able to be reproduced using
a single neural net. Therefore, after training the global policy with supervised learning, we
need to reoptimize the local policies using the global policy so that the policies are consistent
with each other. The sketch of guided policy search is shown in Alg. 26. Note that the cost
function c̃k,i is the modified cost function to keep πLQR close to πθ .
In Divide and Conquer RL, the idea is similar, except that we are replacing the local
LQR controllers with local neural net.
9.3.3 Distillation
In RL, we borrow some ideas from supervised learning to achieve the task of learning a global
policy from a bunch of local policies.
Recall in supervised learning, we use model ensemble to make our predictions more
robust and accurate. However, keeping a lot of models is expensive during test time. Is
there a way to train just one model that can behave as well as a meta-learner?
The idea, proposed by Hinton in [13], is to train a model on the ensemble’s predictions
as “soft” targets using:
exp(zi /T )
pi = P
j (zj /T )

where T is called temperature. The new labels here can be intuitively explained using the
example of MNIST dataset. For example, a handwritten digit “2” looks like a 2 and a
backward 6. Therefore, the soft-labels that we use to train the distilled model is going to be
“80% chance being 2 and 20% chance being 6”.
In RL, to achieve multi-task global policy learning, we can use something similar called
CHAPTER 9. MODEL-BASED POLICY LEARNING 59

policy distillation. The idea is to train a global policy using a bunch of local tasks:
X
L= πEi (a|s) log πAM N (a|s)
a

where the meta-policy πAM N can be trained in a supervised learning fashion.

For more details, please refer to [14, 15].
Chapter 10: Variational Inference and Generative Mod-
els

In this chapter we are going to explore some techniques that allow us to infer latent variables
in latent space. We will try to understand the role of latent probabilistic models in deep
learning and how to use them.
In RL, we are mostly concerned with conditional distributions p(x|y) because we are
trying to fit a policy function πθ (a|s) which is a probabilistic model of action conditioned on
state.
So what are latent variable models? Consider that we have a very complicated distribu-
tion p(x), which cannot be easily modeled by a mixture of Gaussians. By Bayes’ rule, this
complicated prior can be modeled by two other easier distributions:
Z
p(x) = p(x|z)p(z)dz

p(x|z) and p(z) could be modeled by a conditional Gaussian and a Gaussian respectively.
Since any function could be represented by a big enough neural network to an arbitrary pre-
cision, we can then use a neural net to represent p(x|z) as p(x|z) = N (µnn (z), µnn (z)). This
sample distribution is a easy distribution with complicated parameters. Often in practice,
we won’t even learn p(z), because we could just model it as a Gaussian distribution and
transform it to any nonlinear distribution using the integral. The challenge of this approach,
however, is to efficiently approximate the integral, which is quite hard.
In RL, we mainly use latent variable models in the following scenarios. First, we could
use conditional latent variable models for multi-modal policies, as we discussed in imitation
learning. Specifically, we could train a network with Gaussian noise to infer the state from
image-based observations. Another scenario is that we could use latent variable models for
model-based RL. Essentially, we learn a conditional distribution p(ot |xt ) and prior p(xt ).

10.1 Training Latent Variable Models

Pmodel using data D = {x1 , x2 , . . . , xN }.
The model we are trying to fit is pθ (x). We train the
1
We use maximum likelihood R θ ← arg max
θ N i log pθ (xi ). Using latent variables, we
fit:
have θ ← arg maxθ N1 i log pθ (xi |z)p(z)dz . And as we have shown above, the integral
P
is completely intractable.
Alternatively, we could use the expected log-likelihhod:
1 X
theta ← arg max Ez∼p(z|xi ) log pθ (xi )
θ N i

60
CHAPTER 10. VARIATIONAL INFERENCE AND GENERATIVE MODELS 61

However, the conditional distribution p(z|xi ) is unknown. Therefore, we can approximate

this distribution with a simpler distribution qi (z) = N (µi , σi ).
10.1.1 Variational Approximation
It turns out that if we approximate the distribution using qi (z), we can bound the distribution
of interest log p(xi ). Therefore, by maximizing this lower bound, we are maximizing the log
likelihood. We use qi (z) to approximate log p(xi ) by:
Z
log p(xi ) = log p(xi |z)p(z)
Zz
qi (z)
= log p(xi |z)p(z)
z qi (z)
 
p(xi |z)p(z)
= log Ez∼qi (z)
qi (z)
 
p(xi |z)p(z)
≥ Ez∼qi (z) log
qi (z)
= Ez∼qi (z) [log p(xi |z) + log p(z)] − Ez∼qi (z) [log qi (z)]
= Ez∼qi (z) [log p(xi |z) + log p(z)] + H(qi )
where we applied Jensen’s inequality in the second to last step. Jensen’s inequality states
that:
log E[y] ≥ E[log y]
If we maximize log p(xi |z), we will maximize log p(xi ). Also, intuitively, if we maximize
log p(xi |z), we are maximizing the peak of the distribution, and since we are maximizing the
entropy H(qi ) too, we are also making the distribution as wide as possible, which is how
we drive the approximated distribution qi (z) as close as possible to the target distribution
p(xi , z).
Let us take a closer look at this lower bound. Define Ez∼qi (z) [log p(xi |z) + log p(z)] +
H(qi ) as Li (p, qi ). In tuitively, this term measures the likelihood. For a qi (z) to approximate
p(z|xi ) well, we need to minimize the KL-divergence between the two distributions. By
definition, the KL divergence of the two distributions is written as:
 
qi (z)
DKL (qi (z)||p(z|xi )) = Ez∼qi (z) log
p(z|xi )
 
qi (z)p(xi )
= Ez∼qi (z) log
p(xi , z)
= −Ez∼qi (z) [log p(xi |z) + log p(z)] + Ez∼qi (z) [log qi (z)] + Ez∼qi (z) [log p(xi )]
= −Ez∼qi (z) [log p(xi |z) + log p(z)] − H(qi ) + log p(xi )
= −Li (p, qi ) + log p(xi )
Therefore,
log p(xi ) = DKL (qi (z)||p(z|xi )) + Li (p, qi )
log p(xi ) ≥ Li (p, qi )
CHAPTER 10. VARIATIONAL INFERENCE AND GENERATIVE MODELS 62

Note that we eliminated the expectation Ez∼qi (z) [log p(xi )] because p(xi ) does not depend z.
Since DKL (qi (xi )||p(z|xi )) = −Li (p, qi )+log p(xi ), maximizing Li (p, qi ) with respect to qi
minimizes theP KL-divergence. Now in our maximum likelihood training, insteadPof doing θ ←
arg maxθ N1 i log pθ (xi ), we can use the lower bound and do θ ← arg maxθ N1 i Li (p, qi ) to
approximate it. To optimize, for each xi , we calculate ∇θ Li (p, qi ) by sampling z ∼ qi (z) and
the gradient of the likelihood term can be approximated using ∇θ Li (p, qi ) ' ∇θ log pθ (xi |z)
because log pθ (xi |z) is the only term in the likelihood that depends on θ. Then we apply
gradient ascent on the parameter θ by θ ← θ + α∇θ Li (p, qi ).
However, we also need to update qi to maximize Li (p, qi ) because it also depends on
H(qi ). Let’s say qi (z) = N (µi , σi ), then we can apply gradient ascent on both parameters µi ,
σi to update this distribution. The problem here is the above update rule is for each data
point. Therefore, the number of parameters is |θ| + (|µi | + |σi |) ∗ N , where N is the number
of data points. Thus, we can modify the distribution we are learning so that we use a more
general neural network to approximate q(z|xi ) such that q(z|xi ) = qi (z) ' p(z|xi ). Now the
number of the network parameter does not scale with the number of data points.
10.1.2 Amortized Variational Inference
The above idea is called amortized variational inference. When we maximize the likelihood,
instead of using qi for each data point, we use a general neural net qφ , parameterized by φ.
Then when we update qφ , we can just apply gradient ascent on φ by φ ← φ + α∇φ L. The
likelihood can be denoted as Li (pθ (xi |z), qφ (z|xi )).
How do we calculate ∇φ L? Note that

Li = Ez∼qφ (z|xi ) [log pθ (xi |z) + log p(z)] + H(qφ (z|xi ))

to calculate the gradient of the likelihood with respect to φ, we can calculate the entropy
term’s gradient easily using textbook formula. However, the first term is harder because the
expectation is taken under a distribution depending on φ, but the term inside the expectation
is independent of φ. Where have we seen this before? Where have seen the same type of
gradient in policy gradient, and by applying the convenient identity, we can get the same
form of gradient. If we call log pθ (xi |z)+log p(z) as r(xi , z), and Ez∼qφ (z|xi ) as J(φ). Applying
the same trick as in policy gradient, we can calculate ∇J(φ) as:
1 X
∇J(φ) ' ∇φ log qφ (zj |xi )r(xi , zj )
M j

10.1.3 The Reparameterization Trick

Consider qφ (z|x) as a Gaussian distribution N (µφ (x), σφ (x)), then for every z in this distri-
bution, it can be expressed as z = µφ (x) + σφ (x), where  is some type of a Gaussian noise
 ∼ N (0, 1), and the noise is independent of φ. Thus, we have:

J(φ) = Ez∼qφ (z|xi ) [r(xi , z)]

= E∼N (0,1) [r(xi , µφ (x) + σφ (x))]

To estimate ∇φ J(φ), we can just sample M samples of  from a Gaussian N (0, 1).
CHAPTER 10. VARIATIONAL INFERENCE AND GENERATIVE MODELS 63

Figure 10.1: Variational inference

Using this reparameterization trick, we can derive the expression of Li in another way:
Li = Ez∼qφ (z|xi ) [log pθ (xi |z) + log p(z)] + H(qφ (z|xi ))
= Ez∼qφ (z|xi ) [log pθ (xi |z)] + Ez∼qφ (z|xi ) [log p(z)] + H(qφ (z|xi ))
= Ez∼qφ (z|xi ) [log pθ (xi |z)] − DKL (qφ (z|xi )||p(z))
= E∼N (0,1) [log pθ (xi |µφ (xi ) + σφ (xi ))] − DKL (qφ (z|xi )||p(z))
' log pθ (xi |µφ (xi ) + σφ (xi )) − DKL (qφ (z|xi )||p(z))
The complete computational graph for variational inference is shown in Fig. 10.1.
Compared with policy gradient, the reparameterization trick is easy to implement and
as low variance, but it only works for continuous latent variables. Policy gradient can Can
handle both discrete and continuous latent variables, but it is subject to high variance, rand
equires multiple samples and small learning rates.

10.2 Variational Autoencoder (VAE)

The variational autoencoder (VAE) consists of two parts: an encoder and a decoder. The
encoder qφ (z|x) = N (µφ (x), σφ (x)) parameterized by φ gives us a latent variable z, and the
decoder pθ (x|z) = N (µθ (x), σθ (x)) is parameterized
R by θ.
When we are inferring p(x) by p(x) = p(x|z)p(z)dz, we sample z from the distribution
p(z), and sample x from the distribution p(x|z). Why does this work? Recall the evidence
lower bound Li is defined as:
Li = Ez∼qφ (z|xi ) [log pθ (xi |z)] − DKL (qφ (z|xi )||p(z))
qφ should embed your observations xi into z, into a distribution that is closer to the prior.
So if the training data is embedded into the distribution that is similar to the prior, it makes
sense that the samples from the prior will give you things that look like the data.

Figure 10.2: VAE

Chapter 11: Control as Inference

In this chapter, we will talk about how we derive optimal control, reinforcement learning,
and planning as probabilistic inference. In a lot of scenarios that, say, involve biological
behaviors, the data is not optimal. The behavior of the agent might be stochastic, but good
behaviors are still more likely.

11.1 Probabilistic Graphical Model of Decision Making

When we do not make any assumption of optimal behavior, we cannot ensure that the actions
are chosen optimally. In other words, we cannot assume the following relation:
T
X
a1 , . . . , aT = arg max r(st , at )
a1 ,...,aT
t=1

Instead, we should model the probability distribution of seeing a trajectory p(τ ) =

p(s1:T , a1:T ). We also introduce a binary optimality variable Ot , which represents if the
agent if behaving optimally at time step t. Then we are interested in p(τ |O1:T ), where we
infer the probability of the given trajectory given the agent is optimal at every time step.
Now we will model the optimality variable as follows: we model that the probability that
variable is true given state and action is an exponential of the reward:

p(Ot |st , at ) = exp r(st , at )

this might seem an arbitrary choice at the first sight, but we shall see later that this gives
us an elegant mathematical expression in our derivation. We also assume for now that
the reward function is always negative, but we can always take any reward function and

Figure 11.1: Optimalility in stochastic behaviors

64
CHAPTER 11. CONTROL AS INFERENCE 65

normalize it by subtracting the max reward. Then by Bayes’ Rule, we have:

p(τ, O1:T )
p(τ |O1:T ) =
p(O1:T )
Y
∝ p(τ ) exp r(st , at )
t
X
= p(τ ) exp r(st , at )
t

What does the above expression imply? Well, let us pretend that the dynamics are deter-
ministic, then the first term p(τ ) just means if this trajectory is possible. If not, then the
probability is 0. If the trajectory is indeed possible, since we are multiplying by the exponent
of the sum of rewards, then the probability of a trajectory given the agent is acting optimally
is big with high rewards, but small with low rewards.
Let us take a look at the optimality model in Fig. 11.1. Why is this model important?
Because the model is able to model suboptimal behavior, which is important for inverse
RL that will be covered later. We then can apply inference algorithms to solve control and
planning problems. It also provides an explanation for why stochastic behavior might be
preferred, which is useful for exploration and transfer learning.
11.1.1 Inference in the Optimality Model
The first inference we will do is to compute the backward message βt (st , at ) = p(Ot:T |st , at ),
which means the probability of the agent being optimal from the current time step to the
end given state and action. Another inference we will do is the policy p(at |st , O1:T ). Note
that we are inferring the possible actions taken given optimality. The last inference we do is
the forward message αt (st ) = p(st |O1:t−1 ), which is the probability of landing in a particular
state given that the agent is acting optimally up to the current time step.
11.1.2 Inferring the Backward Messages
The backward messages we are inferring is βt (st , at ) = p(Ot:T |st , at ), which we will try to
express in terms of transition probability p(st+1 |st , at ) and optimality probability p(Ot |st , at ).
Mathematically, we can calculate βt (st , at ) as:

βt (st , at ) = p(Ot:T |st , at )

Z
= p(Ot:T , st+1 |st , at )dst+1
Z
= p(Ot+1:T |st+1 )p(st+1 |st , at )p(Ot |st , at )dst+1

The second and the third terms in the product are known, so let us now focus on the first
term:
Z
p(Ot+1:T |st+1 ) = p(Ot+1:T |st+1 , at+1 )p(at+1 |st+1 )dat+1
Z
= β(st+1 , at+1 )dat+1
CHAPTER 11. CONTROL AS INFERENCE 66

we ignored p(at+1 |st+1 ) it means which actions are likely a priori, and we assume it is uniform
(constant) for now.
Therefore, to calculate the backward message, we have a recursive relation. For t =
T − 1 to 1:
βt (st , at ) = p(Ot |st , at )Est+1 ∼p(st+1 |st ,at ) [βt+1 (st+1 )]
βt (st ) = Eat ∼p(at |st ) [βt (st , at )]

11.1.3 A Closer Look

Let us take a closer look at the backward pass. Let Vt (st ) = log βt (st ), and let Qt (st , at ) =
log βt (st , at ). Then Z
Vt (st ) = log exp(Qt (st , at ))dat

As Qt (st , at ) gets bigger Vt (st ) → maxat Qt (st , at ). Using the expression of βt (st , at ), we will
have
Qt (st , at ) = r(st , at ) + log E[exp(Vt+1 (st+1 ))]
Recall in value iteration, we set Q(s, a) ← r(s, a) + γE[V (s0 )]. When the transition
is deterministic, we have Qt (st , at ) = r(st , at ) + Vt+1 (st+1 ), which is similar to value itera-
tion. However, when the transition is stochastic, then the log exp term is like a maximum
operation, so we have a biased optimistic estimation of the Q-function.
11.1.4 Aside: The Action Prior
Recall that we assumed p(at |st ) to be uniform, so it became constant in our integral. How-
ever, we shall see that it does not Rchange much if the action prior is not uniform. Our V
function now becomes Vt (st ) = log exp(Qt (st , at ) + log p(at |st ))dat , and our Q-function be-
comes Q(st , at ) = r(st , at ) + log p(at |st ) + log E[exp(Vt+1 (st+1 ))] We can put the extra p(at |st )
into the reward term, then we will have the same expression of the Q-funtion, thus the V
function. Therefore, uniform action prior can be assumed without loss of generality because
it can always be folded into the reward.
11.1.5 Inferring the Policy
Now with backward messages available to us, we can then proceed to infer the policy
p(at |st , O1:T ). We derive the policy as follows:
p(at |st , O1:T ) = π(at |st )
= p(at |st , Ot:T )
p(at , st |Ot:T )
=
p(st |Ot:T )
p(Ot:T |at , st )p(at , st )/p(Ot:T )
=
p(Ot:T |st )p(st )/p(Ot:T )
p(Ot:T |at , st ) p(at , st )
=
p(Ot:T |st ) p(st )
βt (st , at )
= p(at |st )
βt (st )
CHAPTER 11. CONTROL AS INFERENCE 67

we discard the optimality variables 1, . . . t − 1 because then are conditionally independent of

st . We also discard p(at |st ) since we can assume it as uniform. Using the definition of V, Q,
we have
βt (st , at )
π(at |st ) =
βt (st )
= exp(Qt (st , at ) − Vt (st ))
= exp(At (st , at ))
This result makes sense, because when we have large advantage function values, the action
is more likely to be taken.
11.1.6 Inferring the Forward Messages
We now can infer our third task, the forward message αt (st ) = p(st |O1:t−1 ). The derivation
is as follows:
αt (st ) = p(st |O1:t−1 )
Z
= p(st , st−1 , at−1 |O1:t−1 )dst−1 dat−1
Z
= p(st |st−1 , at−1 , O1:t−1 )p(at−1 |st−1 , O1:t−1 )p(st−1 |O1:t−1 )dst−1 dat−1
Z
= p(st |st−1 , at−1 )p(at−1 |st−1 , Ot−1 )p(st−1 |O1:t−1 )dst−1 dat−1

here we used the fact that the current state is conditionally independent of the previous
optimality variables given the previous state, and we also used the fact that the current
action is conditionally independent of the previous optimality variables given the current
state. The first term is just the dynamics, so we need to figure out what the second and the
third terms by Bayes’ rule:
p(Ot−1 |st−1 , at−1 )p(at−1 |st−1 ) p(Ot−1 |st−1 )p(st−1 |p(O1:t−2 )
p(at−1 |st−1 , Ot−1 )p(st−1 |O1:t−1 ) =
p(Ot−1 |st−1 ) p(Ot−1 |O1:t−2 )
p(Ot−1 |st−1 , at−1 )p(at−1 |st−1 )
= αt−1 (st−1 )
p(Ot−1 |O1:t−2 )
so now we have a recursive relation, and αa (s1 ) = p(s1 ) is usually known.
Another byproduct of having this forward message is that we can combine it with the
backward message to calculate the probability of landing in a particular state given optimality
variables:
p(st , O1:T ) p(Ot:T |st )p(st , O1:t−1 )
p(st |O1:T ) = = ∝ βt (st )p(st |O1:t−1 )p(O1:t−1 ) ∝ βt (st )αt (st )
p(O1:T ) p(O1:T )
Geometrically, the relation between the state marginal and the product of backward and
forward messages is shown in Fig. 11.2. Here the backward messages is a backward cone,
and the forward message is a forward cone. When we take the product of the two, we are
essentially finding the intersection of the two cones. Intuitively, for a state in a trajectory,
the state marginals are tighter near the beginning and the end, but looser near the center
because the state marginals need to close in at the beginning and the end of a trajectory.
CHAPTER 11. CONTROL AS INFERENCE 68

Figure 11.2: Forward/backward messages intersection

11.2 The Optimism Problem

Recall in the dynamic programming view of our backward message inference, the Q-function
can be written as:
Qt (st , at ) = r(st , at ) + log E[exp(Vt+1 (st+1 ))]
We have shown that log E exp behaves like a max, thus bringing us bias in the estimate of
Q-function. Marginalizing and conditioning the backward message βt (st , at ) = p(Ot:T |st , at ),
we can have two different distributions to infer: first, we can have the policy p(at |st , O1:T ),
which means given that you had a high reward (optimal), what was your action probability?
Second, we can have the transition p(st+1 |st , at , O1:T ), and we should notice that this is
not equal to the transition probability p(st+1 |st , at ) because now we are asking given that
you obtained high rewards, what was your transition probability? To address the optimism
problem, we need to ask the first question: given that you obtained high reward, what was
your action probability, assuming that we have the same transition probability, such that we
are no luckier than we usually are.
It turns out the first question is a difficult one. To answer that question, we can find an-
other distribution q(s1:T , a1:T ) that is close to p(st , at |O1:T ), but have the same p(st+1 |st , at ).
So let’s us try variational inference. Let our evidence x, what we have observed, be the
optimality variables O1:T , and the latent variable z, what we have not observed, be the
trajectory s1:T , a1:T . Using variational
Q inference, we find a q(z) to approximate p(z|x).
Let q(s1:T , a1:T ) = p(s1 ) t p(st+1 |st , at )q(at |st ) since we are keeping the same initial
state distribution and the same transition. Recall that the variational lower bound of the
likelihood approximation is:

log p(x) ≥ Ez∼q(z) [log p(x, z) − log q(z)]

CHAPTER 11. CONTROL AS INFERENCE 69

plugging in our previous definition of q(z), we have

 T
X
log p(O1:T ) ≥ E(s1:T ,a1:T )∼q log p(s1 ) + log p(st+1 |st , at ) + log p(OT |st , at )
t=1
T
X T
X 
− log p(s1 ) − log p(st+1 |st , at ) − log q(at |st )
t=1 t=1
" #
X
= E(s1:T ,a1:T )∼q r(st , at ) − log q(at |st )
t
= E(s1:T ,a1:T )∼q [r(st , at ) + H(q(at |st ))]

Therefore, to maximize the lower bound, we maximize the reward and the entropy.
Using dynamic programming, we can get rid of the optimism max in the Bellman backup
term.
Chapter 12: Inverse Reinforcement Learning

So far in our RL algorithms, we have been assuming that the reward function is known a
priori, or it is manually designed to define a task. What if we want to learn the reward
function from observing an expert, and then use reinforcement learning? This is the idea of
inverse RL, where we first figure out the reward function and then apply RL.
Why should we worry about learning rewards at all? From the imitation learning
perspective, the agent learns via imitation by copying the actions performed by the expert,
without any reasoning about outcomes of actions. However, the natural way that human
learn through imitation is that human copy the intent of the expert, and thus might take
very different actions. In RL, it is often the case that the reward function is ambiguous in
the environment. For example, it is hard to hand-design a reward function for autonomous
driving.
The inverse RL problem definition is as follows: we try to infer the reward functions
from demonstrations, and then learn to maximize the inferred reward using any RL algorithm
that was covered so far. Formally, in inverse RL, we learn rψ (s, a), and then use it to learn
π ∗ (a|s). However, this is an underspecified problem, because many reward function can
explain the same behavior. The reward function can take many forms. One potential form
is the linear reward function, which is a weighted sum of features:
X
rψ (s, a) = ψi fi (s, a) = ψ T f (s, a)
i

or it could be a neural net with parameters ψ.

12.1 Feature Matching Inverse RL

Let us focus on the linear reward function design for now. Since it is a weighted sum of
features, one natural interpretation to match the features is to match the expectation of
important features. Let π rψ be the optimal policy for reward functionrψ , then we to design
the reward, we are picking ψ such that

Eπrψ [f (s, a)] = Eπ∗ [f (s, a)]

The right hand side expectation can be estimated using samples from expert: take N samples
of features, and get the average. The left hand side expectation is a little involved. One
way to do it is to use any RL algorithm to maximize rψ , which is defined using the right
hand side samples, and then produce π rψ , and then we can use this policy to generate more
samples. Another way is to use dynamic programming if we are given the transitions. To
ensure the equality holds, we borrow some ideas from the support vector machine classifier,

70
CHAPTER 12. INVERSE REINFORCEMENT LEARNING 71

where we maximize the margin between the optimal policy’s rewards and that of any other
policy:
max m s.t. ψ T Eπ∗ [f (s, a)] ≥ max ψ T Eπ [f (s, a)] + m
ψ,m π∈Π

but we also need to address the similarity between π and π ∗ so that similar policies do not
need to abide by the m margin requirement.
Using the SVM trick (with the use of Lagrangian dual), we can transform the above
optimization into the following which also contains a function that measures the similarity
between policies:
1
min ||ψ||2 s.t. ψ T Eπ∗ [f (s, a)] ≥ max ψ T Eπ [f (s, a)] + D(π, π ∗ )
ψ 2 π∈Π

where D(π, π ∗ ) measures the difference in feature expectations. However, such approaches
have some issues: maximizing the margin is a bit arbitrary, and there is no clear model of ex-
pert suboptimality (can add slack variables). Furthermore, now we have a messy constrained
optimization problem, which is not great for deep learning!

12.2 Learning the Optimality Variable

Recall that in last chapter, we introduced the optimality variable Ot to indicate if the agent
is acting optimally. It turns out that as we learn the reward function, we are also learning
the optimality variable. The optimality variable is defined as p(Ot |st , at ) = exp(rψ (st , at )).
Since the reward parameter ψ is unknown, the optimality distribution should also depend
on ψ: p(Ot |st , at , ψ). Recall that
!
X
p(τ |O1:T , ψ) ∝ exp rψ (st , at )
t

Note that we can ignore p(τ ) in our optimiztion since it does not depend on ψ. We are
given sample trajectories {τi } sampled from expert policy π ∗ (τ ), so the maximum likelihood
training can be done using:
N N
1 X 1 X
max log p(τi |O1:T , ψ) = max rψ (τi ) − log Z
ψ N ψ N
i=1 i=1

where Z is the partition function needed to make the sum of probability with respect to
τ 1.
12.2.1 Inverse RL Partition Function
In our maximum likelihood training, to make the probability with respect to τ sum to 1, we
introduced the IRL partition function Z. Mathematically, Z is the integral of all possible
trajectories: Z
Z= p(τ ) exp(rψ (τ ))dτ
CHAPTER 12. INVERSE REINFORCEMENT LEARNING 72

Then we take the gradient of the likelihood with respect to ψ after plugging in Z:
N Z
1 X 1
∇ψ L = ∇ψ rψ (τi ) − p(τ ) exp(rψ (τ ))∇ψ rψ (τ )dτ
N i=1 Z
= Eτ ∼π∗ (τ ) [∇ψ rψ (τi )] − Eτ ∼p(τ |O1:T ,ψ) [∇ψ rψ (τ )]
The first expectation is estimated with expert samples, and the second expectation is the
soft optimal policy under current reward. To increase the gradient, we want more expert
trajectory and less current agent trajectory.
12.2.2 Estimating the Expectation
In the above derivation of the gradient of the likelihood, the first expectation is easy to
calculate, but the second one is hard. To calculate the second expectation, we need to do
some messaging:
" T
#
X
Eτ ∼p(τ |O1:T ,ψ) [∇ψ rψ (τ )] = Eτ ∼p(τ |O1:T ,ψ) ∇ψ rψ (st , at )
t=1
T
X
= E(st ,at )∼p(st ,at |O1:T ,ψ) [∇ψ rψ (st , at )]
t=1

Note that the distribution p(st , at |O1:T , ψ) can be rewritten using chain rule as:
p(st , at |O1:T , ψ) = p(at |st , O1:T , ψ)p(st |O1:T , ψ)
where
β(st , at )
p(at |st , O1:T , ψ) =
β(st )
p(st |O1:T , ψ) ∝ α(st )β(st )
Therefore, the distribution is directly proportional to the product of the backward message
and the forward message:
p(at |st , O1:T , ψ)p(st |O1:T , ψ) ∝ β(st , at )α(st )
If we let µt (st , at ) ∝ β(st , at )α(st ), then the second expectation can be written as:
T Z Z
X
Eτ ∼p(τ |O1:T ,ψ) [∇ψ rψ (τ )] = µt (st , at )∇ψ rψ (τ )dst dat
t=1
XT
= µTt ∇ψ rψ
t=1

where µt is the state-action visitation probability for each (st , at ).

Now we are ready to sketch out our MaxEnt Inverse RL algorithm in Alg. 27. We
can use this to learn the reward function. Why is it called maximum entropy (MaxEnt)?
Because in cases where rψ (st , at ) = ψ T f (st , at ), we can show that Alg. 27 oprimizes
max H(π rψ ) s.t. Eπrψ [f ] = Eπ∗ [f ]
ψ
CHAPTER 12. INVERSE REINFORCEMENT LEARNING 73

Algorithm 27 MaxEnt Inverse RL

Require: Some random reward parameter ψ
1: while True do
2: Given ψ, compute backward message β(st , at )
3: Given ψ, compute forward message α(st )
4: Compute µt (st , at ) ∝
Pβ(s t , at )α(st )
Evaluate ∇ψ L = N i=1 t=1 ∇ψ rψ (si,t , ai,t ) − Tt=1
1 N PT P RR
5: µt (st , at )∇ψ rψ (τ )dst dat
6: ψ ← ψ + η∇ψ L

12.3 Unknown Dynamics and Large State/Action Spaces

So far, MaxEnt inverse RL requires us to solve for a soft optimal policy in the inner loop,
and it enumerates all state-action tuples for visitation frequency and gradient. To apply the
IRL algorithms in practical problem settings, we need to handle large and continuous state
and action spaces and unknown dynamics.
Recall the gradient of likelihood is calculated as

∇ψ L = Eτ ∼π∗ (τ ) [∇ψ rψ (τi )] − Eτ ∼p(τ |O1:T ,ψ) [∇ψ rψ (τ )]

We know that the first expectation is easy to calculate by sampling expert data, but the
second expectation which is taken under the soft optimal policy under current reward is hard
to calculate. One idea to calculate it is to learn the entire soft optimal policy p(at |st , O1:T , ψ)
using any max-ent RL algorithm and then run this policy to sample {τj } such that:
N M
1 X 1 X
∇ψ L = ∇ψ rψ (τi ) − ∇ψ rψ (τj )
N i=1 M j=1

where we estimate the second expectation using the current policy samples. However, this is
highly impractical because this requires us to run an RL algorithm to convergence in every
gradient step.
12.3.1 More Efficient Updates
As mentioned above, learning p(at |st , O1:T , ψ) in the inner loop in each time step is expensive.
Therefore, we can relax this objective a little to make it more efficient: instead of learning
the policy at each time step, we could improve the policy a little in each time step such that
if the policy keeps getting better, we can generate good samples eventually. Now sampling
from this improved distribution is not actually sampling from the distribution we want, which
is p(τ |O1:T , ψ), we are actually getting a biased estimate of the distribution. Therefore, to
resolve this issue, we use importance sampling:
N M
1 X 1 X
∇ψ L ' ∇ψ rψ (τi ) − P wj ∇ψ rψ (τj )
N i=1 j wj j=1

p(τ ) exp(rψ (τj ))

wj =
π(τj )
CHAPTER 12. INVERSE REINFORCEMENT LEARNING 74

And if we take a closer look at the importance ratio wj :

p(τ ) exp(rψ (τj ))

wj =
π(τ )
Q j
p(s1 ) t p(st+1 |st , at exp(rψ (st , at ))
= Q
p(s1 ) t p(st+1 |st , at π(at |st )
P
exp( t rψ (st , at ))
= Q
t π(at |st )

With the importance ratio, each policy update with respect to rψ brings us closer to the
target distribution.

12.4 Inverse RL as a Generative Adversarial Network

The idea of inverse RL looks like a game. Specifically, we have an initial policy πθ , and
expert demonstrations π ∗ . We sample trajectories τj from the initial policy, and τi from the
expert policy. Then our gradient step looks like:
N M
1 X 1 X
∇ψ L ' ∇ψ rψ (τi ) − P wj ∇ψ rψ (τj )
N i=1 j wj j=1

where demos are made more likely and samples are made less likely. Then we update the
initial policy πθ with respect to rψ :
M
1 X
∇θ L ' ∇θ log πθ (τj )rψ (τj )
M j=1

which in turn changes the policy to make it harder to distinguish from demos.
This looks a lot like a GAN. In a GAN, we have a generator that takes in some noise z,
and outputs a distribution pθ (x|z). We sample from the generator distribution pθ (x). There is
also demonstration data, for example, the real images, which we sample from its distribution
p∗ (x). There is a discriminator parameterized by ψ that determines if the data generated
by the generator is real: D(x) = pψ (real|x). We update the discriminator parameter by
maximizing the binary log likelihood:
1 X 1 X
ψ = arg max log Dψ (x) + log(1 − Dψ (x))
ψ N x∼p∗ M x∼p
θ

where the log likelihood of the data is from demonstration is maximized and that of the data
is from generator is minimized. We also update the generator parameter θ:

θ ← arg max Ex∼pθ log Dψ (x)

θ

so as to make it harder to distinguish from demos.

CHAPTER 12. INVERSE REINFORCEMENT LEARNING 75

Therefore, interestingly, we can frame the IRL problem as a GAN. In a GAN, the
optimal discriminator can be defined as:

p∗ (x)
D∗ (x) =
pθ (x) + p∗ (x)

For inverse RL, the optimal policy approaches πθ (τ ) ∝ p(τ ) exp(rψ (τ )). Choosing the above
optimal parameterization of the discriminator:

p(τ ) Z1 exp(r(τ ))
Dψ (τ ) =
pθ (τ ) + p(τ ) Z1 exp(r(τ ))
p(τ ) Z1 exp(r(τ ))
= Q 1
p(τ ) t πθ (at |st ) + p(τ ) Z exp(r(τ ))
1
exp(r(τ ))
Z
=Q 1
t πθ (at |st ) + Z exp(r(τ ))

then we optimize the discriminator with respect to ψ such that:

ψ ← arg max Eτ ∼p∗ [log Dψ (τ )] + Eτ ∼πθ [log(1 − Dψ (τ ))]

ψ

Now we don’t need the importance ratio anymore, because it is subsumed into Z.
We could also use a general discriminator, where Dψ is just a normal binary neural net
classifier. It is often simpler to set up optimization, because we have fewer moving parts.
However, the discriminator knows nothing at convergence generally cannot reoptimize the
reward.
Chapter 13: Transfer Learning

This chapter is a high-level overview of transfer learning and multi-task learning techniques.
More to be filled in later.

76
Chapter 14: Exploration

In reinforcement learning, we aim to balance exploitation and exploration. In a lot of setting,

exploring with random behaviors does not give us satisfactory result due to the highly com-
plex environment. Explorations mainly concerns with two different questions: how can an
agent discover high-reward strategies that require a temporally extended sequence of com-
plex behaviors that, individually, are not rewarding? How can an agent decide whether to
attempt new behaviors (to discover ones with higher reward) or continue to do the best thing
it knows so far? Long story short, we can define exploitation as doing what you know will
yield highest reward, and exploration as doing things you haven’t done before, in the hopes
of getting even higher reward. In order to explore well, we need to come up with some smart
strategies to discover some better way to gain rewards. To illustrate, let us look at a classic
example of exploration and exploitation trade-off. Say you want to eat in a restaurant, to
exploit, you would eat at your favorite restaurant, but to explore, you would go to a new
restaurant and see if it is better than your favorite one.

14.1 Multi-arm Bandits

A bandit problem is a type of simple exploration problem. The term comes from the name
of a popular slot machine. We are interested in multi-arm bandits where each arm gives us
different reward, and we are mainly concerned with the question of pulling which arm gives
us the best reward among all arms.
Mathematically, our actions to choose are

A = {pull1 , pull2 , . . . , pulln }

since we have n arms, and assume there is a true distribution of which arm gives us a higher
reward, which we do not know a priori:

r(an ) ∼ p(r|an )

14.1.1 Defining a Bandit

Assume our reward for each arm ri comes from a distribution r(ai ) ∼ pθi (ri ). For example,
if our reward is a binary variable then we can define pθi (ri ) as

p(ri = 1) = θi , p(ri = 0) = 1 − θi

We also know that θi ∼ p(θ), but we do not know anything else about the distribution.
This actually defines a meta-level POMDP. Our latent state is actually s = [θ1 , . . . , θn ],
which is the true parameterization of the arms’ reward distribution. We also have a belief

77
CHAPTER 14. EXPLORATION 78

state, which is our observation in some sense. The belief state is an estimate of the probability
of getting high reward of each arm:

p̂(θ1 , . . . , θn )

To measure the goodness of exploration algorithm, we define the regret of exploration.

The regret of exploration is the difference from optimal policy at time step T :
T
X
∗
Reg(T ) = T E[r(a )] − r(at )
t=1

where the first term means in hindsight, how much reward could I have got if I had taken
the best action all the way until the end, and the second term means the actual reward I
have got. The difference of these two gives us the reward.
14.1.2 Optimistic Exploration
One simple way to explore is to use an optimistic exploration strategy, where we keep track
of average reward µ̂a for each action a. Naturally, one way to pick an action is the greedy
exploitation:
a = arg max µ̂a
then to explore better, we need to add bonus to new actions too:

a = arg max µ̂a + Cσa

where σa is some quantification of uncertainty about action a.

The intuition behind this strategy is to try each arm until you are sure that it is not
great. Therefore, if you think an action might be good, go on and try it for a few more
times, and try something else if you are sure it is not good.
One popular method to gauge the uncertainty about an action is to use the upper
confidence bound (UCB). Specifically,
s
2 ln T
a = arg max µ̂a +
N (a)

where N (a) counts the number of times that we have applied action a. Using this strategy,
we can get a bound of regret O(log T ).
14.1.3 Probability Matching
Recall we have a belief state model that represents our own estimate of each arm’s reward
parameterization:
p̂(θ1 , . . . , θn )
We can improve our belief state by keep updating it. The idea is to sample (θ1 , . . . , θn ) from
the distribution, and pretend that the model (θ1 , . . . , θn ) is correct, then we take the optimal
action and update the belief model. Then we take the action by a = arg maxa Eθa [r(a)].
This is called posterior sampling or Thompson sampling. This method is harder to
analyze theoretically, but can work very well empirically.
CHAPTER 14. EXPLORATION 79

14.1.4 Information Gain

Say we want to determine some latent variable z, one way to decide which action to take is
look at the entropy of the estimate of the prior H(p̂(z)). Intuitively, the entropy should be
high if our estimate is off, and low if our estimate is accurate. By the same token, we can
incorporate evidence y with this entropy so that we look at the entropy H(p̂(z|y)) in order
to see if the entropy changes after y is known to us. For example, y could be the reward
r(a).
The information gain of observing y is defined as

IG(z, y) = Ey [H(p̂(z)) − H(p̂(z|y))]

which is the expected decrease of entropy after observing y. Note that we do not know what
y actually is, but we have some knowledge of what it might be. This is why we are taking
the expected value. Typically, the information gain also depends on the action so we can
have IG(z, y|a). Hence,

IG(z, y) = Ey [H(p̂(z)) − H(p̂(z|y))|a]

it measures how much we learn from z from action a, given the current beliefs.
In our exploration setting, the observation is the observed reward:

y = r(a)

the latent state is the parameters for model p(ra ):

z = θa

then the information gain of a is calculated as:

g(a) = IG(θa , ra |a)

we also define another quantity ∆(a) that measures the expected suboptimality of a:

∆(a) = E[r(a∗ ) − r(a)]

As a result we take action according to the rule

∆(a)2
a = arg min
a g(a)

This rule intuitively means that we do not take an action if we are sure if it is optimal (large
∆(a)), or if we cannot learn anything from applying that action (small g(a)).
We talked about bandits models because bandits are easier to analyze and understand.
We can derive foundations for exploration methods, ane then apply these methods to more
complex MDPs. Most exploration strategies require some kind of uncertainty estimation
(even if it’s naïve). We usually assumes some value to new information. For example, we
assume unknown means good (optimism), sample means truth, and information gain means
good.
CHAPTER 14. EXPLORATION 80

Algorithm 28 Exploring with Pseudo-count

Require: Some base model pθ (s)
1: while not done do
2: Fit model pθ (s) to all the states D seen so far
3: Take a step i and observe si
4: Fit model pθ0 (s) to D ∪ si
5: Use pθ (si ) and pθ0 (s0 ) to estimate N̂ (s)
6: Set r+ (s, a) = r(s, a) + B(N̂ (s))

14.2 Exploration in MDPs

Recall our UCB exploration policy to choose action:
s
2 ln T
a = arg max µ̂a +
N (a)

here N (a) is the exploration bonus.

Can we apply the same idea in MDPs, which are what we work with in RL? Essentially,
we can do the same exploration bonus N (s, a) or N (s) and add it to the reward:

r+ (s, a) = r(s, a) + B(N (s))

where the bonus N (s) decreases with the increase of visitation frequency, and then we use
r+ (s, a) instead of r(s, a) in any model-free algorithm. This is a simple addition to any RL
algorithm, but we need to tune the bonus weight.
14.2.1 Counting the Exploration Bonus
We count the number of times that we have encountered the state s using N (s). However,
in many situations such as video games or autonomous driving, we never actually see the
exact same state twice. Therefore, we need to take the notion of similarity into account: we
count the number of times we have encountered similar states, instead of the same states.
The idea is to fit a density model pθ (s) or pθ (s, a) to the states. pθ (s) is low for very
novel states, and high for states that are very similar to the states we have seen, even if
it might be completely new. To design this density model, we can seek some inspirations
from a simple small MDP. If we have a small MDP, then the density of visiting a state s is
modeled as:
N (s)
P (s) =
n
and if we see the same state again, this density becomes:

N (s) + 1
P 0 (s) =
n+1
we design our neural net density model obeying the same rule.
We devise a deep pseudo-count procedure to count the states as shown in Alg. 28 In
CHAPTER 14. EXPLORATION 81

step 5, we solve for N̂ using the following equations:

N̂ (si )
pθ (si ) =
n̂
N̂ (si ) + 1
pθ0 (si ) =
n̂ + 1
this two equations with two unknowns, and we solve for N̂ , n̂ as follows:

N̂ = n̂pθ (si )
1 − pθ0 (si )
n̂ = pθ (si )
pθ0 (si ) − pθ (si )
These counters are able to count similar states.

14.3 Exploration with Q-functions

Recall in earlier chapters, we covered epsilon-greedy exploration strategy. This strategy is
essentially taking random actions with a probability of . However, in many cases, explor-
ing by taking random actions might not be ideal. For example, in the Atari game called
SeaQuest, taking random actions by going back and forth might make the submarine run
out of oxygen quickly without exploring meaningful states. Therefore, we need to explore
more efficiently by sticking to one general strategy for an extended period of time.
Thus, we introduce exloring with Q-functions. Exploring with random actions (e.g.,
epsilon-greedy) is not efficient enough because we oscillate back and forth, so we might not
go to a coherent or interesting place. However, exploring with random Q-functions make us
commit to a randomized but internally consistent strategy for an entire episode, so we act
coherently in the same episode.
To do this, we maintain a distribution of Q-functions. This distribution could be any Q-
function with artificial Gaussian noise. Then we sample a Q-function from this distribution
p(Q), and act according to this function for the entire episode. Then we update p(Q) and
repeat. Since Q-learning is off-policy, we don’t care which Q-function was used to collect
data.
Using this method, we don’t need to modify the original reward function since we are
just doing off-policy Q-learning, but in many cases good exploration bonus functions tend
to do better.

14.4 Revisiting Information Gain in MDP Exploration

In MDPs, we can also use information gain IG(z, y|a) which was introduced in the multi-
arm bandit problem. However, we need to figure out what information gain we are looking
for exactly. First, we can find information gain about reward r(s, a). However, this is not
useful when the reward is sparse, so nothing meaningful could come out from this. We could
also find information gain about state density p(s), where we can find how much the state
visitation changes after we know something. This is useful because p(s) changes drastically
if the information gain is big enough. We could also learn the information gain about the
transition model p(s0 |s, a), which is good for learning the MDP itself. However, none of the
CHAPTER 14. EXPLORATION 82

above three different settings can be calculated exactly. Thus, we need a proxy to estimate
the information gain.
14.4.1 Prediction Gain
One way to approximate the information gain is to use the prediction gain:
log pθ0 (s) − logθ (s)
which is used on state densities. This quantity takes the difference between the density
before and after seeing the state s. Therefore, if the prediction gain is big, then the state s
is novel.
14.4.2 Variational Information Maximization for Exploration
(VIME)
This method to approximate the information gain was first introduced in [16] by Houthooft
et al.. Mathematically, the information gain can be equivalently written in terms of KL
divergence as:
DKL (p(z|y)||p(z))
There is some quantity about the MDP we want to learn about, which in this case, without
loss of generality, is the transition
pθ (st+1 |st , at )
then in the parametrization of the information gain, what we want to learn about is the
parameter of the quantity of interest θ. z could also be some other distributions that involve
θ such as pθ (s) and pθ (r|s, a). The evidence y we observe is the transition. Therefore:
z=θ
y = (st , at , st+1 )
Our information gain in terms of KL-divergence can be set up as
DKL (p(θ|h, st+1 , st , at )||p(θ|h))
where h is the history of all prior transitions. Therefore, intuitively, the transition we observe
is more intuitive if it causes the belief over θ to change.
The idea of VIME is to use variational inference to approximate p(θ|h) since maintaining
the whole history h is not feasible. So we use a distribution to approximate the history:
q(θ|φ) ' p(θ|h)
the new distribution is parameterized by φ, so when we observe a new transition, we update
φ to get φ0 .
As you recall, we update the parameters by optimizing the variational lower bound
DKL (q(θ|φ)||p(h|θ)p(θ)
and we represent q(θ|φ) as a product of independent Gaussians of parameter distributions
with mean φ.
After updating φ0 , we use DKL (q(θ|φ0 )||q(θ|φ)) as the approximate information gain.
CHAPTER 14. EXPLORATION 83

Algorithm 29 Pretrain and Fine Tune

Require: Demonstrations data D
1: Collect demonstration data D = {(si , ai )}
P
2: Initialize πθ as maxθ i log πθ (ai |si )
3: while not done do
4: Run πθ to collect experience
5: Improve πθ with any RL algorithm

14.5 Improving RL with Imitation

Imitation learning is simple, stable supervised learning, but it requires demonstrations, and
it must address distributional shift. Furthermore, the state of the art imitation learning can
only be as good as the demo. In contrast, reinforcement learning can become arbitrarily
good, but it requires reward function, and must address exploration. Also, it often does not
have convergence guarantees.
14.5.1 Pretrain and Finetune
Can we somehow combine the two such that we have both demonstrations data and rewards?
The answer is yes. The simplest idea that is practical and works is to pretrain with imitation
learning and fine tune with RL. This idea is shown in Alg. 29.
The problem with Alg. 29 is that in step 4, we might collect very bad experience due to
distribution shift, so the first batch of data might be bad, thus destroying the initialization.
14.5.2 Off-policy RL
One way to mitigate the issue of forgetting the demonstrations is to use off-policy RL because
off-policy RL can use any data, and if we let it use demonstrations as off-policy samples,
since demonstrations are provided as data in every iteration, they are never forgotten. Fur-
thermore, the policy can still become better than the demos, since it is not forced to mimic
them. To achieve this, we could use off-policy policy gradients and off-policy Q-learning.
Recall policy gradients with importance sampling:
" T t
! T !#
X X Y πθ (at0 |st0 ) X
∇θ J(θ) = ∇θ log πθ (at |st ) r(st0 , at0 )
q(a t0 |st0 )
τ ∈D t=1 0
t =1 0
t =t

The trick here is when we collect the sum of samples, in our samples, we include both our
experience and demonstration. However, it seems a little weird because in policy gradients
we actually want on-policy data, so why are we including off-policy data. To answer this
question, let us build up some intuition by looking at the optimal importance sampling
distribution. Say we want to estimate Ep(x) [f (x)] using importance sampling:

1 X p(xi )
Ep(x) [f (x)] ' f (xi )
N i q(xi )

and it can be proven that

q ∝ p(x)|f (x)|
CHAPTER 14. EXPLORATION 84

gives us the smallest variance. Therefore, by taking off-policy demonstration samples, we are
motivating importance sampling to use distributions that have higher reward than current
policy in order to get closer to the optimal distribution. To construct the sampling distribu-
tion, first we need to figure out which distribution the demonstrations come from. First, we
could use supervised behavioral cloning to learn πdemo . If the demonstration is from multiple
distribution, we could instead use fusion distribution by
1 X
q(x) = qi (x)
M i

14.5.3 Q-learning with Demonstrations

Since Q-learning is already off-policy, there is actually no need to bother with importance
weights like we did in policy gradients. Therefore, one simple solution is just drop demon-
strations into the replay buffer.
We can modify Alg. 12 slightly such that we initialize B with some demonstrations
data, and then do the same things as before.
14.5.4 Imitation as an Auxiliary Loss Function
Recall the imitation learning maximum likelihood training objective is
X
log πθ (a|s)
(s,a)∼Ddemo

and the RL objective is

Eπθ [r(s, a)]
to combine the two, we can come up with a hybrid objective:
X
Eπθ [r(s, a)] + λ log πθ (a|s)
(s,a)∼Ddemo
Chapter 15: Offline RL

RL is fundamentally an “active” learning paradigm: the agent needs to collect its own dataset
to learn meaningful policies. However, this might be unsafe or expensive in real world
problems (e.g., autonomous driving). Therefore it would be more data-efficient to learn
from a previously collected static dataset, which we call Offline (Batch) RL.

15.1 Offline RL Performance

In regular supervised learning problems such as classification, the algorithm can do as good
as the dataset. But in offline RL, due to the “stitching” property, it can sometimes do better
than the dataset. In fact, one can show that Q-learning recovers optimal policy from random
data.

85
Bibliography

[1] S. Ross, G. Gordon, and D. Bagnell, “A reduction of imitation learning and structured
prediction to no-regret online learning,” in Proceedings of the fourteenth international
conference on artificial intelligence and statistics, 2011, pp. 627–635.
[2] P. de Haan, D. Jayaraman, and S. Levine, “Causal confusion in imitation learning,”
arXiv preprint arXiv:1905.11979, 2019.
[3] P. Thomas, “Bias in natural actor-critic algorithms,” in International conference on
machine learning, 2014, pp. 441–448.
[4] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and
M. Riedmiller, “Playing atari with deep reinforcement learning,” arXiv preprint
arXiv:1312.5602, 2013.
[5] R. Munos, T. Stepleton, A. Harutyunyan, and M. Bellemare, “Safe and efficient off-
policy reinforcement learning,” in Advances in Neural Information Processing Systems,
2016, pp. 1054–1062.
[6] S. Gu, T. Lillicrap, I. Sutskever, and S. Levine, “Continuous deep q-learning with model-
based acceleration,” in International Conference on Machine Learning, 2016, pp. 2829–
2838.
[7] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and
D. Wierstra, “Continuous control with deep reinforcement learning,” arXiv preprint
arXiv:1509.02971, 2015.
[8] J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz, “Trust region policy
optimization,” in International conference on machine learning, 2015, pp. 1889–1897.
[9] C. B. Browne, E. Powley, D. Whitehouse, S. M. Lucas, P. I. Cowling, P. Rohlfshagen,
S. Tavener, D. Perez, S. Samothrakis, and S. Colton, “A survey of monte carlo tree
search methods,” IEEE Transactions on Computational Intelligence and AI in games,
vol. 4, no. 1, pp. 1–43, 2012.
[10] X. Guo, S. Singh, H. Lee, R. L. Lewis, and X. Wang, “Deep learning for real-time
atari game play using offline monte-carlo tree search planning,” in Advances in neural
information processing systems, 2014, pp. 3338–3346.
[11] M. Watter, J. Springenberg, J. Boedecker, and M. Riedmiller, “Embed to control: A
locally linear latent dynamics model for control from raw images,” in Advances in neural
information processing systems, 2015, pp. 2746–2754.

86
BIBLIOGRAPHY 87

[12] M. Zhang, S. Vikram, L. Smith, P. Abbeel, M. J. Johnson, and S. Levine, “Solar:

Deep structured latent representations for model-based reinforcement learning,” arXiv
preprint arXiv:1808.09105, 2018.

[13] G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,”
arXiv preprint arXiv:1503.02531, 2015.

[14] A. A. Rusu, S. G. Colmenarejo, C. Gulcehre, G. Desjardins, J. Kirkpatrick, R. Pas-

canu, V. Mnih, K. Kavukcuoglu, and R. Hadsell, “Policy distillation,” arXiv preprint
arXiv:1511.06295, 2015.

[15] E. Parisotto, J. L. Ba, and R. Salakhutdinov, “Actor-mimic: Deep multitask and transfer
reinforcement learning,” arXiv preprint arXiv:1511.06342, 2015.

[16] R. Houthooft, X. Chen, Y. Duan, J. Schulman, F. De Turck, and P. Abbeel, “Vime:

Variational information maximizing exploration,” in Advances in Neural Information
Processing Systems, 2016, pp. 1109–1117.