Lecture 4: Sequential decision making

Simon Parsons

Department of Informatics
Kings College London

Version 1

1 / 92
Today

Introduction
Probabilistic Reasoning I
Probabilistic Reasoning II
Sequential Decision Making
Argumentation I
Argumentation II
Temporal Probabilistic Reasoning
Game Theory
(A peek at) Machine Learning
AI & Ethics

2 / 92
What to do?

(40 Acres and a Mule Filmworks/Universal Pictures)

3 / 92
What to do?

(mystorybook.com/books/42485)

4 / 92
What to do?

(Sebastian Thrun & Chris Urmson/Google )

5 / 92
Sequential decision making?

(mystorybook.com/books/42485)
One decision leads to another.
Each decision depends on the ones before, and affects
the ones after.

6 / 92
How to decide what to do

Start simple.
Single decision.
Consider being offered a bet in which you pay 2 if an odd
number is rolled on a die, and win 3 if an even number
appears.
Is this a good bet?

7 / 92
How to decide what to do

Consider being offered a bet in which you pay 2 if an odd

number is rolled on a die, and win 3 if an even number
appears.
Is this a good bet?
To analyse this, we need the expected value of the bet.

8 / 92
How to decide what to do

We do this in terms of a random variable, which we will call X .

X can take two values:
3 if the die rolls odd
2 if the die rolls even
And we can also calculate the probability of these two values
PrpX 3q 0.5
PrpX 2q 0.5

9 / 92
How to decide what to do

The expected value is then the weighted sum of the values,

where the weights are the probabilities.
Formally the expected value of X is defined by:

E pX q k PrpX k q
k

where the summation is over all values of k for which

PrpX k q , 0.
Here the expected value is:

E pX q 0.5 3 ` 0.5 2

Thus the expected value of X is 0.5, and we take

this to be the value of the bet.

10 / 92
How to decide what to do

Do you take the bet?

Compare that 0.5 with not taking the bet.
Not taking the bet has (expected) value 0

11 / 92
How to decide what to do

0.5 is not the value you will get.

You can think of it as the long run average if you were offered
the bet many times.

12 / 92
How to decide what to do

(fivethirtyeight.com)
13 / 92
How to decide what to do

Another bet: you get 1 if a 2 or a 3 is rolled, 5 if a six is

rolled, and pay 3 otherwise.
The expected value here is:

E pX q 0.333 1 ` 0.166 5 ` 0.5 3

which is 0.33.

14 / 92
Example

Pacman is at a T-junction
Based on their knowledge, estimates that if they go Left:
Probability of 0.3 of getting a payoff of 10
Probability of 0.2 of getting a payoff of 1
Probability of 0.5 of getting a payoff of -5

What is the expected value of Left?

15 / 92
Example

16 / 92
How an agent might decide what to do

Consider an agent with a set of possible actions A .

Each a P A has a set of possible outcomes sa .
Which action should the agent pick?

17 / 92
How an agent might decide what to do

The action a which a rational agent should choose is that

which maximises the agents utility.
In other words the agent should pick:

a arg max upsa q

a PA

The problem is that in any realistic situation, we dont know

which sa will result from a given a, so we dont know the utility
of a given action.
Instead we have to calculate the expected utility of each
action and make the choice on the basis of that.

18 / 92
How an agent might decide what to do

In other words, for each action a with a set of outcomes sa ,

the agent should calculate:

E pupa qq ups 1 q. Prpsa s 1 q
s 1 Psa

and pick the best.

s a2 s6

a1
s5
s3 s4
s1
s2

19 / 92
How an agent might decide what to do

That is it picks the action that has the greatest expected utility.
The right thing to do.

(40 Acres and a Mule Filmworks/Universal Pictures)

Here rational means rational in the sense of maximising
expected utility.

20 / 92
Non-deterministic

Note that we are dealing with non-deterministic actions here.

s a2 s6

a1
s5
s3 s4
s1
s2
A given action has several possible outcomes.
We dont know, in advance, which one will happen.

21 / 92
Non-deterministic

(fivethirtyeight.com)
A lot like life.

22 / 92
Other notions of rational

There are other criteria for decision-making than maximising

expected utility.
One approach is to look at the option which has the least-bad
worst outcome.
This maximin criterion can be formalised in the same
framework as MEU, making the rational (in this sense) action:

a arg maxt min

1
ups 1 qu
a PA s Psa

Its effect is to ignore the probability of outcomes and

concentrate on optimising the worst case outcome.

23 / 92
Other notions of rational

The opposite attitude, that of optimisitic risk-seeker, is

captured by the maximax criterion:

a arg maxtmax ups 1 qu

a PA s 1 Psa

This will ignore possible bad outcomes and just focus on the
best outcome of each action.

24 / 92
Example

Pacman is at a T-junction
Based on their knowledge, estimates that if they go Left:
Probability of 0.3 of getting a payoff of 10
Probability of 0.2 of getting a payoff of 1
Probability of 0.5 of getting a payoff of -5

If they go Right:
Probability of 0.5 of getting a payoff of -5
Probability of 0.4 of getting a payoff of 3
Probability of 0.1 of getting a payoff of 15

Should they choose Left or Right (MEU)?

25 / 92
Example

26 / 92
Sequential decision problems

These approaches give us a battery of techniques to apply to

individual decisions by agents.
However, they arent really sufficient.
Agents arent usually in the business of taking single
decisions
Life is a series of decisions.
The best overall result is not necessarily obtained by a greedy
approach to a series of decisions.
The current best option isnt the best thing in the long-run.

27 / 92
Sequential decision problems

Otherwise Id only ever eat cherry pie

(pillsbury.com)
(Damn fine pie.)

28 / 92
An example

The agent has to pick a sequence of actions.

A ps q tUp , Down, Left , Right u

for all states s.

29 / 92
An example

The world is fully observable.

End states have values `1 or 1.

30 / 92
An example

If the world were deterministic, the choice of actions would be

easy here.
Up , Up , Right , Right , Right
But actions are stochastic.

31 / 92
An example

80% of the time the agent moves as intended.

20% of the time the agent moves perpendicular to the
intended direction. Half the time to the left, half the time to the
right.
The agent doesnt move if it hits a wall.

32 / 92
An example

So Up , Up , Right , Right , Right succeeds with probability:

0.85 0.32768

33 / 92
An example

Also a small chance of going around the obstacle the other

way.

34 / 92
An example

We can write a transition model to describe these actions.

Since the actions are stochastic, the model looks like:

P ps 1 |s , a q

where a is the action that takes the agent from s to s 1 .

Transitions are assumed to be (first order) Markovian.
They only depend on the current and next states.
So, we could write a large set of probability tables that would
describe all the possible actions executed in all the possible
states.
This would completely specify the actions.

35 / 92
An example

The full description of the problem also has to include the

utility function.
This is defined over sequences of states runs in the
terminology of the first lecture.
We will assume that in each state s the agent receives a
reward R ps q.
This may be positive or negative.

36 / 92
An example

The reward for non-terminal states is 0.04.

We will assume that the utility of a run is the sum of the
utilities of states, so the 0.04 is an incentive to take fewer
steps to get to the terminal state.
(You can also think of it as the cost of an action).

37 / 92
How do we tackle this?

(Pendleton Ward/Cartoon Network)

38 / 92
Markov decision process

The overall problem the agent faces here is a Markov decision

process (MDP)
Mathematically we have
a set of states s P S with an initial state s0 .
A set of actions A ps q in each state.
A transition model P ps 1 |s , a q; and
A reward function R ps q.
Captures any fully observable non-deterministic environment
with a Markovian transition model and additive rewards.

Leslie Pack Kaelbling

39 / 92
Markov decision process

What does a solution to an MDP look like?

40 / 92
Markov decision process

A solution is a policy, which we write as .

This is a choice of action for every state.
that way if we get off track, we still know what to do.
In any state s, ps q identifies what action to take.

41 / 92
Markov decision process

Naturally wed prefer not just any policy but the optimum
policy.
But how to find it?
Need to compare policies by the reward they generate.
Since actions are stochastic, policies wont give the same
reward every time.
So compare the expected value.
The optimum policy is the policy with the highest expected
value.
At every stage the agent should do ps q.

42 / 92
Markov decision process

(40 Acres and a Mule Filmworks/Universal Pictures)

ps q is the right thing.

43 / 92
An example

(a) Optimal policy for the original problem.

(b) Optimal policies for different values of R ps q.

44 / 92
An example

R ps q 1.6284, life is painful so the agent heads for the exit,

even if is a bad state.
0.4278 R ps q 0.0850, life is unpleasant so the agent
heads for the `1 state and is prepared to risk falling into the
1 state.
0.0221 R ps q 0, life isnt so bad, and the optimal policy
doesnt take any risks.
R ps q 0, the agent doesnt want to leave.

45 / 92
How utilities are calculated

So far we have assumed that utilities are summed along a

run.
Not the only way.
In general we need to compute Ur prs0 , s1 , . . . , sn sq.
Can consider finite and infinite horizons.
Is it game over at some point?
Turns out that infinite horizons are mostly easier to deal with.
That is what we will use.

46 / 92
How utilities are calculated

Also have to consider whether utilities are stationary or

non-stationary.
Does the same state always have the same value?
Normally if we prefer one state to another
Passing the AI module to failing it
when we have the exam, today or next week, is irrelevant.
So we can reasonably assume utilities are stationary.

47 / 92
But are they?

Not clear that utilities are always stationary.

In truth, I dont always most want to eat cherry pie.

Despite this, we will assume that utilities are stationary.

48 / 92
How utilities are calculated

With stationary utilities, there are two ways to establish

Ur prs0 , s1 , . . . , sn sq from R ps q.
Additive rewards:

Ur prs0 , s1 , . . . , sn sq R ps0 q ` R ps1 q ` . . . ` R psn q

as above.
Discounted rewards:

Ur prs0 , s1 , . . . , sn sq R ps0 q ` R ps1 q ` . . . ` n R psn q

where the discount factor is a number between 0 and 1.

The discount factor models the preference of the agent
for current over future rewards.

49 / 92
How utilities are calculated

There is an issue with infinite sequences with additive,

undiscounted rewards.
What will the utility of a policy be?

50 / 92
How utilities are calculated

There is an issue with infinite sequences with additive,

undiscounted rewards.
What will the utility of a policy be?
8 or 8.
This is problematic if we want to compare policies.

51 / 92
How utilities are calculated

Some solutions are:

Proper policies
Average reward
Discounted rewards
As follows . . .

52 / 92
How utilities are calculated

Proper policies always end up in a terminal state eventually.

Thus they have a finite expected utility.

53 / 92
How utilities are calculated

We can compute the average reward per time step.

Even for an infinite policy this will (usually) be finite.

54 / 92
How utilities are calculated

With discounted rewards the utility of an infinite sequence is

finite:
8

Ur prs0 , s1 , . . . , sn sq t R p st q
t 0
8

t Rmax
t 0
Rmax

p1 q

where 0 1 and rewards are bounded by Rmax

55 / 92
Optimal policies

With discounted rewards we compare policies by computing

their expected values.
The expected utility of executing starting in s is given by:
ff
8
U ps q E

t R pSt q
t 0

where St is the state the agent gets to at time t.

St is a random variable and we compute the probability of all
its values by looking at all the runs which end up there after t
steps.

56 / 92
Optimal policies

The optimal policy is then:

arg max U ps q

It turns out that this is independent of the state the agent

starts in.

57 / 92
Optimal policies

Here we have the values of states if the agent executes an

optimal policy
U ps q

58 / 92
Optimal policies

Here we have the values of states if the agent executes an

optimal policy
U ps q

What should the agent do if it is in (3, 1)?

59 / 92
Example

Wrong!

60 / 92
Example

The answer is Left.

The best action is the one that maximises expected utility.
(You have to calculate the expected utility of all the actions to
see ahy Left is the best choice.)

61 / 92
Optimal policies

If we have these values, the agent has a simple decision

process
It just picks the action a that maximises the expected utility of
the next state:

P ps 1 |s , a qU ps 1 q

ps q arg max
a PA ps q
s1

Only have to consider the next step.

The big question is how to compute U ps q.

62 / 92
Optimal policies

Note that this is specific to the value of the reward R ps q for

non-terminal states different rewards will give different
values and policies.

63 / 92
Bellman equation

How do we find the best policy (for a given set of rewards)?

Turns out that there is a neat way to do this, by first computing
the utility of each state.
We compute this using the Bellman equation

U ps q R ps q ` max Prps 1 |s , a qU ps 1 q
a PA ps q
s1

is a discount factor.

64 / 92
Not this Bellman

Just the place for a Snark! the Bellman cried,

As he landed his crew with care;
Supporting each man on the top of the tide
By a finger entwined in his hair.

Just the place for a Snark! I have said it twice:

That alone should encourage the crew.
Just the place for a Snark! I have said it thrice:
What I tell you three times is true.

Lewis Carroll

(Mervyn Peakes illustrations to The Hunting of the Snark).

65 / 92
Bellman equation

Apply:

U ps q R ps q ` max Prps 1 |s , a qU ps 1 q
a PA ps q
s1

and we get:

66 / 92
Bellman equation

U p1, 1q 0.04`
max r0.8U p1, 2q ` 0.1U p2, 1q ` 0.1U p1, 1q, pUp q
0.9U p1, 1q ` 0.1U p1, 2q, pLeft q
0.9U p1, 1q ` 0.1U p2, 1q, pDownq
0.8U p2, 1q ` 0.1U p1, 2q ` 0.1U p1, 1qs pRight q

67 / 92
Value iteration

In an MDP wth n states, we will have n Bellman equations.

(Pendleton Ward/Cartoon Network)

Hard to solve these simultaneously because of the
max operation
Makes them non-linear

68 / 92
Value iteration

Luckily an iterative approach works.

Start with arbitrary values for states and apply the Bellman
update:

Ui `1 ps q R ps q ` max P ps 1 |s , a qUi ps 1 q
a PA ps q
s1

simultaneously to all the states.

Continue until the values of states do not change.
After an infinite number of applications, the values are
guaranteed to converge on the optimal values.

69 / 92
Value iteration

How the values of states change as updates occur.

70 / 92
Value iteration

U p4, 3q is pinned to 1.
U p3, 3q quickly settles to a value close to 1
U p1, 1q becomes negative, and then grows as positive utility
form the goal feeds back to it.

71 / 92
Rewards

The example so far has a negative reward R ps q for each state.

Encouragement for an agent not to stick around.
Can also think of R ps q is being the cost of moving to the next
state (where we obatin the utility):

R ps q c ps , a q

where s is the action used.

Bellman becomes:

Ui `1 ps q max p P ps 1 |s , a qUi ps 1 qq c ps , a q
a PA ps q
s1

Note that the action can be dependent on the state.

72 / 92
Policy iteration

Rather than compute optimal utility values, policy iteration

looks through the space of possible policies.
Starting from some initial policy 0 we do:
Policy evaluation
Given a policy i , calculate Ui ps q.
Policy improvement
Given Ui ps q, compute i `1
We will look at each of these steps in turn.
But not in order.

73 / 92
Policy improvement

Easy
Calculate a new policy i `1 by applying:

i `1 ps q arg max P ps 1 |s , a qUi ps 1 q
a PA ps q
s1

For each state we do a one-step lookahead.

A simple decision.

74 / 92
Policy evaluation

How do we calculate the utility of each step given the policy

i ?
Turns out not to be so hard.
Given a policy, the choice of action in a given state is fixed
(that is what a policy tells us) so:

Ui ps q R ps q ` P ps 1 |s , i ps qqUi ps 1 q
s1

Again there are lots of simultaneous equations, but now they

are linear (no max) and so standard linear algebra solutions
will work.

75 / 92
Policy iteration

Put these together to get:

Starting from some initial policy 0 we do:
1 Policy evaluation
Compute:

Ui ps q R ps q ` P ps 1 |s , i ps qqUi ps 1 q
s1

for every state.

2 Policy improvement
Calculate a new policy i `1 by applying:

i `1 ps q arg max P ps 1 |s , a qUi ps 1 q
a PA ps q
s1

for every state s.

Until convergence.

76 / 92
Policy iteration

The iteration will terminate when there is no improvement in

utility from one iteration to the next.
At this point the utility Ui is a fixed point of the Bellman update
and so i must be optimal.

77 / 92
Policy evaluation

There is a problem with the policy evaluation stage of the

policy iteration approach.
If we have n states, we have n linear equations with n
unknowns in the evaluation stage.
Solution in O pn3 q.
For large n, can be a problem.
So, an approximate solution.

78 / 92
Approximate you say?

(Pendleton Ward/Cartoon Network)

79 / 92
Approximate policy evaluation

Run a simplified value iteration.

Policy is fixed, so we know what action to do in each state.
Repeat:

Ui `1 ps q R ps q ` P ps 1 |s , i ps qqUi ps 1 q
s1

a fixed number of times.

80 / 92
Modified policy iteration

Starting from some initial policy 0 we do:

1 Approximate policy evaluation
Repeat

Ui `1 ps q R ps q ` P ps 1 |s , i ps qqUi ps 1 q
s1

a fixed number of times.

2 Policy improvement

i `1 ps q arg max P ps 1 |s , a qUi ps 1 q
a PA ps q
s1

for every state s.

Until convergance
Often more efficient than policy iteration or
value iteration.

81 / 92
Solving MDPs

Have covered three methods for solving MDPs

Value iteration
(Exact)
Policy iteration
(Exact)
Modified policy iteration
(Approximate)
Which to use is somewhat problem specific.

82 / 92
Bellman redux

The Bellman equation(s)/update are widely used.

D. Romer, Its Fourth Down and What Does the Bellman

Equation Say? A Dynamic Programming Analysis of Football
Strategy, NBER Working Paper No. 9024, June 2002

83 / 92
Bellman redux

This paper uses play-by-play accounts of virtually all

regular season National Football League games for
1998-2000 to analyze teams choices on fourth
down between trying for a first down and kicking.
Dynamic programming is used to estimate the
values of possessing the ball at different points on
the field. These estimates are combined with data
on the results of kicks and conventional plays to
estimate the average payoffs to kicking and going for
it under different circumstances. Examination of
teams actual decisions shows systematic,
overwhelmingly statistically significant, and
quantitatively large departures from the decisions
the dynamic-programming analysis implies are
preferable.

84 / 92
Limitations of MDPs?

(Pendleton Ward/Cartoon Network)

85 / 92
Partially observable MDPs

MDPs made the assumption that the environment was fully

observable.
Agent always knows what state it is in.
The optimal policy only depends on the current state.
Not the case in the real world.
We only have a belief about the current state.
POMDPs extend the model to deal with partial observability.

86 / 92
Partially observable MDPs

Basic addition to the MDP model is the sensor model:

P pe |s q

probability of perceiving e in state s.

As a result of noise in the sensor model, the agent only has a
belief about which state it is in.
Probability distribution over the possible states.

The world is a POMDP

87 / 92
Partially observable MDPs

PpS q : P ps1,1 q 0.05, P ps1,2 q 0.01, . . .

88 / 92
Partially observable MDPs

The agent can compute its current belief as the conditional

probability distribution over the states given the sequence of
actions and percepts so far.

89 / 92
Partially observable MDPs

The agent can compute its current belief as the conditional

probability distribution over the states given the sequence of
actions and percepts so far.
We will come across this task again in Lecture 7
Filtering
Computing the state that matches best with a stream of
evidence.

90 / 92
Partially observable MDPs

If b ps q was the distribution before an action and an

observation, then afterwards the distribution is:

b 1 ps 1 q P pe |s 1 q P ps 1 |s , a qb ps q
s

Everything in a POMDP hinges on the belief state b.

Including the optimal action.
Indeed, the optimal policy is a mapping pb q from beliefs to
actions.
If you think you are next to the wall, turn left
The agent executes the optimal action given its beliefs,
receives a percept e and then recomputes the belief
state.

91 / 92
Partially observable MDPs

The big issue in solving POMDPs is that beliefs are

continuous.
When we solved MDPs, we could search through the set of
possible actions in each state to find the best.
To solve a POMDP, we need to look through the possible
actions for each belief state.
But belief is continuous, so there are a lot of belief states.
Exact solutions to POMDPs are intractable for even small
problems (like the example we have been using).
Need (once again) to use approximate techniques.

92 / 92
Mathematical!

(Pendleton Ward/Cartoon Network)

93 / 92
Summary

Today we looked at practical decision making for agents.

Practical in the sense that agents will need this kind of
decision making to do the things they need to do.
This built on the last lecture on probability, and extended that
with expected values.
We looked in detail at solutions for techniques that work in
fully observable worlds
MDPs
We also briefly mentioned the difficulties of extending this
work to partially observable worlds.

94 / 92