The Bellman Equation

1 Statement of the Problem

V (x) = sup F (x, y) + βV (y)
x,y

s.t. (1)

y ∈ Γ (x)

• Some terminology:

– The Functional Equation (1) is called a Bellman equation.

– x is called a state variable.
– G (x) = {y ∈ Γ (x) : V (x) = F (x, y) + βV (y)} is called a policy correspondence. It
spells out all the values of y that attain the maximum in the RHS of (1).
– If G (x) is single-valued (i.e. there is a unique optimum), G is called a policy function.

• Questions:

1. Does (1) have a solution?

2. Is it unique?
3. How do we find it?

2 The Bellman Equation as a Fixed-Point Problem

• Define the operator T by

T (f ) (x) = sup F (x, y) + βf (y)

x,y

s.t. y ∈ Γ (x)

• V can be defined as a fixed point of T , i.e. a function such that T (V ) (x) = V (x) ∀x

1
Econ 210, Fall 2013 Pablo Kurlat

• Does T have a fixed point? How do we find it?

Assumption 1. (Assumption 4.3 in SLP) X ⊆ Rn is convex. Γ : X ⇒ X is nonempty, compact-

valued and continuous.

Assumption 2. (Assumption 4.4 in SLP) F : X × X → R is bounded, i.e. ∃F̄ such that

F (x, y) < F̄ for all {x, y} with x ∈ X and y ∈ Γ(x).

• What space is the operator T defined in?

• Define the metric space (S, ρ) by

S ≡ {f : X → R continuous and bounded} (2)

with the norm

kf k = sup |f (x)|
x∈X

and thus the distance

ρ (f, g) = kf − gk = sup |f (x) − g (x)| (3)
x∈X

• Notice that T : S → S, i.e. if f is continuous and bounded, then g is continuous and

bounded.

• How do we know this?

1. T (f ) (x) is continuous
– Recall Theorem of the Maximum:
Proposition 1. (Theorem of the Maximum). X ⊆ Rl and Y ⊆ Rm . f : X ×Y → R
is continuous. Γ : X → Y is compact-valued and continuous. Then the function
h : X → R defined by
h(x) = max f (x, y)
y∈Γ(x)

is continuous; and the correspondence G : X → Y defined by

G(x) = {y ∈ Γ(x) : f (x, y) = h(x)}

is non-empty, compact-valued and upper hemi-continuous.

– Applied to this problem:
∗ f (x, y) becomes F (x, y) + βf (y)
∗ h (x) becomes T (f ) (x)
2. This is because F is bounded (Assumption 2) and f is bounded.

2
Econ 210, Fall 2013 Pablo Kurlat

• Now we want to show that T has a unique fixed point. Two steps:

1. Show that T is a contraction (Blackwell’s sufficient conditions hold)

2. Appeal to contraction mapping theorem

1. Blackwell’s sufficient conditions:

Proposition 2. (Blackwell’s sufficient conditions) X ⊆ Rl and B(X) is the space of bounded

functions f : X → R, with the sup norm. T is a contraction with modulus β if:
a. [Monotonicity] f, g ∈ B(X) and f (x) ≤ g(x) for all x ∈ X ⇒ (T f )(x) ≤ (T g)(x) for all
x ∈ X;
b. [Discounting] There exists some β ∈ (0, 1) such that [T (f + a)](x) ≤ (T f )(x) + βa for all
f ∈ B(X), a ≥ 0, x ∈ X.

These conditions hold in our problem because

(a) For any x

F (x, y) + βf (y) ≤ F (x, y) + βg (y)

sup F (x, y) + βf (y) ≤ sup F (x, y) + βg (y)
y∈Γ(x) y∈Γ(x)

T (f ) (y) ≤ T (g) (y)

(b)

T (f + a) (x) = sup F (x, y) + β [f (y) + a]

y∈Γ(x)

= T (f ) (x) + βa

2. Contraction Mapping Theorem

Proposition 3. (Contraction Mapping Theorem). If (S, ρ) is a complete metric space and

T : S → S is a contraction mapping with modulus β, then:

a. T has exactly one fixed point V in S;

b. For any V0 ∈ S, ρ(T n V0 , V ) ≤ β n ρ(V0 , V ), n = 0, 1, 2, ....

• The only missing step is to show that (S, ρ) defined by (2) and (3) indeed constitutes a
complete metric space. (SLP Thm 3.1). Notice that if we used
ˆ
ρ (f, g) = |f (x) − g (x)| dx

3
Econ 210, Fall 2013 Pablo Kurlat

then (S, ρ) would NOT be a complete metric space. (SLP exercise 3.6.a., due this week).

Proposition 4. (SLP 4.6) If Assumptions

refregular and 2 hold, then T has a unique fixed point in S, i.e. there is a unique continuous
bounded function that solves (1).

Proof. From Contraction Mapping Theorem, knowing that Blackwell’s sufficient conditions are
met.

Proposition 5. The policy correspondence G(x) = {y ∈ Γ (x) : V (x) = F (x, y) + βV (y)} is compact-
valued and u.h.c.

Proof. From the Theorem of the Maximum

Proposition 6. If Assumptions
refregular and 2 hold, then V is the value function of the sequence problem.

Proof. V solves (1) and, because V is bounded, then limT →∞ β T V (xT ) = 0 ∀x̃Π (x0 ) , ∀x0 ∈ X,
so the sufficient conditions for Theorem SLP 4.3 hold.

3 Proving Properties of V
Proposition 7. If (S, ρ) is a complete metric space and S 0 is a closed subset of S, then S 0 is a
complete metric space

Proof. SLP Exercise 3.6.b. (due this week)

Proposition 8. (SLP Corollary 1, page 52). Let (S, ρ) be a complete metric space and T : S → S
be a contraction mapping with fixed point V ∈ S.

1. If S 0 is a closed subset of S and T (f ) ∈ S 0 for all f ∈ S 0 , then V ∈ S 0

2. If in addition S 00 ⊆ S 0 and T (f ) ∈ S 00 for all f ∈ S 0 , then V ∈ S 00

Proof.

1. Choose V0 ∈ S 0 . T n (V ) is a sequence in S 0 converging to V . Since S 0 is closed, V ∈ S 0 .

2. Since V ∈ S 0 , then T (V ) ∈ S 00 . But T (V ) = V so V ∈ S 00

4
Econ 210, Fall 2013 Pablo Kurlat

• Example:

– S: all continuous functions f : [a, b] → R

– S 0 : all increasing functions f : [a, b] → R
– S 00 : all strictly increasing functions f : [a, b] → R

• Note: we require the subset S 0 to be closed but not the sub-subset S 00

• In our example:

1. If T maps increasing functions into increasing functions, then the fixed point must be
an increasing function
2. If T maps increasing functions into strictly increasing functions, then the fixed point
must be a strictly increasing function

– What the result does not say is that if T maps strictly increasing functions into strictly
increasing functions, then the fixed point must be a strictly increasing function (because
the set of strictly increasing functions is not closed)

3.1 V increasing
Assumption 3. (Assumption 4.5 in SLP) F (x, y) is strictly increasing in x.

Assumption 4. (Assumption 4.6 in SLP) x ≤ x0 implies Γ(x) ⊆ Γ(x0 )

• Do Assumptions (3) and (4) hold in the Neoclassical model?

Proposition 9. (SLP 4.7). Suppose Assumptions (1)-(4) hold. Then V is strictly increasing.

Proof. Let x0 > x and f ∈ S.

T (f ) (x) = max F (x, y) + βf (y)

y∈Γ(x)

≤ max0 F (x, y) + βf (y)

y∈Γ(x )

< max0 F (x0 , y) + βf (y)

y∈Γ(x )

= T (f ) (x0 )

This implies that T maps any continuous bounded function into a strictly increasing function.
Proposition 8 gives the result.

5
Econ 210, Fall 2013 Pablo Kurlat

3.2 V concave
Assumption 5. (Assumption 4.7 in SLP) F (x, y) is strictly concave.

Assumption 6. (Assumption 4.8 in SLP) Γ is convex

Proposition 10. (SLP 4.8) Suppose Assumptions (1), (2), (5) and (6) hold. Then V is strictly
concave and G is continuous and single-valued.

Proof. We want to show that T maps concave functions into strictly concave functions. Strict
concavity of G follows by Proposition 8.

• Let x0 6= x1 and xθ = θx0 + (1 − θ) x1 for θ ∈ (0, 1).

• Let y0 ∈ Γ (x0 ) be such that T (f ) (x0 ) = F (x0 , y0 ) + βf (y0 ) and similarly y1 ∈ Γ (x1 ) be
such that T (f ) (x1 ) = F (x1 , y1 ) + βf (y1 )

• Then:

T (f ) (xθ ) ≥ F (xθ , yθ ) + βf (yθ )

(Γ concave makes xθ , yθ feasible)
> [θF (x0 , y0 ) + (1 − θ) F (x1 , y1 )] + β [θf (y0 ) + (1 − θ) f (y1 )]
(f concave and F strictly concave)
= θ [F (x0 , y0 ) + βf (y0 )] + (1 − θ) [F (x1 , y1 ) + βf (y1 )]
(rearranging)
= T (f ) (x0 ) + T (f ) (x1 )
(by assumption)

• G single-valued follows from strict concavity

• G continuous follows from Theorem of the Maximum

4 Is V differentiable? (Benveniste & Scheinkman, 1979)

• Cannot use same proof technique:

– Space of differentiable functions is not closed

– T does not necessarily map f into a differentiable function

• Instead, rely on the following result

6
Econ 210, Fall 2013 Pablo Kurlat

Proposition 11. Suppose V : X → R is concave. Let x0 be an interior and D be a neighborhood

around x0 . Suppose exists w : D → R such that:

1. w (x) ≤ V (x)

2. V (x0 ) = w (x0 )

3. w is differentiable at x0

Then V is differentiable at x0

Proof. Any subgradient p of V at x0 must satisfy;

p · (x − x0 ) ≥ V (x) − V (x0 ) ≥ w (x) − V (x0 ) ≥ w (x) − w (x0 )

but since w is differentiable, then p is unique, which implies V differentiable.

• (Graph)

• This result is useful to establish the following:

Proposition 12. (SLP 4.11) Suppose Assumptions (1), (2), (5) and (6) hold and F is continuosly
differentiable. Then V is differentiable and

Vi (x0 ) = Fi (x0 , g (x0 ))

Proof. Define
w (x) = F (x, g (x0 )) + βV (g (x0 ))

• w is concave, differentiable and satisfies

w (x) ≤ F (x, g (x)) + βV (g (x)) ∀x

⇒ w (x) ≤ max F (x, g (x)) + βV (g (x))
y∈Γ(x)

= V (x)

and
w (x0 ) = V (x0 )

• The result then follows from (11)