Homework 7, Random Processes

Nguyen Minh Quan

Instructor: Dr. Pham Hai Ha

September 15, 2023

Problem 1. Let  
1/4 1/2 1/4
P = 1/3 0 2/3
1/2 0 1/2
be the transition matrix of a Markov chain {Xn }n≥0 . If P(X0 = 1) = 1/3, determine

(a) P(X4 = 3|X3 = 2);

(b) P(X3 = 1|X2 = 1);

(c) P(X0 = 1, X1 = 2);

(d) P(X0 = 1, X1 = 2, X2 = 3).

Solution. blank

(a) P(X4 = 3|X3 = 2) = P23 = 2/3;

(b) P(X3 = 1|X2 = 1) = P11 = 1/4;

(c) By the Total Probability formula,

1 1 1 1
P(X0 = 1, X1 = 2) = P(X0 = 1) · P(X1 = 2|X0 = 1) = · P12 = · = .
3 3 2 6

(d) By the Total Probability formula,

P(X0 = 1, X1 = 2, X2 = 3) = P(X0 = 1, X1 = 2) · P(X2 = 3|X0 = 1, X1 = 2)

1 1 1 2 1
= · P(X2 = 3|X1 = 2) = · P23 = · = .
6 6 6 3 9

Problem 2. Consider a system that can be in one of two possible states, S = {0, 1} . The transition
matrix is given by  
1/2 1/2
P =
1/3 2/3
Suppose that X0 = 0, i.e. the system is in state 0 at time n = 0.

(a) Draw the state transition diagram;

(b) Determine P(X3 = 1).

Solution. blank

1
 (a) The state transition diagram is given below.

HW/hw7.png

(b) Note that X0 = 0 and

 3  
3 1/2 1/2 29/72 43/72
P = = ,
1/3 2/3 43/108 65/108
so
3 43
P(X3 = 1) = P01 = .
72

Problem 3. A Markov chain {Xn }n≥0 with states 0, 1, 2 has the transition matrix
 
1/2 1/3 1/6
P =  0 1/3 2/3
1/2 0 1/2

If P(X0 = 0) = P(X0 = 1) = 1/4, determine E(X3 ).

Solution. Note that

 3  
1/2 1/3 1/6 13/36 11/54 47/108
P 3 =  0 1/3 2/3 =  4/9 4/27 11/27  ,
1/2 0 1/2 5/12 2/9 13/36

yielding
2
X 2
X
3
P(X3 = 0) = P(X0 = i) · P(X3 = 0|X0 = i) = P(X0 = i) · Pi0
i=0 i=0
1 13 1 4 1 5 59
= · + · + · = ,
4 36 4 9 2 12 144

2
X 2
X
3
P(X3 = 1) = P(X0 = i) · P(X3 = 1|X0 = i) = P(X0 = i) · Pi1
i=0 i=0
1 11 1 4 1 2 43
= · + · + · = ,
4 54 4 27 2 9 216
59 43 169
P(X3 = 2) = 1 − P(X3 = 0) − P(X3 = 1) = 1 − − = ,
144 216 432
thus the distribution of X3 is given by
x 0 1 2
P(X3 = x) 59/144 43/216 169/432
implying
59 43 169 53
E(X3 ) = 0 · +1· +2· = ≈ 0.982.
144 216 432 54

Problem 4. For gambler’s ruin, assume that the gambler’s initial state is $3 and the gambler plays
until either gaining $8 or going bust. At each play the gambler wins $1, with probability 0.6, or
loses $1, with probability 0.4. Find the gambler’s expected fortune after four plays.

2
Solution. Denote Xn the fortune after the ith play, then {Xn }n≥0 is a Markov chain of states
0, 1, ..., 8 and transition matrix

1 0 0 0 0 0 0 0 0
 
0.4 0 0.6 0 0 0 0 0 0
 
 0 0.4 0 0.6 0 0 0 0 0
 
0 0 0.4 0 0.6 0 0 0 0
 
P =
 0 0 0 0.4 0 0.6 0 0 0 

0 0 0 0 0.4 0 0.6 0 0
 
0 0 0 0 0 0.4 0 0.6 0 
 
0 0 0 0 0 0 0.4 0 0.6
0 0 0 0 0 0 0 0 1

Note that
1 0 0 0 0 0 0 0 0
 
 62/125 72/625 0 162/625 0 81/625 0 0 0 
 
148/625 0 36/125 0 216/625 0 81/625 0 0 
 
 8/125 72/625 0 216/625 0 216/625 0 81/625 0 
4
 
P =  16/625
 0 96/625 0 216/625 0 216/625 0 81/625 
,
 0 16/625 0 96/625 0 216/625 0 162/625 27/125 
 
 0 0 16/625 0 96/625 0 36/125 0 333/625
 
 0 0 0 16/625 0 72/625 0 72/625 93/125 
0 0 0 0 0 0 0 0 1

we obtain the distribution of X4 as

x 0 1 3 5 7
P(X4 = x) 8/125 72/625 216/625 216/625 81/625
implying
8 72 216 216 81
E(X4 ) = 0 · +1· +3· +5· +7· = 3.7872.
125 625 625 625 625

Problem 5. Simple random walk provides a model of the wealth process of a person who makes
a living by flipping a coin and making bets. Consider a gambler who at each play of the game has
probability p of winning one unit and probability 1 − p of losing one unit. Let Xi be the amount
that he gains at the ith play, then

P(Xi = 1) = p, P(Xi = −1) = 1 − p, ∀i ∈ N.

Assuming that successive plays of the game are independent. Let S0 denote an arbitrary integer
that we view as our gambler’s initial wealth and
n
X
Sn = S0 + Xi , ∀n ∈ N
i=1

then Sn − S0 can be considered as the net winnings after n wagers. If p = 0.4,

(a) Find the transition probability of the Markov chain {Sn }n≥0 .

(b) Given that S0 = 3, find the probability that S2 = 4.

(c) What is the probability that his wealth reaches $8 before going bust given that S0 = 3?

Solution. In this solution, we assume for convenience that 0 ∈ N.

3
(a) Assume that the gambler plays until going bust, the transition probability is given by


 0.4 0 < i = j − 1
0.6 0 < i = j + 1

Pi,j =

 1 i=j=0
0 otherwise


(b) Note that S0 = 3 is odd, then S1 = S0 ± 1 is even and S2 = S1 ± 1 is odd, thus S2 ̸= 4 since
4 is even. In other words,
P(S2 = 4|S0 = 3) = 0.
(c) Let A be the event that the gambler’s wealth reaches $8 before going bust, that is,
A = {∃n ∈ N : Sn = 8} .
For each i = 0, 8, denote Pi the probability that the gambler’s wealth reaches $8 before going
bust, given that his initial wealth is $i, i.e.
Pi = P(A|S0 = i).
Clearly P0 = 0 and P8 = 1. Now pick an i ∈ N ∩ [0, 8] and consider a scenario in which the
gambler’s wealth is $i at some time t > 0, but his wealth never exceeded 7 during the time
horizon [0, t − 1]. Since all successive plays are independent, what happens after time t does
not depend on the result of previous plays. Hence, we can drop all these assumptions and
consider a new timeline T in which all parameters stay the same except two things:
• The initial time (of the new timeline T ) is at time t of the original timeline;
• The initial wealth (of the new timeline T ) is $i;
Then the probability that A occurs in the new timeline T is the same as the probability that
A occurs in the current timeline after time t, under the given constraints. In short,
 
P A St = i, max Sn < 8 = P(A|S0 = i) = Pi , ∀i = 1, 8. (1)
0≤n≤t−1

Equation (1) and the Total Probability formula implies that for any i = 1, 7,
Pi = P(A|S0 = i)
= P(S1 = i + 1|S0 = i) · P(A|S0 = i = S1 − 1) + P(S1 = i − 1|S0 = i) · P(A|S0 = i = S1 + 1)
= Pi,i+1 · P(A|S0 = i + 1) + Pi,i−1 · P(A|S0 = i − 1) = Pi,i+1 · Pi+1 + Pi,i−1 · Pi−1
which yields the following system of equations


 P0 = 0, P8 = 1,



 P 1 = P 1,2 · P2 + P1,0 · P0 = 0.4 · P2
 P
 2

 = P 2,3 · P 3 + P2,1 · P1 = 0.4 · P3 + 0.6 · P1
P3 = P3,4 · P4 + P3,2 · P2 = 0.4 · P4 + 0.6 · P2


 P 4 = P4,5 · P5 + P4,3 · P3 = 0.4 · P5 + 0.6 · P3
P5 = P5,6 · P6 + P5,4 · P4 = 0.4 · P6 + 0.6 · P4



P6 = P6,7 · P7 + P6,5 · P5 = 0.4 · P7 + 0.6 · P5



P7 = P7,8 · P8 + P7,6 · P6 = 0.4 + 0.6 · P6


Solving the system gives

 
128 64 608 16 1688 532 4118
(P0 , P1 , P2 , P3 , P4 , P5 , P6 , P7 , P8 ) = 0, , , , , , , ,1
6305 1261 6305 97 6305 1261 6305
and P3 = 608/6305, i.e. the probability that the gambler’s wealth reaches $8 before going
bust given that S0 = 3 is 608/6305.