Lecture 5: Conditional Expectation

John Sylvester Nicolás Rivera Luca Zanetti Thomas Sauerwald

Lent 2020
Outline

Conditional Expectation

Quick-Sort

Coin Flips and Balls-in-Bins

Lecture 5: Conditional Expectation 2

Conditional Probability and Expectation

Given two events A, B with P[ A ] > 0 define the Conditional Probability

P[ B|A ] = P[ B ∩ A ] /P[ A ] .

If P[ A ] = 0, the usual convention is that P[ B|A ] = 0.

Given a discrete random variable Y , we define its Conditional Expectation
with respect to the event A by
X X
E[ Y |A ] = Y (ω)P[ {ω}|A ] = bP[ Y = b|A ]
ω∈Ω b

When the event A = {X = a} where X is another discrete random

variable, we define the function f (a) by

f (a) = E[ Y |X = a ] ,

We define the conditional expectation E[ Y |X ], as the random variable

that takes the value E[ Y |X = a ] then X = a, i.e. f (X ).

Lecture 5: Conditional Expectation 3

Important Remarks

The conditional expectation E[ Y |A ] of Y w.r.t an event A is a

deterministic number .
The conditional expectation E[ Y |X ] of Y w.r.t a random variable X is a
random variable .
In the definition of E[ Y |X ] above X can be a random vector (X1 , . . . , XN ).
Example: Single Dice
Let Y be 1 if the dice rolls 1 and 0 otherwise
Let X1 be 1 if the dice shows odd number, 0 otherwise
Let X2 be 1 is the dice shows a number ≤ 2 , 0 otherwise

E[ Y |X1 ] = X1 /3 E[ Y |X2 ] = X2 /2 E[ Y |(X1 , X2 ) ] = X1 · X2

Lecture 5: Conditional Expectation 4

Important Remarks (continued)

The conditional expectation E[ Y |X ] is always a function of X .

Behind conditional expectation there is the notion of information.

The standard notion of expectation E[ Y ] can be thought of as

’the best estimate of a random variable Y given no information about it,’

while the conditional expectation E[ Y |I ] given I can be thought of as

’the best estimate of a random variable Y given information I,’

where I above may be an event or a random variable etc.

Lecture 5: Conditional Expectation 5

Conditional Expectation: Two Dice
Suppose we independently roll two standard 6-sided dice. Let X1 and X2 the
observed number in the first and second dice respectively. Then,
E[ X1 + X2 |X1 ] = 3.5 + X1 .
Why? Because if X1 = a then
12
X
E[ X1 + X2 |X1 = a ] = bP[ X1 + X2 = b|X1 = a ]
b=1
12
X
= bP[ X1 + X2 = b, X1 = a ] /P[ X1 = a ]
b=1
12
X
= bP[ X2 = b − a, X1 = a ] /P[ X1 = a ]
b=1
12
X
(X1 independent of X2 ) = bP[ X2 = b − a ]
b=1
6
X
= (c + a)P[ X2 = c ]
c=1
= 3.5 + a
Lecture 5: Conditional Expectation 6
Conditional Expectation: Properties

(1) E[ E[ Y |X ] ] = E[ Y ].
(2) E[ 1|X ] = 1
(3) Linearity:
For any constant c ∈ R, E[ cY |X ] = cE[ Y |X ]
E[ Y + Z |X ] = E[ Y |X ] + E[ Z |X ]
(4) If X is independent of Y , then E[ Y |X ] = E[ Y ] .
(5) if Y is a function of X , i.e. Y = f (X ), then E[ YZ |X ] = Y E[ Z |X ].
Particularly, E[ X |X ] = X
(6) Tower Property:
E[ E[ X |(Z , Y ) ] |Y ] = E[ X |Y ].
(7) Jensen Inequality:
if f is a convex real function, then f (E[ X |Y ]) ≤ E[ f (X )|Y ].
These properties greatly simplify calculations. Example: for our two dice
p3 p5,p4
E[ X1 + X2 |X1 ] = E[ X1 |X1 ] + E[ X2 |X1 ] = X1 + E[ X2 ] = X1 + 3.5

Lecture 5: Conditional Expectation 7

Exercise: Prove the properties from slide 7

Example: we can prove Property (1), that is E[ E[ Y |X ] ] = E[ Y ], as follows:

X
E[ E[ Y |X ] ] = E[ Y |X ](ω) · P[ {ω} ]
ω∈Ω
X
= E[ Y |X = x ] P[ X = x ]
x
XX
= y P[ Y = y |X = x ] P[ X = x ]
x y
XX
= y P[ Y = y , X = x ]
x y
X
= y P[ Y = y ]
y

= E[ Y ]

Bonus Exercise: Prove Property (1) using Properties (4) and (2).

Lecture 5: Conditional Expectation 8

Outline

Conditional Expectation

Quick-Sort

Coin Flips and Balls-in-Bins

Lecture 5: Conditional Expectation 9

Quick-Sort with Random Pivot

Algorithm: QuickSort
Input: Array of different number A.
Output: array A sorted in increasing order
Pick an element uniformly from the array, the so-called pivot .
If |A| = 0 or |A| = 1; return A.
Else
Generate two subarrays A1 and A2 :
A1 contains the elements that are smaller than the pivot ;
A2 contains the elements that are greater than the pivot ;
Recursively sort A1 and A2 .

Let Cn be the number of comparisons made by Qucik-Sort on n elements.

Recall that Nicolas showed something along the lines of

P[ Cn ≥ 21n log n ] = 1/n.

What is E[ Cn ] - the expected number of comparisons?

Lecture 5: Conditional Expectation 10

The expected number of comparisons in Quick-Sort
Let Mn = E[ Cn ] be the expected number of comparisons needed by
quick-sort to sort n distinct values. Conditioning on the rank of the pivot gives
n
X   1
Mn = E Cn pivot selected is jth smallest value ·
n
j=1

If the initial pivot selected is the jth smallest value, then the set of values
smaller than it has size j − 1, and the set of values greater has size n − j.
Hence, as n − 1 comparisons with the pivot must be made, we have
n n−1
X 1 2X
Mn = (n − 1 + Mj−1 + Mn−j ) = n − 1 + Mj .
n n
j=1 j=1

Since M0 = 0.
Thus
(n + 1)Mn+1 − nMn = 2n + 2Mn .
Or equivalently
(n + 1)Mn+1 = 2n + (n + 2)Mn .

Lecture 5: Conditional Expectation 11

The expected number of comparisons in Quick-Sort

Rearranging (n + 1)Mn+1 = 2n + (n + 2)Mn we have

Mn+1 2n Mn
= +
n+2 (n + 1)(n + 2) n+1
2n 2(n − 1) Mn−1
= + +
(n + 1)(n + 2) n(n + 1) n
n−1
X n−k
=2
(n + 1 − k )(n + 2 − k )
k =0
n
Since M1 = 0.
X i
=2
(i + 1)(i + 2)
i=0
" n n
#
X 2 X 1
=2 − ∼ 2 log n.
i +2 i +1
i=1 i=1

Thus Quick-Sort makes E[ Cn ] ∼ 2n log n comparisons in expectation.

Lecture 5: Conditional Expectation 12

Outline

Conditional Expectation

Quick-Sort

Coin Flips and Balls-in-Bins

Lecture 5: Conditional Expectation 13

Example: Expectation of a Geometric Random Variable
Suppose X1 , X2 , . . . , is an infinite sequence of independent Bernoulli Ber(p)
random variables with parameter p. That is
P[ Xi = 1 ] = p, P[ Xi = 0 ] = 1 − p.
We shall think of the Xi0 s as coin flips.

Let G = min{k ≥ 1 : Xk = 1}, the number of coin flips until we get a head.

G has geometric distribution Geo(p) with parameter p. Indeed,

P[ G = k ] = p(1 − p)k −1 .

The expectation of G is given by the formula

∞
X
E[ G ] = kp(1 − p)k −1
k =1

Let say that we forgot how to compute sums of this type...

Lecture 5: Conditional Expectation 14

We can compute E[ G ] by other means.
p1
E[ G ] = E[ E[ G|X1 ] ]
Conditional on X1 ,
G = X1 + (1 − X1 )(1 + G0 ),
where G0 is the number of coins we need to wait to see a head after the
first coin.
p3,p5
E[ X1 + (1 − X1 )(1 + G0 )|X1 ] = X1 + (1 − X1 )E[ 1 + G0 |X1 ]
G0 has geometric distribution of parameter p and it is independent of X1 .
Hence
 p4 
E 1 + G0 |X1 = E 1 + G0 = 1 + E[ G ]
 

Solve
E[ G ] = p + (1 − p)(1 + E[ G ])

To give
E[ G ] = 1/p.

Lecture 5: Conditional Expectation 15

Example: Balls into Bins

Suppose we have n bins but a random number of balls, say M. Suppose M

has finite expectation.What is the expected number of balls in the first bin?
Set Up:
Balls are assigned to bins uniformly and independently at random
Let Xi = 1 if the ball i falls in bin 1, 0 otherwise
PM
The total number of balls in bin 1 is i=1 Xi
M is a random variable, and M is independent of all Xi ’s
Question Rephrased
Let Xi be i.i.d. and M beh independent
i of {Xi }i≥0 . If E[ Xi ] < ∞ and
PM
E[ M ] < ∞ then what is E i=1 Xi ?

Lecture 5: Conditional Expectation 16

 PM
Expectation of the Compound Random Variable i=1 Xi

hP i h hP ii
M P1 M
We shall use E i=1 Xi = E E i=1 Xi |M . Observe that for any k ∈ N

" M
# " k
#
X X
f (k ) := E Xi M = k =E Xi M = k
i=1 i=1
k k
P3  P4 X
X 
= E Xi M = k = E[ Xi ] = k · E[ Xi ] ,
i=1 i=1
hP i
M
since Xi are all equidistributed. Thus E i=1 Xi |M = f (M) = M · E[ Xi ] .

To conclude we have
" M #
P3
X
E Xi = E[ ME[ Xi ] ] = E[ M ] E[ Xi ] .
i=1

Lecture 5: Conditional Expectation 17