18.

600: Lecture 24
Covariance and some conditional
expectation exercises

Scott Sheffield

MIT
Outline

Covariance and correlation

Paradoxes: getting ready to think about conditional expectation

Outline

Covariance and correlation

Paradoxes: getting ready to think about conditional expectation

A property of independence

I If X and Y are independent then

E [g (X )h(Y )] = E [g (X )]E [h(Y )].
A property of independence

I If X and Y are independent then

E [g (X )h(Y )] = E [g (X )]E [h(Y )].
R R
I Just write E [g (X )h(Y )] = g (x)h(y )f (x, y )dxdy .
A property of independence

I If X and Y are independent then

E [g (X )h(Y )] = E [g (X )]E [h(Y )].
R R
I Just write E [g (X )h(Y )] = g (x)h(y )f (x, y )dxdy .
RSince f (x, y ) = fXR(x)fY (y ) this factors as
I

h(y )fY (y )dy g (x)fX (x)dx = E [h(Y )]E [g (X )].
Defining covariance and correlation

I Now define covariance of X and Y by

Cov(X , Y ) = E [(X E [X ])(Y E [Y ]).
Defining covariance and correlation

I Now define covariance of X and Y by

Cov(X , Y ) = E [(X E [X ])(Y E [Y ]).
I Note: by definition Var(X ) = Cov(X , X ).
Defining covariance and correlation

I Now define covariance of X and Y by

Cov(X , Y ) = E [(X E [X ])(Y E [Y ]).
I Note: by definition Var(X ) = Cov(X , X ).
I Covariance (like variance) can also written a different way.
Write x = E [X ] and Y = E [Y ]. If laws of X and Y are
known, then X and Y are just constants.
Defining covariance and correlation

I Now define covariance of X and Y by

Cov(X , Y ) = E [(X E [X ])(Y E [Y ]).
I Note: by definition Var(X ) = Cov(X , X ).
I Covariance (like variance) can also written a different way.
Write x = E [X ] and Y = E [Y ]. If laws of X and Y are
known, then X and Y are just constants.
I Then

Cov(X , Y ) = E [(X X )(Y Y )] = E [XY X Y Y X +X Y ] =

E [XY ] X E [Y ] Y E [X ] + X Y = E [XY ] E [X ]E [Y ].
Defining covariance and correlation

I Now define covariance of X and Y by

Cov(X , Y ) = E [(X E [X ])(Y E [Y ]).
I Note: by definition Var(X ) = Cov(X , X ).
I Covariance (like variance) can also written a different way.
Write x = E [X ] and Y = E [Y ]. If laws of X and Y are
known, then X and Y are just constants.
I Then

Cov(X , Y ) = E [(X X )(Y Y )] = E [XY X Y Y X +X Y ] =

E [XY ] X E [Y ] Y E [X ] + X Y = E [XY ] E [X ]E [Y ].
I Covariance formula E [XY ] E [X ]E [Y ], or expectation of
product minus product of expectations is frequently useful.
Defining covariance and correlation

I Now define covariance of X and Y by

Cov(X , Y ) = E [(X E [X ])(Y E [Y ]).
I Note: by definition Var(X ) = Cov(X , X ).
I Covariance (like variance) can also written a different way.
Write x = E [X ] and Y = E [Y ]. If laws of X and Y are
known, then X and Y are just constants.
I Then

Cov(X , Y ) = E [(X X )(Y Y )] = E [XY X Y Y X +X Y ] =

E [XY ] X E [Y ] Y E [X ] + X Y = E [XY ] E [X ]E [Y ].
I Covariance formula E [XY ] E [X ]E [Y ], or expectation of
product minus product of expectations is frequently useful.
I Note: if X and Y are independent then Cov(X , Y ) = 0.
Basic covariance facts
I Using Cov(X , Y ) = E [XY ] E [X ]E [Y ] as a definition,
certain facts are immediate.
Basic covariance facts
I Using Cov(X , Y ) = E [XY ] E [X ]E [Y ] as a definition,
certain facts are immediate.
I Cov(X , Y ) = Cov(Y , X )
Basic covariance facts
I Using Cov(X , Y ) = E [XY ] E [X ]E [Y ] as a definition,
certain facts are immediate.
I Cov(X , Y ) = Cov(Y , X )
I Cov(X , X ) = Var(X )
Basic covariance facts
I Using Cov(X , Y ) = E [XY ] E [X ]E [Y ] as a definition,
certain facts are immediate.
I Cov(X , Y ) = Cov(Y , X )
I Cov(X , X ) = Var(X )
I Cov(aX , Y ) = aCov(X , Y ).
Basic covariance facts
I Using Cov(X , Y ) = E [XY ] E [X ]E [Y ] as a definition,
certain facts are immediate.
I Cov(X , Y ) = Cov(Y , X )
I Cov(X , X ) = Var(X )
I Cov(aX , Y ) = aCov(X , Y ).
I Cov(X1 + X2 , Y ) = Cov(X1 , Y ) + Cov(X2 , Y ).
Basic covariance facts
I Using Cov(X , Y ) = E [XY ] E [X ]E [Y ] as a definition,
certain facts are immediate.
I Cov(X , Y ) = Cov(Y , X )
I Cov(X , X ) = Var(X )
I Cov(aX , Y ) = aCov(X , Y ).
I Cov(X1 + X2 , Y ) = Cov(X1 , Y ) + Cov(X2 , Y ).
I General statement of bilinearity of covariance:
Xm n
X m X
X n
Cov( ai Xi , bj Yj ) = ai bj Cov(Xi , Yj ).
i=1 j=1 i=1 j=1
Basic covariance facts
I Using Cov(X , Y ) = E [XY ] E [X ]E [Y ] as a definition,
certain facts are immediate.
I Cov(X , Y ) = Cov(Y , X )
I Cov(X , X ) = Var(X )
I Cov(aX , Y ) = aCov(X , Y ).
I Cov(X1 + X2 , Y ) = Cov(X1 , Y ) + Cov(X2 , Y ).
I General statement of bilinearity of covariance:
Xm n
X m X
X n
Cov( ai Xi , bj Yj ) = ai bj Cov(Xi , Yj ).
i=1 j=1 i=1 j=1

I Special case:
Xn n
X X
Var( Xi ) = Var(Xi ) + 2 Cov(Xi , Xj ).
i=1 i=1 (i,j):i<j
Defining correlation

I Again, by definition Cov(X , Y ) = E [XY ] E [X ]E [Y ].

Defining correlation

I Again, by definition Cov(X , Y ) = E [XY ] E [X ]E [Y ].

I Correlation of X and Y defined by

Cov(X , Y )
(X , Y ) := p .
Var(X )Var(Y )
Defining correlation

I Again, by definition Cov(X , Y ) = E [XY ] E [X ]E [Y ].

I Correlation of X and Y defined by

Cov(X , Y )
(X , Y ) := p .
Var(X )Var(Y )
I Correlation doesnt care what units you use for X and Y . If
a > 0 and c > 0 then (aX + b, cY + d) = (X , Y ).
Defining correlation

I Again, by definition Cov(X , Y ) = E [XY ] E [X ]E [Y ].

I Correlation of X and Y defined by

Cov(X , Y )
(X , Y ) := p .
Var(X )Var(Y )
I Correlation doesnt care what units you use for X and Y . If
a > 0 and c > 0 then (aX + b, cY + d) = (X , Y ).
I Satisfies 1 (X , Y ) 1.
Defining correlation

I Again, by definition Cov(X , Y ) = E [XY ] E [X ]E [Y ].

I Correlation of X and Y defined by

Cov(X , Y )
(X , Y ) := p .
Var(X )Var(Y )
I Correlation doesnt care what units you use for X and Y . If
a > 0 and c > 0 then (aX + b, cY + d) = (X , Y ).
I Satisfies 1 (X , Y ) 1.
I Why is that? Something to do with E [(X + Y )2 ] 0 and
E [(X Y )2 ] 0?
Defining correlation

I Again, by definition Cov(X , Y ) = E [XY ] E [X ]E [Y ].

I Correlation of X and Y defined by

Cov(X , Y )
(X , Y ) := p .
Var(X )Var(Y )
I Correlation doesnt care what units you use for X and Y . If
a > 0 and c > 0 then (aX + b, cY + d) = (X , Y ).
I Satisfies 1 (X , Y ) 1.
I Why is that? Something to do with E [(X + Y )2 ] 0 and
E [(X Y )2 ] 0?
I If a and b are constants and a > 0 then (aX + b, X ) = 1.
Defining correlation

I Again, by definition Cov(X , Y ) = E [XY ] E [X ]E [Y ].

I Correlation of X and Y defined by

Cov(X , Y )
(X , Y ) := p .
Var(X )Var(Y )
I Correlation doesnt care what units you use for X and Y . If
a > 0 and c > 0 then (aX + b, cY + d) = (X , Y ).
I Satisfies 1 (X , Y ) 1.
I Why is that? Something to do with E [(X + Y )2 ] 0 and
E [(X Y )2 ] 0?
I If a and b are constants and a > 0 then (aX + b, X ) = 1.
I If a and b are constants and a < 0 then (aX + b, X ) = 1.
Important point

I Say X and Y are uncorrelated when (X , Y ) = 0.

Important point

I Say X and Y are uncorrelated when (X , Y ) = 0.

I Are independent random variables X and Y always
uncorrelated?
Important point

I Say X and Y are uncorrelated when (X , Y ) = 0.

I Are independent random variables X and Y always
uncorrelated?
I Yes, assuming variances are finite (so that correlation is
defined).
Important point

I Say X and Y are uncorrelated when (X , Y ) = 0.

I Are independent random variables X and Y always
uncorrelated?
I Yes, assuming variances are finite (so that correlation is
defined).
I Are uncorrelated random variables always independent?
Important point

I Say X and Y are uncorrelated when (X , Y ) = 0.

I Are independent random variables X and Y always
uncorrelated?
I Yes, assuming variances are finite (so that correlation is
defined).
I Are uncorrelated random variables always independent?
I No. Uncorrelated just means E [(X E [X ])(Y E [Y ])] = 0,
i.e., the outcomes where (X E [X ])(Y E [Y ]) is positive
(the upper right and lower left quadrants, if axes are drawn
centered at (E [X ], E [Y ])) balance out the outcomes where
this quantity is negative (upper left and lower right
quadrants). This is a much weaker statement than
independence.
Examples

I Suppose that X1 , . . . , Xn are i.i.d. random variables with

variance 1. For example, maybe each Xj takes values 1
according to a fair coin toss.
Examples

I Suppose that X1 , . . . , Xn are i.i.d. random variables with

variance 1. For example, maybe each Xj takes values 1
according to a fair coin toss.
I Compute Cov(X1 + X2 + X3 , X2 + X3 + X4 ).
Examples

I Suppose that X1 , . . . , Xn are i.i.d. random variables with

variance 1. For example, maybe each Xj takes values 1
according to a fair coin toss.
I Compute Cov(X1 + X2 + X3 , X2 + X3 + X4 ).
I Compute the correlation coefficient
(X1 + X2 + X3 , X2 + X3 + X4 ).
Examples

I Suppose that X1 , . . . , Xn are i.i.d. random variables with

variance 1. For example, maybe each Xj takes values 1
according to a fair coin toss.
I Compute Cov(X1 + X2 + X3 , X2 + X3 + X4 ).
I Compute the correlation coefficient
(X1 + X2 + X3 , X2 + X3 + X4 ).
I Can we generalize this example?
Examples

I Suppose that X1 , . . . , Xn are i.i.d. random variables with

variance 1. For example, maybe each Xj takes values 1
according to a fair coin toss.
I Compute Cov(X1 + X2 + X3 , X2 + X3 + X4 ).
I Compute the correlation coefficient
(X1 + X2 + X3 , X2 + X3 + X4 ).
I Can we generalize this example?
I What is variance of number of people who get their own hat
in the hat problem?
Examples

I Suppose that X1 , . . . , Xn are i.i.d. random variables with

variance 1. For example, maybe each Xj takes values 1
according to a fair coin toss.
I Compute Cov(X1 + X2 + X3 , X2 + X3 + X4 ).
I Compute the correlation coefficient
(X1 + X2 + X3 , X2 + X3 + X4 ).
I Can we generalize this example?
I What is variance of number of people who get their own hat
in the hat problem?
I Define Xi to be 1 if ith person gets own hat, zero otherwise.
Examples

I Suppose that X1 , . . . , Xn are i.i.d. random variables with

variance 1. For example, maybe each Xj takes values 1
according to a fair coin toss.
I Compute Cov(X1 + X2 + X3 , X2 + X3 + X4 ).
I Compute the correlation coefficient
(X1 + X2 + X3 , X2 + X3 + X4 ).
I Can we generalize this example?
I What is variance of number of people who get their own hat
in the hat problem?
I Define Xi to be 1 if ith person gets own hat, zero otherwise.
I Recall
Pformula P
Var( ni=1 Xi ) = ni=1 Var(Xi ) + 2 (i,j):i<j Cov(Xi , Xj ).
P
Examples

I Suppose that X1 , . . . , Xn are i.i.d. random variables with

variance 1. For example, maybe each Xj takes values 1
according to a fair coin toss.
I Compute Cov(X1 + X2 + X3 , X2 + X3 + X4 ).
I Compute the correlation coefficient
(X1 + X2 + X3 , X2 + X3 + X4 ).
I Can we generalize this example?
I What is variance of number of people who get their own hat
in the hat problem?
I Define Xi to be 1 if ith person gets own hat, zero otherwise.
I Recall
Pformula P
Var( ni=1 Xi ) = ni=1 Var(Xi ) + 2 (i,j):i<j Cov(Xi , Xj ).
P

I Reduces problem to computing Cov(Xi , Xj ) (for i 6= j) and

Var(Xi ).
Outline

Covariance and correlation

Paradoxes: getting ready to think about conditional expectation

Outline

Covariance and correlation

Paradoxes: getting ready to think about conditional expectation

Famous paradox

I Certain corrupt and amoral banker dies, instructed to spend

some number n (of bankers choosing) days in hell.
Famous paradox

I Certain corrupt and amoral banker dies, instructed to spend

some number n (of bankers choosing) days in hell.
I At the end of this period, a (biased) coin will be tossed.
Banker will be assigned to hell forever with probability 1/n
and heaven forever with probability 1 1/n.
Famous paradox

I Certain corrupt and amoral banker dies, instructed to spend

some number n (of bankers choosing) days in hell.
I At the end of this period, a (biased) coin will be tossed.
Banker will be assigned to hell forever with probability 1/n
and heaven forever with probability 1 1/n.
I After 10 days, banker reasons, If I wait another day I reduce
my odds of being here forever from 1/10 to 1/11. Thats a
reduction of 1/110. A 1/110 chance at infinity has infinite
value. Worth waiting one more day.
Famous paradox

I Certain corrupt and amoral banker dies, instructed to spend

some number n (of bankers choosing) days in hell.
I At the end of this period, a (biased) coin will be tossed.
Banker will be assigned to hell forever with probability 1/n
and heaven forever with probability 1 1/n.
I After 10 days, banker reasons, If I wait another day I reduce
my odds of being here forever from 1/10 to 1/11. Thats a
reduction of 1/110. A 1/110 chance at infinity has infinite
value. Worth waiting one more day.
I Repeats this reasoning every day, stays in hell forever.
Famous paradox

I Certain corrupt and amoral banker dies, instructed to spend

some number n (of bankers choosing) days in hell.
I At the end of this period, a (biased) coin will be tossed.
Banker will be assigned to hell forever with probability 1/n
and heaven forever with probability 1 1/n.
I After 10 days, banker reasons, If I wait another day I reduce
my odds of being here forever from 1/10 to 1/11. Thats a
reduction of 1/110. A 1/110 chance at infinity has infinite
value. Worth waiting one more day.
I Repeats this reasoning every day, stays in hell forever.
I Standard punch line: this is actually what banker deserved.
Famous paradox

I Certain corrupt and amoral banker dies, instructed to spend

some number n (of bankers choosing) days in hell.
I At the end of this period, a (biased) coin will be tossed.
Banker will be assigned to hell forever with probability 1/n
and heaven forever with probability 1 1/n.
I After 10 days, banker reasons, If I wait another day I reduce
my odds of being here forever from 1/10 to 1/11. Thats a
reduction of 1/110. A 1/110 chance at infinity has infinite
value. Worth waiting one more day.
I Repeats this reasoning every day, stays in hell forever.
I Standard punch line: this is actually what banker deserved.
I Fairly dark as math humor goes (and no offense intended to
anyone...) but dilemma is interesting.
I Paradox: decisions seem sound individually but together yield
worst possible outcome. Why? Can we demystify this?
I Paradox: decisions seem sound individually but together yield
worst possible outcome. Why? Can we demystify this?
I Variant without probability: Instead of tossing (1/n)-coin,
person deterministically spends 1/n fraction of future days
(every nth day, say) in hell.
I Paradox: decisions seem sound individually but together yield
worst possible outcome. Why? Can we demystify this?
I Variant without probability: Instead of tossing (1/n)-coin,
person deterministically spends 1/n fraction of future days
(every nth day, say) in hell.
I Paradox: decisions seem sound individually but together yield
worst possible outcome. Why? Can we demystify this?
I Variant without probability: Instead of tossing (1/n)-coin,
person deterministically spends 1/n fraction of future days
(every nth day, say) in hell.
I Paradox: decisions seem sound individually but together yield
worst possible outcome. Why? Can we demystify this?
I Variant without probability: Instead of tossing (1/n)-coin,
person deterministically spends 1/n fraction of future days
(every nth day, say) in hell.
I Even simpler variant: infinitely many identical money sacks
have labels 1, 2, 3, . . . I have sack 1. You have all others.
I Paradox: decisions seem sound individually but together yield
worst possible outcome. Why? Can we demystify this?
I Variant without probability: Instead of tossing (1/n)-coin,
person deterministically spends 1/n fraction of future days
(every nth day, say) in hell.
I Even simpler variant: infinitely many identical money sacks
have labels 1, 2, 3, . . . I have sack 1. You have all others.
I You offer me a deal. I give you sack 1, you give me sacks 2
and 3. I give you sack 2 and you give me sacks 4 and 5. On
the nth stage, I give you sack n and you give me sacks 2n and
2n + 1. Continue until I say stop.
I Paradox: decisions seem sound individually but together yield
worst possible outcome. Why? Can we demystify this?
I Variant without probability: Instead of tossing (1/n)-coin,
person deterministically spends 1/n fraction of future days
(every nth day, say) in hell.
I Even simpler variant: infinitely many identical money sacks
have labels 1, 2, 3, . . . I have sack 1. You have all others.
I You offer me a deal. I give you sack 1, you give me sacks 2
and 3. I give you sack 2 and you give me sacks 4 and 5. On
the nth stage, I give you sack n and you give me sacks 2n and
2n + 1. Continue until I say stop.
I Lets me get arbitrarily rich. But if I go on forever, I return
every sack given to me. If nth sack confers right to spend nth
day in heaven, leads to hell-forever paradox.
I Paradox: decisions seem sound individually but together yield
worst possible outcome. Why? Can we demystify this?
I Variant without probability: Instead of tossing (1/n)-coin,
person deterministically spends 1/n fraction of future days
(every nth day, say) in hell.
I Even simpler variant: infinitely many identical money sacks
have labels 1, 2, 3, . . . I have sack 1. You have all others.
I You offer me a deal. I give you sack 1, you give me sacks 2
and 3. I give you sack 2 and you give me sacks 4 and 5. On
the nth stage, I give you sack n and you give me sacks 2n and
2n + 1. Continue until I say stop.
I Lets me get arbitrarily rich. But if I go on forever, I return
every sack given to me. If nth sack confers right to spend nth
day in heaven, leads to hell-forever paradox.
I I make infinitely many good trades and end up with less than I
started with. Paradox is really just existence of 2-to-1 map
from (smaller set) {2, 3, . . .} to (bigger set) {1, 2, . . .}.
Money pile paradox

I You have an infinite collection of money piles with labeled

0, 1, 2, . . . from left to right.
Money pile paradox

I You have an infinite collection of money piles with labeled

0, 1, 2, . . . from left to right.
I Precise details not important, but lets say you have 1/4 in
the 0th pile and 38 5j in the jth pile for each j > 0. Important
thing is that pile size is increasing exponentially in j.
Money pile paradox

I You have an infinite collection of money piles with labeled

0, 1, 2, . . . from left to right.
I Precise details not important, but lets say you have 1/4 in
the 0th pile and 38 5j in the jth pile for each j > 0. Important
thing is that pile size is increasing exponentially in j.
I Banker proposes to transfer a fraction (say 2/3) of each pile
to the pile on its left and remainder to the pile on its right.
Do this simultaneously for all piles.
Money pile paradox

I You have an infinite collection of money piles with labeled

0, 1, 2, . . . from left to right.
I Precise details not important, but lets say you have 1/4 in
the 0th pile and 38 5j in the jth pile for each j > 0. Important
thing is that pile size is increasing exponentially in j.
I Banker proposes to transfer a fraction (say 2/3) of each pile
to the pile on its left and remainder to the pile on its right.
Do this simultaneously for all piles.
I Every pile is bigger after transfer (and this can be true even if
banker takes a portion of each pile as a fee).
Money pile paradox

I You have an infinite collection of money piles with labeled

0, 1, 2, . . . from left to right.
I Precise details not important, but lets say you have 1/4 in
the 0th pile and 38 5j in the jth pile for each j > 0. Important
thing is that pile size is increasing exponentially in j.
I Banker proposes to transfer a fraction (say 2/3) of each pile
to the pile on its left and remainder to the pile on its right.
Do this simultaneously for all piles.
I Every pile is bigger after transfer (and this can be true even if
banker takes a portion of each pile as a fee).
I Banker seemed to make you richer (every pile got bigger) but
really just reshuffled your infinite wealth.
Two envelope paradox
I X is geometric with parameter 1/2. One envelope has 10X
dollars, one has 10X 1 dollars. Envelopes shuffled.
Two envelope paradox
I X is geometric with parameter 1/2. One envelope has 10X
dollars, one has 10X 1 dollars. Envelopes shuffled.
I You choose an envelope and, after seeing contents, are
allowed to choose whether to keep it or switch. (Maybe you
have to pay a dollar to switch.)
Two envelope paradox
I X is geometric with parameter 1/2. One envelope has 10X
dollars, one has 10X 1 dollars. Envelopes shuffled.
I You choose an envelope and, after seeing contents, are
allowed to choose whether to keep it or switch. (Maybe you
have to pay a dollar to switch.)
I Maximizing conditional expectation, it seems its always
better to switch. But if you always switch, why not just
choose second-choice envelope first and avoid switching fee?
Two envelope paradox
I X is geometric with parameter 1/2. One envelope has 10X
dollars, one has 10X 1 dollars. Envelopes shuffled.
I You choose an envelope and, after seeing contents, are
allowed to choose whether to keep it or switch. (Maybe you
have to pay a dollar to switch.)
I Maximizing conditional expectation, it seems its always
better to switch. But if you always switch, why not just
choose second-choice envelope first and avoid switching fee?
I Kind of a disguised version of money pile paradox. But more
subtle. One has to replace jth pile of money with
restriction of expectation sum to scenario that first chosen
envelop has 10j . Switching indeed makes each pile bigger.
Two envelope paradox
I X is geometric with parameter 1/2. One envelope has 10X
dollars, one has 10X 1 dollars. Envelopes shuffled.
I You choose an envelope and, after seeing contents, are
allowed to choose whether to keep it or switch. (Maybe you
have to pay a dollar to switch.)
I Maximizing conditional expectation, it seems its always
better to switch. But if you always switch, why not just
choose second-choice envelope first and avoid switching fee?
I Kind of a disguised version of money pile paradox. But more
subtle. One has to replace jth pile of money with
restriction of expectation sum to scenario that first chosen
envelop has 10j . Switching indeed makes each pile bigger.
I However, Higher expectation given amount in first envelope
may not be right notion of better. If S is payout with
switching, T is payout without switching, then S has same
law as T 1. In that sense S is worse.
Moral

I Beware infinite expectations.

Moral

I Beware infinite expectations.

I Beware unbounded utility functions.
Moral

I Beware infinite expectations.

I Beware unbounded utility functions.
I They can lead to strange conclusions, sometimes related to
reshuffling infinite (actual or expected) wealth to create
more paradoxes.
Moral

I Beware infinite expectations.

I Beware unbounded utility functions.
I They can lead to strange conclusions, sometimes related to
reshuffling infinite (actual or expected) wealth to create
more paradoxes.
I Paradoxes can arise even when total transaction is finite with
probability one (as in envelope problem).