Probability and Stochastic Processes

Chapter 4 Pairs of Random Variables

0-0
 A Pair of Random Variables
• An experiment produces both X and Y
• Joint range:

SX,Y = {(x, y)|PX,Y (x, y) > 0} ⊂ SX × SY

1
 Joint Cumulative Distribution Function (CDF)
• The joint CDF of random variables X and Y is

FX,Y (x, y) = P [X ≤ x, Y ≤ y]

2
Joint CDF Area
Y

{X<x, Y<y} (x,y)

3
 Joint CDF Properties
For any pair of random variable X and Y ,
• 0 ≤ FX,Y (x, y) ≤ 1

• FX (x) = FX,Y (x, ∞)

• FY (y) = FX,Y (∞, y)

• FX,Y (−∞, y) = FX,Y (x, −∞) = 0

• FX,Y (∞, ∞) = 1
• If x ≤ x1 and y ≤ y1 , then

FX,Y (x, y) ≤ FX,Y (x1 , y1 )

4
 Proof Example

FX (x) = P [X ≤ x]
= P [X ≤ x, Y < ∞]
= lim FX,Y (x, y) = FX,Y (x, ∞)
y→∞

5
 Joint Probability Mass Function (PMF)
• Definition 4.2: The joint probability mass function of X and Y is

PX,Y (x, y) = P [X = x, Y = y]

6
 Example
• Flip a coin three times.

• X = no. of H

• Y = no. of T after the first H

• Write all 8 outcomes: {HHH, . . . , T T T }.

• For each outcome, record X and Y

7
 Probability of Event {(X, Y ) ∈ B}
• Theorem 4.2: For discrete random variables X and Y and any set B in
the X, Y plane, the probability of the event {(X, Y ) ∈ B} is

P [B] = PX,Y (x, y)
(x,y)∈B

8
 Marginal PMF
• Theorem 4.3: For discrete random variables X and Y with joint PMF
PX,Y (x, y),

 
PX (x) = PX,Y (x, y) , PY (y) = PX,Y (x, y) .
y∈SY x∈SX

• Calculate by summing across rows, write PMF in margin

9
 Joint Probability Density Function (PDF)
• Definition 4.3: The joint PDF of the continuous random variables X and
Y is a function fX,Y (x, y) satisfying
 x  y
FX,Y (x, y) = fX,Y (u, v) dv du
−∞ −∞

∂ 2 FX,Y (x,y)
• Theorem 4.4: fX,Y (x, y) = ∂x ∂y

10
 Joint PDF Properties
• fX,Y (x, y) ≥ 0 for all (x, y)
∞ ∞
• −∞ −∞ fX,Y (x, y) dx dy = 1

• The probability that the continuous random variables (X, Y ) are in A

(region of X, Y plane) is

P [A] = fX,Y (x, y) dx dy
A

• Example 4.5 (MATLAB Demo)

11
 Marginal PDF
• If X and Y are random variables with joint PDF fX,Y (x, y),
 ∞
fX (x) = fX,Y (x, y) dy
−∞
 ∞
fY (y) = fX,Y (x, y) dx
−∞

• Proof: Write FX (x) as an integral, take the derivative with respect to x.

• Example 4.7 (MATLAB Demo)

12
 Functions of Two Random Variables
• Theorem 4.9: For discrete random variables X and Y , the derived
random variable W = g(X, Y ) has PMF

PW (w) = P [g(X, Y ) = w] = PX,Y (x, y)
(x,y):g(x,y)=w

• Theorem 4.10: For continuous random variables X and Y , the CDF of

W = g(X, Y ) is

FW (w) = P [W ≤ w] = P [g(X, Y ) ≤ w] = fX,Y (x, y) dx dy
g(x,y)≤w
.

13
 W = g(X, Y ): Examples
• Examples:

W1 = X + Y
W2 = max(X, Y )
W3 = XY
W4 = X/Y

• Example 4.10 (MATLAB Demo)

• Quiz 4.6(B) (MATLAB Demo)

14
 Expected Values
• Theorem 4.12: For random variables X and Y , the expected value of
W = g(X, Y ) is
– Discrete:
 
E[W ] = E[g(X, Y )] = g(x, y)PX,Y (x, y)
x∈SX y∈SY

– Continuous:
 ∞  ∞
E[W ] = E[g(X, Y )] = g(x, y)fX,Y (x, y) dx dy
−∞ −∞

15
 Expectations of Functions
• Theorem 4.13:

E[g1 (X, Y ) + g2 (X, Y ) + · · · + gn (X, Y )] = E[g1 (X, Y )]+· · ·+E[gn (X, Y )]

16
 Expectations of Sums
For any two random variables X and Y ,
• E[X + Y ] = E[X] + E[Y ]

• Var [X + Y ] = Var [X] + Var [Y ] + 2E[(X − μX )(Y − μY )]

17
 Covariance
• Definition 4.4: The covariance of two random variables X and Y is

Cov [X, Y ] = E[(X − μX ) (Y − μY )]

• Covariance is also Cov [X, Y ] = E[XY ] − μX μY

• Cov [X, Y ] > 0 says, X > E[X] implies Y > E[Y ] is likely (X goes
up, Y goes up)

• Var [X + Y ] = Var [X] + Var [Y ] + 2 Cov [X, Y ]

18
 Correlation
• Definition 4.5: The correlation of X and Y is rX,Y = E[XY ]
• Cov [X, Y ] = rX,Y − μX μY

• Correlation = Covariance if E[X] = 0 or E[Y ] = 0

• If X = Y, Cov [X, Y ] = Var [X] = Var [Y ] and

 2  2
rX,Y = E X = E Y

• E[XY ] > 0 suggests that X > 0 increases chance Y > 0

19
 Orthogonality and Uncorrelateness
• Definition 4.6: X and Y are orthogonal if E[XY ] = 0
• Definition 4.7: X and Y are uncorrelated if Cov [X, Y ] = 0

20
 Correlation Coefficient
• Definition 4.8: The correlation coefficient of two random variables X
and Y is
Cov [X, Y ]
ρX,Y = 
Var [X] Var [Y ]
• Theorem 4.17: −1 ≤ ρX,Y ≤ 1
• Theorem 4.18: If Y = aX + b, then
⎧
⎪
⎨ −1
⎪ a<0
ρX,Y = 0 a=0
⎪
⎪
⎩
1 a>0

• Quiz 4.7(B) (MATLAB Demo)

21
 Conditioning by an Event
• Definition 4.9: For discrete random variables X and Y and an event B
with P [B] > 0, the conditional joint PMF of X and Y given B is
1
PX,Y |B (x, y) = P [X = x, Y = y|B] = P [(X = x, Y = y) ∩ B]
P [B]

• Theorem 4.19: For any event B, a region of the X, Y plane with

P [B] > 0,
⎧
⎨ PX,Y (x,y) (x, y) ∈ B
P [B]
PX,Y |B (x, y) =
⎩ 0 otherwise

22
 Conditional Joint PDF
• Definition 4.10: Given an event B with P [B] > 0, the conditional joint
PDF of X and Y is
⎧
⎨ fX,Y (x,y) (x, y) ∈ B
P [B]
fX,Y |B (x, y) =
⎩ 0 otherwise

23
 Conditional Expected Value
• Theorem 4.20: For random variables X and Y and an event B with
P [B] > 0, the conditional expected value of W = g(X, Y ) given B is
– Discrete:
 
E[W |B] = E[g(X, Y )|B] = g(x, y)PX,Y |B (x, y)
x∈SX y∈SY

– Continuous:
 ∞  ∞
E[W |B] = E[g(X, Y )|B] = g(x, y)fX,Y |B (x, y) dx dy
−∞ −∞

24
 Conditional Variance
• Definition 4.11: The conditional variance of the random variable
W = g(X, Y ) is

Var [W |B] = E (W − E[W |B])2 |B

 
• Theorem 4.21: Var [W |B] = E W |B − (E[W |B])2
2

25
 Conditioning by a Random Variable
• Definition 4.12: For any event Y = y such that PY (y) > 0, the
conditional PMF of X given Y = y is
P [X = x, Y = y]
PX|Y (x|y) = P [X = x|Y = y] =
P [Y = y]

• Theorem 4.22: For random variables X and Y with joint PMF

PX,Y (x, y), and x and y such that PX (x) > 0 and PY (y) > 0,

PX,Y (x, y) = PX|Y (x|y) PY (y) = PY |X (y|x) PX (x) .

26
 Conditional Expected Value
• Theorem 4.23: X and Y are discrete random variables. For any y ∈ SY ,
the conditional expected value of g(X, Y ) given Y = y is

E[g(X, Y )|Y = y] = g(x, y)PX|Y (x|y)
x∈SX

• Special cases
– E[X|Y = y]
– Cond. Variance:
2
Var [X|Y = y] = E (X − E[X|Y = y]) |Y = y

27
 Conditional PDF
• Definition 4.13: For y such that fY (y) > 0, the conditional PDF of X
given Y = y is
fX,Y (x, y)
fX|Y (x|y) = .
fY (y)

• Theorem 4.24: fX,Y (x, y) = fY |X (y|x) fX (x) = fX|Y (x|y) fY (y)

• fX|Y (x|y):

fX|Y (x|y) dx = P [x < X ≤ x + dx|y < Y ≤ y + dy]

• Example 4.22 (MATLAB Demo)

28
 Conditional Expected Value
• Definition 4.14: For continuous random variables X and Y and any y
such that fY (y) > 0, the conditional expected value of g(X, Y ) given
Y = y is
 ∞
E[g(X, Y )|Y = y] = g(x, y)fX|Y (x|y) dx.
−∞

• The conditional expected value of X given Y = y is a special case of

Definition 4.14:
 ∞
E[X|Y = y] = xfX|Y (x|y) dx
−∞

29
 Conditional Expected Value
• Definition 4.15: The conditional expected value E[X|Y ] is a function of
random variable Y such that if Y = y then E[X|Y ] = E[X|Y = y].

• Let h(y) = E[X|Y = y]. When Y = y, E[X|Y ] takes on the value

h(y). That is, E[X|Y ] = h(Y ).

• Theorem 4.25: EY [EX|Y [X|Y ]] = EX [X].

• Theorem 4.26:
 +∞
EY [EX|Y [g(X, Y )|Y ]] = EX|Y [g(X, Y )|Y = y]fY (y)dy
−∞
= EX,Y [g(X, Y )]

30
 Independent Random Variables
• Definition 4.16: X and Y are independent if and only if for all values of
x and y
– Discrete: PX,Y (x, y) = PX (x) PY (y)
– Continuous: fX,Y (x, y) = fX (x) fY (y)

• Theorem 4.22 implies that if X and Y are independent discrete random

variables, then

PX|Y (x|y) = PX (x) , PY |X (y|x) = PY (y) .

• Theorem 4.24 implies that if X and Y are independent continuous

random variables, then

fX|Y (x|y) = fX (x) , fY |X (y|x) = fY (y) .

31
 Independence: Example 4.23
• Are X and Y independent?
⎧
⎨ 4xy 0 ≤ x ≤ 1, 0 ≤ y ≤ 1
fX,Y (x, y) =
⎩ 0 otherwise

..................................................................

• The marginal PDFs of X and Y are

⎧
⎨ 2x 0≤x≤1
fX (x) =
⎩ 0 otherwise
⎧
⎨ 2y 0≤y≤1
fY (y) =
⎩ 0 otherwise

• Is fX,Y (x, y) = fX (x) fY (y)? Yes.

32
 Independence: Example 4.24
• Are U and V independent?
⎧
⎨ 24uv u ≥ 0, v ≥ 0, u + v ≤ 1
fU,V (u, v) =
⎩ 0 otherwise

33
 Example 4.24 (continued)
• Region of nonzero density is triangular and
⎧
⎨ 12u(1 − u)2 0≤u≤1
fU (u) =
⎩ 0 otherwise
⎧
⎨ 12v(1 − v)2 0≤v≤1
fV (v) =
⎩ 0 otherwise

• U and V are not independent!

34
 Properties of Independent RVs
• Theorem 4.27: For independent random variables X and Y ,
1. E[g(X)h(Y )] = E[g(X)]E[h(Y )]
2. rX,Y = E[XY ] = E[X]E[Y ]
3. Cov [X, Y ] = 0, ρX,Y = 0
4. Var [X + Y ] = Var [X] + Var [Y ]
5. E[X|Y = y] = E[X] for all y ∈ SY
6. E[Y |X = x] = E[Y ] for all x ∈ SX

35
 Bivariate Gaussian Random Variables
• Definition 4.17: X and Y have a bivariate Gaussian PDF with
parameters μ1 , σ1 , μ2 , σ2 , and ρ if
⎡  2  2 ⎤
x−μ1 2ρ(x−μ1 )(y−μ2 ) y−μ2
⎢ σ1 − σ1 σ2 + σ2 ⎥
exp ⎣− ⎦
2(1 − ρ2 )
fX,Y (x, y) = 
2πσ1 σ2 1 − ρ2
where μ1 and μ2 can be any real numbers, σ1 > 0, σ2 > 0, and
−1 ≤ ρ ≤ 1.

36
 ρ=0
fX,Y (x, y)
0.35
0.3
0.25
0.2 -3

0.15 -2
0.1 -1
0.05
0
0
-3 1
y
-2
-1
0 2
1
2
x 33

37
 ρ = 0.9
fX,Y (x, y)

0.35
0.3
0.25
0.2 -3

0.15 -2

0.1 -1
0.05
0
0 y
-3 1
-2
-1
0 2
x 1
2
33

• Three-dimensional plot of a bivariate Gaussian PDF (MATLAB Demo)

38
 ρ = −0.9
fX,Y (x, y)

0.35
0.3
0.25
0.2 -3

0.15 -2

0.1 -1
0.05
0
0 y
-3 1
-2
-1
0 2
x 1
2
33

39
 Bivariate Gaussian PDF
• Theorem 4.28: If X and Y are the bivariate Gaussian random variables
in Definition 4.17, X is the Gaussian (μ1 , σ1 ) random variable and Y is
the Gaussian (μ2 , σ2 ) random variable:

1 2 2
fX (x) =  2
exp [−(x − μ 1 ) /2σ1 ]
2πσ1
and
1 2 2
fY (y) =  2
exp [−(y − μ 2 ) /2σ2 ].
2πσ2

40
 Rewriting the Bivariate Gaussian PDF
Complete the square of the exponent to write

fX,Y (x, y) = fX (x) fY |X (y|x)

where
1 −(x−μ1 )2 /2σ12
fX (x) = √ e
σ1 2π
1 −(y−μ̃2 (x))2 /2σ̃22
fY |X (y|x) = √ e
σ̃2 2π
σ2
μ̃2 (x) = μ2 + ρ (x − μ1 ) = E[Y |X = x]
σ1
 
σ̃2 = σ2 1 − ρ = Var [Y |X = x]
2

41
 Conditional PDF of the Bivariate Gaussian RV
• Theorem 4.29: If X and Y are the bivariate Gaussian random variables
in Definition 4.17, the conditional PDF of Y given X is
1 2 2
fY |X (y|x) = √ e−(y−μ̃2 (x)) /2σ̃2
σ̃2 2π
where
σ2
μ̃2 (x) = μ2 + ρ (x − μ1 ) = E[Y |X = x]
σ1
 
2
σ̃2 = σ2 1 − ρ = Var [Y |X = x]

42
Gaussian PDF

43
 Bivariate Gaussian Properties
• E[X] = μ1 , Var [X] = σ12
• Given X = x, Y is Gaussian

• Conditional mean of Y given X = x:

σ2
μ̃2 (x) = E[Y |X = x] = μ2 + ρ (x − μ1 )
σ1
 
σ̃2 = Var [Y |X = x] = σ2 1 − ρ2

44
 Bivariate Gaussian Properties
• Theorem 4.31: Bivariate Gaussian random variables X and Y in
Definition 4.17 have correlation coefficient ρX,Y = ρ.

• Theorem 4.32: Bivariate Gaussian random variables X and Y are

uncorrelated if and only if they are independent.

• Bivariate Normal Experiment (Virtual Laboratories)

45
 Virtual Laboratories
• Bivariate Normal Experiment
• Bivariate Uniform Experiment

46
 Chapter Summary
• Joint CDF FX,Y (x, y), joint PMF PX,Y (x, y), and joint PDF
fX,Y (x, y)
• Marginal PMF and marginal PDF

• Expected value E[g(X, Y )] and covariance Cov [X, Y ]

• Conditional joint PMF PX,Y |A (x, y) and conditional PMF PX|Y (x|y)
as well as conditional joint PDF fX,Y |A (x, y) and conditional PDF
fX|Y (x|y)

• Independent random variables

• Bivariate Gaussian random variables

47