Linear MMSE Estimation of Random Variables https://www.probabilitycourse.com/chapter9/9_1_6_linear_MMSE_est...

9.1.6 Linear MMSE Estimation of Random Variables

Suppose that we would like to estimate the value of an unobserved random variable X , given that we have
observed Y = y. In general, our estimate x ̂ is a function of y

x ̂ = g(y).
For example, the MMSE estimate of X given Y = y is
g(y) = E[X|Y = y].
We might face some difficulties if we want to use the MMSE in practice. First, the function
g(y) = E[X|Y = y] might have a complicated form. Specifically, if X and Y are random vectors,
computing E[X|Y = y] might not be easy. Moreover, to find E[X|Y = y] we need to know fX|Y (y),
which might not be easy to find in some problems. To address these issues, we might want to use a simpler
function g(y) to estimate X . In particular, we might want g(y) to be a linear function of y .

Suppose that we would like to have an estimator for X of the form

X̂ L = g(Y) = aY + b,
where a and b are some real numbers to be determined. More specifically, our goal is to choose a and b such
that the MSE of the above estimator

MSE = E[(X − X̂ L )2 ]

is minimized. We call the resulting estimator the linear MMSE estimator. The following theorem gives us the
optimal values for a and b.

Theorem 9.1
Let X and Y be two random variables with finite means and variances. Also, let ρ be the correlation
coefficient of X and Y . Consider the function

h(a, b) = E[(X − aY − b)2 ].

Then,

1. The function h(a, b) is minimized if

Cov(X, Y)
a = a∗ = , b = b∗ = EX − aEY.
Var(Y)
2. We have h(a∗ , b∗ ) = (1 − ρ2 )Var(X).
3. E[(X − a∗ Y − b∗ )Y] = 0 (orthogonality principle).

1 of 6 10/12/22, 3:54 pm
Linear MMSE Estimation of Random Variables https://www.probabilitycourse.com/chapter9/9_1_6_linear_MMSE_est...

Proof: We have

h(a, b) = E[(X − aY − b)2 ]

= E[X 2 + a2 Y 2 + b2 − 2aXY − 2bX + 2abY]
= EX 2 + a2 EY 2 + b2 − 2aEXY − 2bEX + 2abEY.
Thus, h(a, b) is a quadratic function of a and b. We take the derivatives with respect to a and b and set
them to zero, so we obtain

EY 2 ⋅ a + EY ⋅ b = EXY (9.4)
EY ⋅ a + b = EX (9.5)
Solving for a and b, we obtain

Cov(X, Y)
a∗ = , b∗ = EX − aEY.
Var(Y)
It can be verified that the above values do in fact minimize h(a, b) . Note that Equation 9.5 implies that
E[X − a∗ Y − b∗ ] = 0 . Therefore,

h(a∗ , b∗ ) = E[(X − a∗ Y − b∗ )2 ]
= Var(X − a∗ Y − b∗ )
= Var(X − a∗ Y)
= Var(X) + a∗2 Var(Y) − 2a∗ Cov(X, Y)
Cov(X, Y)2 Cov(X, Y)
= Var(X) + Var(Y) − 2 Cov(X, Y)
Var(Y)2 Var(Y)
Cov(X, Y)2
= Var(X) −
Var(Y)
2
= (1 − ρ )Var(X).
Finally, note that

E[(X − a∗ Y − b∗ )Y] = EXY − a∗ EY 2 − b∗ EY

= 0 (by Equation 9.4).

Note that X̃  = X − a∗ Y − b∗ is the error in the linear MMSE estimation of X given Y . From the above
theorem, we conclude that

E[X̃ ] = 0,
E[X̃ Y] = 0.

2 of 6 10/12/22, 3:54 pm
Linear MMSE Estimation of Random Variables https://www.probabilitycourse.com/chapter9/9_1_6_linear_MMSE_est...

In sum, we can write the linear MMSE estimator of X given Y as

Cov(X, Y)
X̂ L = (Y − EY) + EX.
Var(Y)
If ρ = ρ(X, Y) is the correlation coefficient of X and Y , then Cov(X, Y) = ρσX σY , so the above
formula can be written as
ρσX
X̂ L = (Y − EY) + EX.
σY
Linear MMSE Estimator

The linear MMSE estimator of the random variable X , given that we have observed Y , is given by

Cov(X, Y)
X̂ L = (Y − EY) + EX
Var(Y)
ρσX
= (Y − EY) + EX.
σY
The estimation error, defined as X̃  = X − X̂ L , satisfies the orthogonality principle:

E[X̃ ] = 0,
Cov(X̃ , Y) = E[X̃ Y] = 0.
The MSE of the linear MMSE is given by

E[(X − XL )2 ] = E[X̃  ] = (1 − ρ2 )Var(X).

2

Note that to compute the linear MMSE estimates, we only need to know expected values, variances, and the
covariance. Let us look at an example.

Example 9.8
1
Suppose X ∼ Unif orm(1, 2) , and given X = x , Y is exponential with parameter λ = x.

a. Find the linear MMSE estimate of X given Y .

b. Find the MSE of this estimator.
c. Check that E[X̃ Y] = 0.
• Solution
◦ We have

Cov(X, Y)
X̂ L = (Y − EY) + EX.
Var(Y)
Therefore, we need to find EX , EY , Var(Y) , and Cov(X, Y). First, note that we have

3 of 6 10/12/22, 3:54 pm
Linear MMSE Estimation of Random Variables https://www.probabilitycourse.com/chapter9/9_1_6_linear_MMSE_est...

3
EX = 2
, and

EY = E[E[Y|X]] (law of iterated expectations)

1
= E [X] (since Y|X ∼ Exponential( X ))
3
= .
2
EY 2 = E[E[Y 2 |X]] (law of iterated expectations)
1
= E [2X 2 ] ( since Y|X ∼ Exponential(
X )
)
2

∫1
= 2x 2 dx
14
= .
3
Therefore,

Var(Y) = EY 2 − (EY )2
14 9
= −
3 4
29
= .
12
We also have

EXY = E[E[XY|X]] (law of iterated expectations)

EXY = E[XE[Y|X]] (given X, X is a constant)
1
= E [X ⋅ X] (since Y|X ∼ Exponential( X ))
2

∫1
= x 2 dx
7
= .
3
Thus,

4 of 6 10/12/22, 3:54 pm
Linear MMSE Estimation of Random Variables https://www.probabilitycourse.com/chapter9/9_1_6_linear_MMSE_est...

Cov(X, Y) = E[XY] − (EX)(EY)

7 3 3
= − ⋅
3 2 2
1
= .
12
a. The linear MMSE estimate of X given Y is

Cov(X, Y)
X̂ L = (Y − EY) + EX
Var(Y)

29 ( 2) 2
1 3 3
= Y − +
Y 42
= + .
29 29
b. The MSE of X̂ L is

MSE = (1 − ρ2 )Var(X).
1
Since X ∼ Unif orm(1, 2) , Var(X) = 12
. Also,

2 Cov2 (X, Y)
ρ =
Var(X)Var(Y)
1
= .
29
Thus,

MSE = (1 −
29 ) 12
1 1 7
= .
87
c. We have

X̃  = X − X̂ L
Y 42
=X− − .
29 29
Therefore,

5 of 6 10/12/22, 3:54 pm
Linear MMSE Estimation of Random Variables https://www.probabilitycourse.com/chapter9/9_1_6_linear_MMSE_est...

E[X̃ Y] = E [(X −
29 ) ]
Y 42
− Y
29
EY 2 42
= E[XY] − − EY
29 29
7 14 42 3
= − − ⋅
3 3 ⋅ 29 29 2
= 0.

← previous
next →

The print version of the book is available through Amazon here.

6 of 6 10/12/22, 3:54 pm