NOVEMBER 19, 2014

LECTURE 13
COVARIANCE, CORRELATION, AND REGRESSION

1 Expectations with bivariate distributions

In the previous lecture, we extended the concept of distributions from the one-variable scenario
to multivariate (bivariate) distributions. One can similarly extend the concept of the expected
value or expectation from univariate distributions to multivariate.
Let X be a random variable with a PMF pX and support SX , or a PDF fX . Recall that
the expected value of u(X), where u(x) is some function is defined as
X
EX = u(x)pX (x), or
x∈SX
ˆ ∞
EX = u(x)fX (x)dx
−∞

in the discrete and continuous cases respectively. In the bivariate case, the expectation is
defined similarly, except the function u is a function with two arguments, and one must
average across two variables using the joint PMF or PDF.

Definition 1. (Expected value) Let X and Y be two random variables with a joint PMF
pX,Y (x, y) and supports SX and SY , or a joint PDF fX,Y (x, y). The expected value of U (X, Y )
is defined as
X X
EU (X, Y ) = U (x, y)pX,Y (x, y)
x∈SX y∈SY

in the discrete case, or

ˆ ∞ ˆ ∞
EU (X, Y ) = u(x, y)fX,Y (x, y)dydx
−∞ −∞

in the continuous case.

2 Covariance
One of the important examples is the covariance, which is a measure of association between
two random variables.

Definition 2. (Covariance) Let X and Y be two random variables. The covariance between
X and Y is defined as
Cov(X, Y ) = E ((X − EX)(Y − EY )) .

1
Remark. 1. If X and Y are discrete with a joint PMF pX,Y and supports SX , SY , then:
X X
Cov(X, Y ) = (x − EX)(y − EY )pX,Y (x, y).
x∈SX y∈SY

Recall that EX and EY are two numbers. They must be computed before computing the
covariance. In the continuous case, let fX,Y denote the joint PDF of X and Y :
ˆ ˆ
Cov(X, Y ) = (x − EX)(y − EY )fX,Y (x, y)dydx.

2. Covariance is a number that measures the co-variation or co-movement of two random

variables. It is positive if the deviations of X and Y from their respective means tend to
have the same sign: positive with positive and negative with negative. In this case, the two
variables are positively related. The covariance is negative if the deviations of X and Y from
their respective means tend to have opposite signs. In that case, the variables are negatively
related.

Example. Consider the following example from Lecture 12.

Table 1: Example of a joint PMF

Price of a share (S)
$100 $250 $400
Earnings per share (E) 2 1
$10 6 6 0
2 1
$20 0 6 6

The marginal PMF of the earnings per share (E) is pE (10) = pE (20) = 0.5, and therefore
the mean of the earnings per share is 10 × 0.5 + 20 × 0.5 = 15. The marginal PMF of the price
(S) is pS (100) = 1/3, pS (250) = 1/2, pS (400) = 1/6. Hence, the expected value of the price is

1 1 1
100 × + 250 × + 400 × = 225.
3 2 6

Next, the covariance between the two variables is

2 1
(10 − 15) × (100 − 225) × + (10 − 15) × (250 − 225) ×
6 6
2 1
+ (20 − 15) × (250 − 225) × + (20 − 15) × (400 − 225) × = 375.
6 6

The covariance is positive and, therefore, the earnings per share and the price per share are
positively associated.

Some properties of the covariance are given in the following theorem. Note that the theorem

2
applies to both discrete and continuous cases, and the proof relies only on the linearity of the
expectation.

Theorem 3. (a) Cov(X, Y ) = E(XY ) − (EX)(EY ).

(b) Cov(X, X) = V ar(X).
(c) Cov(X, Y ) = Cov(Y, X).
(d) Let a be a constant. Cov(X, a) = 0.
(e) Let a and b be two constants. Cov(aX, bY ) = abCov(X, Y ).
(f ) Cov(X + Y, Z) = Cov(X, Z) + Cov(Y, Z).
(g) V ar(X + Y ) = V ar(X) + V ar(Y ) + 2Cov(X, Y ).
(h) V ar(X − Y ) = V ar(X) + V ar(Y ) − 2Cov(X, Y ).

Proof. (a) First, consider

(X − EX)(Y − EY ) = XY − X(EY ) − (EX)(Y − EY ).

Taking the expectations on both sides, we obtain

Cov(X, Y ) = E(XY ) − E (X(EY )) − E ((EX)(Y − EY ))

= E(XY ) − (EY )(EX) − (EX)E(Y − EY )
= E(XY ) − (EY )(EX),

where the equality in the last line holds since E(Y − EY ) = 0.

(b) Omitted.
(c) Omitted.
(d) Omitted.
(e) From the definition of the covariance,

Cov(aX, bY ) = E (aX − E(aX)) (bY − E(bY ))

= E(aX − aEX)(bY − bEY )
= abE(X − EX)(Y − EY )
= abCov(X, Y ).

3
 (f) From the definition of the covariance,

Cov(X + Y, Z) = E [(X + Y − E(X + Y )) (Z − EZ)]

= E [(X − EX + Y − EY )) (Z − EZ)]
= E [(X − EX)(Z − EZ) + (Y − EY )(Z − EZ)]
= E(X − EX)(Z − EZ) + E(Y − EY )(Z − EZ)
= Cov(X, Z) + Cov(Y, Z),

where the equality in the last line holds by the definition of the covariance.
(g) From the definition of the variance,

V ar(X + Y ) = E [X + Y − E(X + Y )]2

= E [X − EX + Y − EY ]2
= E (X − EX)2 + (Y − EY )2 + 2(X − EX)(Y − EY )
 

= E(X − EX)2 + E(Y − EY )2 + 2E(X − EX)(Y − EY )

= V ar(X) + V ar(Y ) + 2Cov(X, Y )

where the equality in the last line holds by the definition of the covariance.

3 Correlation coefficient
Covariance can be difficult to interpret since it can take any value from −∞ to +∞. For
that reason and to obtain a unit-free measure of association1 , it is common to standardize the
covariance by the product of standard deviations of the two variables. The resulting measure
is called the coefficient of correlation.

Definition 4. (Correlation coefficient) The correlation coefficient between two random

variables X and Y (denoted as ρX,Y ) is defined as

Cov(X, Y )
ρX,Y = p .
V ar(X)V ar(Y )

The correlation coefficient is a unit-free measure. Hence, it is unaffected by the change of

1
Cov(X, Y ) is measured in units of X × units of Y .

4
units of measurement. Consider the correlation between X and cY , where c > 0.

Cov(X, cY )
ρX,cY = p
V ar(X)V ar(cY )
cCov(X, Y )
= p
V ar(X)c2 V ar(Y )
Cov(X, Y )
= p
V ar(X)V ar(Y )
= ρX,Y .

More importantly, the correlation coefficient is restricted to the interval [−1, 1] thus providing
a very intuitive measure of the strength of association. A larger positive value of the corre-
lation coefficient indicates a stronger positive association between two variables. Similarly, a
smaller negative (closer to -1) value of the correlation coefficient indicates a stronger negative
association between two variables.

Theorem 5. The correlation coefficient between X and Y satistfies:

(a) −1 ≤ ρX,Y ≤ 1.
(b) ρX,Y = 1 if and only if Y = a + bX for some constants a ∈ R and b > 0.
(c) ρX,Y = −1 if and only if Y = a + bX for some constants a ∈ R and b < 0.

Remark. When ρX,Y = 1 or −1, there is a perfect linear relationship between X and Y .
Note that if there is a perfect nonlinear relationship between X and Y (such as Y = X 2 ),
the correlation coefficient will be strictly between −1 and 1, i.e −1 < ρX,Y < 1. Thus,
the correlation coefficient measures the degree of linearity in the relationship between two
variables.
The proof of Theorem 5 requires another result.

Lemma 6. (Cauchy-Schwartz inequality) Let X and Y be two random variables. Then,

p
(a) |Cov(X, Y )| ≤ V ar(X)V ar(Y ).
p
(b) |Cov(X, Y )| = V ar(X)V ar(Y ) if and only if Y = a + bX for some constants a and
b.

Proof. Define a new random variable

Cov(X, Y )
U = Y − X
V ar(X)
= Y − βX, (1)

where
Cov(X, Y )
β= .
V ar(X)

5
By the results of Theorem 3(e) and (h),

V ar(U ) = V ar(Y − βX)

= V ar(Y ) + β 2 V ar(X) − 2βCov(X, Y ).

Using the definition of β, we have:

(Cov(X, Y ))2 Cov(X, Y )

V ar(U ) = V ar(Y ) + 2
V ar(X) − 2 Cov(X, Y )
(V ar(X)) V ar(X)
(Cov(X, Y ))2
= V ar(Y ) − . (2)
V ar(X)

Since V ar(U ) ≥ 0, it follows that

(Cov(X, Y ))2
V ar(Y ) − ≥ 0,
V ar(X)

or
(Cov(X, Y ))2 ≤ V ar(X)V ar(Y ). (3)

Taking the square root of the both sides of the inequality in (3), we obtain:
p
|Cov(X, Y )| ≤ V ar(X)V ar(Y ),

which completes the proof of part (a).

To show part (b), note that it follows from (2) that

(Cov(X, Y ))2 = V ar(X)V ar(Y )

if and only if V ar(U ) = 0. However, that implies that U is not a random variable, i.e. U = a
for some constant a. From the definition of U we obtain

a = Y − βX.

Hence, the second part of the lemma holds with b = β = Cov(X, Y )/V ar(X).

The result of Theorem 5 follows immediately from the Cauchy-Schwartz inequality. The
latter implies that
p p
− V ar(X)V ar(Y ) ≤ Cov(X, Y ) ≤ V ar(X)V ar(Y ).

6
 p
Dividing all sides of the inequality by V ar(X)V ar(Y ), we obtain

−1 ≤ ρX,Y ≤ 1,

where the inequalities can hold as equalities if and only if Y = a + βX.

Example. Consider again the example in Table 1. The variance of earnings is

1 1
(10 − 15)2 × + (20 − 15)2 × = 25.
2 2

The variance of the price is

1 1 1
(100 − 225)2 × + (250 − 225)2 × + (400 − 225)2 × = 10625.
3 2 6

We previously have computed that the covariance between the earnings and the price is 375.
Hence, the correlation is equal to

375
√ ≈ 0.73.
25 × 10625

Hence, the two variables are positively associated.

Definition 7. (Uncorrelatedness) We say that X and Y are uncorrelated when ρX,Y = 0

(Cov(X, Y ) = 0).

The meaning and implications of uncorrelatedness are discussed in the following section.

4 Regression and least squares.

We saw in Theorem 5 that the correlation coefficient equals 1 or −1 if and only if all possible
realizations of the pair of random variables X and Y form a straight line with the slope
β = Cov(X, Y )/V ar(X), i.e. with probability one, Y = a + βX. In that case, while still
random, X and Y have a perfect linear relationship.
When the correlation coefficient is strictly between −1 and 1, it is impossible to draw
a straight line connecting all possible realization of X and Y . As a matter of fact, there
might not be a deterministic relationship between X and Y : there is no function h(·) such
that Y = h(X) with probability 1. Nevertheless, let’s try to find a linear function that
approximates the relationship between X and Y as good as possible (in the sense that will
become clear immediately).
Let a + bX be the approximating function, and therefore the approximation error is

ε = Y − a − bX.

7
Unless the correlation coefficient is exactly 1 or −1, it is impossible to find constants a and
b so that ε is zero with probability one. Nevertheless, let’s try to find a and b so that the
variance of the approximation error is as small as possible. Note that small variability of the
approximation error corresponds to a more accurate approximation. Hence, we are interested
in solving the following problem:

min E(Y − a − bX)2 .

a,b

This optimization problem is known as a least squares problem.

Definition 8. (Least squares and regression) The values of a and b that solve mina,b E(Y −
a − bX)2 are called the least squares coefficients. Let α and β denote the least squares coef-
ficients (the values of a and b respectively that solve the least squares problem). A function
y = α + βx is called the regression (or the regression line).
Remark. The regression line is the best linear approximation to the stochastic relationship
between X and Y . It is best in the sense that the approximation error has the smallest
variance.
Theorem 9. The least squares coefficients are given by

Cov(X, Y )
β = ,
V ar(X)
α = (EY ) − β(EX). (4)

Proof. We can find the least squares coefficients by solving the first-order conditions for the
least squares problem. Suppose that we can change the order of differentiation and integration.
 
∂ ∂
E(Y − a − bX)2 = E (Y − a − bX) 2
∂b ∂b
= −2 [E(Y − a − bX)X] .
 
∂ ∂
E(Y − a − bX)2 = E (Y − a − bX)2
∂a ∂a
= −2E(Y − a − bX).

Hence, α and β must satisfy:

E [(Y − α − βX)X] = 0, (5)

E(Y − α − βX) = 0. (6)

From the second equation, it follows that

EY − α − βEX = 0,

8
which proves (4).
Next, substitute the expression for α into 5. We have

0 = E [(Y − α − βX) X]
= E [(Y − (EY − βEX) − βX) X]
= E [((Y − EY ) − β(X − EX)) X]
= E [(Y − EY )X − β(X − EX)X]
= E [(Y − EY )X] − βE [(X − EX)X] .

Hence,
E [(Y − EY )X]
β= .
E [(X − EX)X]
However,

E [(X − EX)X] = EX 2 − (EX)EX

= V ar(X).

Also,

E [(Y − EY )X] = EY X − (EY )EX

= Cov(X, Y ).

Remark. 1. The regression line goes through the point given by the expected values of X and
Y , since by (4),
EY = α + βEX.

2. The least squares error is

ε = Y − α − βX.

Hence, we can write

Y = α + βX + ε.

The first-order condition (6) implies that

Eε = 0.

Hence, the random approximation error has zero mean. The first-order condition (5) implies

9
that the approximation error is uncorrelated with X:

0 = EεX
= Cov(ε, X),

where the second line follows from Cov(ε, X) = EεX − (Eε)(EX) = EεX, since the mean of
ε is zero.
3. When X and Y are uncorrelated, ρX,Y = 0 and Cov(X, Y ) = 0, and therefore,

Cov(X, Y )
β= = 0.
V ar(X)

In that case, the best linear approximation to the relationship between X and Y is a flat
line (with zero slope). Note that uncorrelatedness does not necessarily mean that X and Y
are unrelated. It is possible that the relationship between X and Y can be described by a
non-trivial nonlinear function (and therefore the two random variables are related). However,
the best linear approximation to this relationship is a flat line.
4. The relationship between positively correlated variables is best approximated by a
regression line with a positive slope. The relationship between negatively correlated variables
is best approximated by a regression line with a negative slope.

5 Expectations and independence

Suppose that X and Y are continuously distributed, and let fX,Y be the joint PDF of X and
Y . Recall that ˆ ˆ
EXY = xyfX,Y (x, y)dydx.

Suppose that X and Y are statistically independent: for any x and y,

fX,Y (x, y) = fX (x)fY (y).

10
In that case,
ˆ ˆ
EXY = xyfX (x)fY (y)dydx
ˆ ˆ 
= xfX (x) yfY (y)dy dx
ˆ
= xfX (x)E(Y )dx
ˆ
= (EY ) xfX (x)dx
= (EY )(EX).

Thus, under independence the expected value of a product of random variables is equal to the
product of expectations. In particular, this implies that under independence,

Cov(X, Y ) = EXY − (EX)(EY ) = 0.

Theorem 10. If X and Y are independent, then X and Y are uncorrelated.

Remark. In general, uncorrelatedness does not imply independence.

The result can be extended to functions of random variables. Recall that by Theorem
9 in Lecture 12, functions of random variables are independent if the random variables are
independent. Therefore, when X and Y are independent,

E[U (X)V (Y )] = E[U (X)]E[V (Y )].

The same results also hold for discrete random variables. The proof in the discrete case is
identical after replacing integrals with sums and PDFs with PMFs.

11