Signal Analysis for Communications

Statistical Signal Processing: Random variables

Dept. Signals, Systems and Radiocommunications (ETSIT-UPM)

Course 2022/2023
Contents

1 Random variable

2 Random vector
Two-dimensional random vector
N-dimensional random vector

3 Exercises

Dept. SSR (ETSIT-UPM) Random variables Course 2022/2023 2 / 47

Random variable. Basic concepts

Let us consider a random experiment with a probabilistic space ( , F, P) .

A random variable (RV) is a function X : ‘æ R that assigns a real
number to every outcome of the experiment
For a RV X we define the cumulative distribution function (CDF) as

FX (x ) = P (X Æ x ) , ’x œ R

The shape of the CDF determines the type of RV:

Continuous RV: the CDF is a continuous function
Discrete RV: the CDF is a staircase function
The set X of all outcomes that the RV X can take is known as
range
The range of a continuous RV is uncountably infinite
The range of a discrete RV is finite or countably infinite

Dept. SSR (ETSIT-UPM) Random variables Course 2022/2023 3 / 47

Random variable. Characterization.
A continuous RV is usually characterized by its probability density
function (pdf)
dFX (x )
fX (x ) = , ’x œ R
dx
that can be used to calculate probabilities:
x2
P (x1 Æ X Æ x2 ) = fX (x ) dx
x1

A discrete RV is usually characterized by its probability mass function

(pmf)
pX (x ) = P (X = x ) , ’x œ X
that can be also used to calculate probabilities:
ÿ
P (X œ A) = pX (x )
x œA

Dept. SSR (ETSIT-UPM) Random variables Course 2022/2023 4 / 47

Random variable. Moments
Mean or Expected value: µX Ë= E (X ) È
! "
Variance: ‡X2 = Var(X ) = E (X ≠ µX )2 = E X 2 ≠ µ2X
Continuous RV
Œ
µX = E (X ) = xfX (x ) dx
≠Œ
Œ Œ
2
‡X2 = Var (X ) = (x ≠ µX ) fX (x ) dx = x 2 fX (x ) dx ≠µ2X
≠Œ ≠Œ
¸ ˚˙ ˝ ¸ ˚˙ ˝
E [(X ≠µX )2 ] E (X 2 )

Discrete RV
ÿ
µX = E (X ) = xpX (x )
xœ X
ÿ ÿ
‡X2 = Var(X ) = (x ≠ µX )2 pX (x ) = x 2 pX (x ) ≠µ2X
xœ X xœ X
¸ ˚˙ ˝ ¸ ˚˙ ˝
E [(X ≠µX )2 ] E (X 2 )

Dept. SSR (ETSIT-UPM) Random variables Course 2022/2023 5 / 47

Random variable. Transformations
Given a RV X and a function g : R ‘æ R, Y = g(X ) is another RV
Moments of the transformation
I Œ
g (x ) fX (x ) dx continuous RV
E [g (X )] = q≠Œ
xœ X
g(x )pX (x ) discrete RV
Ë È Ë È
Var [g (X )] = E (g (X ) ≠ E [g (X )])2 = E g 2 (X ) ≠ E 2 [g (X )]

Case study: linear transformation

Y = g(X ) = aX + b

Moments
µY = E (Y ) = E (aX + b) = aE (X ) + b = aµX + b
Ë È
‡Y2 = Var (Y ) = Var (aX + b) = E (Y ≠ µY )2
Ë È Ë È
2
= E (aX + ◆
b ≠ aµX ≠ ◆
b) = a2 E (X ≠ µX )2 = a2 ‡X2

Dept. SSR (ETSIT-UPM) Random variables Course 2022/2023 6 / 47

Gaussian random variable
! "
A Gaussian RV X v N µX , ‡X2 is described by its pdf:
(x ≠µX )2
1 ≠
2‡ 2
fX (x ) =  e X ≠Œ<x <Œ
2fi‡X2
Probability density function (pdf) Cumulative distribution Function (CDF)
0.4 1

0.9
0.35
0.8
0.3
0.7
0.25
0.6

0.2 0.5

0.4
0.15
0.3
0.1
0.2
0.05
0.1

0 0
-3 -2 -1 0 1 2 3 -3 -2 -1 0 1 2 3

where µX = E (X ) and ‡X2 = Var(X ) are the parameters of the distribution

! "
Theorem: If X v N
!
µX , ‡"X2 and Y = aX + b, we have:
Y ≥ N µY , ‡Y2 , with µY = aµX + b, ‡Y2 = a2 ‡X2
Dept. SSR (ETSIT-UPM) Random variables Course 2022/2023 7 / 47
Random vector

Two-dimensional random vector

N-dimensional random vector

Dept. SSR (ETSIT-UPM) Random variables Course 2022/2023 8 / 47

Bidimensional random vector. Characterization I

A bidimensional random vector is a pair of RVs: X = [X1 , X2 ]T that can

only take values in a certain set X (range)
X2

ΩX
X1

Joint cumulative distribution function (joint CDF)

FX (x) © FX1 ,X2 (x1 , x2 ) = P (X1 Æ x1 , X2 Æ x2 ) , ’x = [x1 , x2 ]T œ R2

Dept. SSR (ETSIT-UPM) Random variables Course 2022/2023 9 / 47

Bidimensional random vector. Characterization II

Continuous RVs: Joint probability density function (joint pdf)

ˆ 2 FX1 ,X2 (x1 , x2 )

fX (x) © fX1 ,X2 (x1 , x2 ) = , ’x = [x1 , x2 ]T œ R2
ˆx1 ˆx2

P (X œ A) = fX1 ,X2 (x1 , x2 ) dx1 dx2 © fX (x) dx

A A

Discrete RVs: Joint probability mass function (joint pmf)

pX (x) © pX1 ,X2 (x1 , x2 ) = P (X1 = x1 , X2 = x2 ) , ’x = [x1 , x2 ]T œ X

ÿÿ ÿ
P (X œ A) = pX1 ,X2 (x1 , x2 ) © pX (x)
(x1 ,x2 )œA xœA

Dept. SSR (ETSIT-UPM) Random variables Course 2022/2023 10 / 47

Bidimensional random vector. Characterization III

The marginal distributions of X1 and X2 can be obtained from the

joint distributions:

FX1 (x1 ) = FX1 ,X2 (x1 , Œ) FX2 (x2 ) = FX1 ,X2 (Œ, x2 )
Œ Œ
fX1 (x1 ) = fX1 ,X2 (x1 , x2 ) dx2 fX2 (x2 ) = fX1 ,X2 (x1 , x2 ) dx1
≠Œ ≠Œ
ÿ ÿ
pX1 (x1 ) = pX1 ,X2 (x1 , x2 ) pX2 (x2 ) = pX1 ,X2 (x1 , x2 )
x2 œ X2 x1 œ X1

Dept. SSR (ETSIT-UPM) Random variables Course 2022/2023 11 / 47

Bidimensional random vector. Characterization IV

Also, we define the conditional distributions:

Continuous RVs:
fX1 ,X2 (x1 , x2 )
fX1 (x1 | x2 ) © fX1 (x1 | X2 = x2 ) =
fX2 (x2 )
fX1 ,X2 (x1 , x2 )
fX2 (x2 | x1 ) © fX2 (x2 | X1 = x1 ) =
fX1 (x1 )

Discrete RVs:
pX1 ,X2 (x1 , x2 )
pX1 (x1 | x2 ) © pX1 (x1 | X2 = x2 ) =
pX2 (x2 )
pX1 ,X2 (x1 , x2 )
pX2 (x2 | x1 ) © pX2 (x2 | X1 = x1 ) =
pX1 (x1 )

Dept. SSR (ETSIT-UPM) Random variables Course 2022/2023 12 / 47

Bidimensional random vector. Moments I

Correlation:
I Œ Œ
x1 x2 fX1 ,X2 (x1 , x2 )dx1 dx2 continuous
RX1 X2 = E (X1 X2 ) = q≠Œ
q ≠Œ
(x1 ,x2 )œ X
x1 x2 pX1 ,X2 (x1 , x2 ) discrete

|RX1 X2 | represents a measure of proportional relation between X1 and

X2 : maximum value is reached if X2 = aX1 .
Covariance:

CX1 X2 = Cov (X1 , X2 ) = E [(X1 ≠ µX1 ) (X2 ≠ µX2 )]

= E (X1 X2 ) ≠ µX1 E (X2 ) ≠ µX2 E (X1 ) + µX1 µX2
= E (X1 X2 ) ≠ E (X1 ) E (X2 ) = RX1 X2 ≠ µX1 µX2

|CX1 X2 | represents a measure of linear relation between X1 and X2 :

maximum value is reached if X2 = aX1 + b.

Dept. SSR (ETSIT-UPM) Random variables Course 2022/2023 13 / 47

Bidimensional random vector. Moments II
Correlation coefficient:
Cov (X1 , X2 )
flX1 X2 = 
Var (X1 ) Var (X2 )

that satisfies the condition: |flX1 X2 | Æ 1.

|flX1 X2 | is a measure of the degree of linear dependence between X1
and X2 : |flX1 X2 | = 1 iff X2 = aX1 + b.
Mean vector (using vector notation X = [X1 , X2 ]T )
C D C D
E (X1 ) µX1
µX = = = E (X)
E (X2 ) µX2

Correlation matrix
C ! " D AC D B
E X12 E (X X )
! 1 2 "2
X1 1 2
RX = =E [X1 X2 ] = E XXT
E (X2 X1 ) E X2 X2
Dept. SSR (ETSIT-UPM) Random variables Course 2022/2023 14 / 47
Bidimensional random vector. Moments III

Covariance matrix
5 6 5 6
Var (X1 ) Cov (X1 , X2 ) ‡X2 1 flX1 X2 ‡X1 ‡X2
CX = =
Cov (X2 , X1 ) Var (X2 ) flX 1 X 2 ‡ X 1 ‡ X 2 ‡X2 2
# $
= E (X ≠ µX )(X ≠ µX ) = RX ≠ µX µT
T
X

Properties of the correlation and covariance matrices:

Symmetric:

RX = RT
X
CX = CT
X

Positive semi-definite: ’a œ R2

aT RX a Ø 0
aT CX a Ø 0
Dept. SSR (ETSIT-UPM) Random variables Course 2022/2023 15 / 47
Bidimensional random vector. Moments IV

Finally, it is also possible to define the conditional moments

Y
] Πx f (x | x )dx if X1 and X2 continuous
1 X1 1 2 1
E (X1 | X2 = x2 ) = q≠Œ
[ xœ x1 pX1 (x1 | x2 )
1 X1
if X1 and X2 discrete
Var (X1 | X2 = x2 ) = E (X12 | X2 = x2 ) ≠ E 2 (X1 | X2 = x2 )

Dept. SSR (ETSIT-UPM) Random variables Course 2022/2023 16 / 47

Vectors and matrices of cross-correlations and cross-covariances I

Given the random vector X = [X1 , X2 ]T and the RV Y , we define

Cross-correlation vector
C D
E (X1 Y )
rXY = E (XY ) =
E (X2 Y )

Cross-covariance vector
C D
Cov (X1 , Y )
cXY = E [(X ≠ µX )(Y ≠ µY )] =
Cov (X2 , Y )

Dept. SSR (ETSIT-UPM) Random variables Course 2022/2023 17 / 47

Vectors and matrices of cross-correlations and cross-covariances II

Given the random vectors X = [X1 , X2 ]T and Y = [Y1 , Y2 ]T , we define

Cross-correlation matrices
C D
1
T
2 E (X1 Y1 ) E (X1 Y2 )
RXY = E XY =
E (X2 Y1 ) E (X2 Y2 )
C D
1
T
2 E (Y1 X1 ) E (Y1 X2 )
RYX = E YX = = RT
XY
E (Y2 X1 ) E (Y2 X2 )

Cross-covariance matrices
5 6
# $ Cov (X1 , Y1 ) Cov (X1 , Y2 )
CXY = E (X ≠ µX )(Y ≠ µY )T =
Cov (X2 , Y1 ) Cov (X2 , Y2 )
5 6
# $ Cov (Y1 , X1 ) Cov (Y1 , X2 )
CYX = E (Y ≠ µY )(X ≠ µX )T = = CT
Cov (Y2 , X1 ) Cov (Y2 , X2 ) XY

CXY = RXY ≠ µX µT
Y

Dept. SSR (ETSIT-UPM) Random variables Course 2022/2023 18 / 47

Independence, uncorrelatedness and orthogonality I

X1 and X2 are independent iff:

fX1 ,X2 (x1 , x2 ) = fX1 (x1 )fX2 (x2 ) ’x1 , x2 œ R (continuous RVs)
pX1 ,X2 (x1 , x2 ) = pX1 (x1 )pX2 (x2 ) ’(x1 , x2 ) œ X (discrete RVs)

or:

Continuous: Discrete:
fX1 (x1 | x2 ) = fX1 (x1 ) ’x1 , x2 pX1 (x1 | x2 ) = pX1 (x1 ) ’x1 , x2
fX2 (x2 | x1 ) = fX2 (x2 ) ’x1 , x2 pX2 (x2 | x1 ) = pX2 (x2 ) ’x1 , x2

X1 and X2 are uncorrelated iff: E (X1 X2 ) = E (X1 ) E (X2 )

or: CX1 X2 = 0
X1 and X2 are orthogonal iff: E (X1 X2 ) = 0 or: RX1 X2 = 0

Dept. SSR (ETSIT-UPM) Random variables Course 2022/2023 19 / 47

Independence, uncorrelatedness and orthogonality II

Theorem 1: If X1 and X2 are independent, then they are uncorrelated.

Theorem 2: If X1 and X2 are uncorrelated, then the covariance matrix is
diagonal: C D
Var (X1 ) 0
CX =
0 Var (X2 )

Dept. SSR (ETSIT-UPM) Random variables Course 2022/2023 20 / 47

Bidimensional random vector. One transformation

Given the random vector X = [X1 , X2 ]T and the function g : R2 ‘æ R,

Y = g(X1 , X2 ) is a RV.
Mean of the transformation
I Œ Œ
g(x1 , x2 )fX1 ,X2 dx1 dx2 continuous
E [g (X1 , X2 )] = q≠Œ
q ≠Œ
(x1 ,x2 )œ X
g(x1 , x2 )pX1 ,X2 (x1 , x2 ) discrete

Case study: one linear transformation

C D
X1
Y = a1 X 1 + a2 X 2 + b = [a1 a2 ] + b = aT X + b
X2
C D
a1
= [X1 X2 ] + b = XT a + b
a2

where a = [a1 , a2 ]T

Dept. SSR (ETSIT-UPM) Random variables Course 2022/2023 21 / 47

Bidimensional random vector. One linear transformation
Mean
µY = E (Y ) = E (a1 X1 + a2 X2 + b) = a1 E (X1 ) + a2 E (X2 ) + b
1 2
µY = E (Y ) = E aT X + b = aT E (X) + b = aT µX + b = µT
Xa + b

Variance
‡Y2 = Var (Y ) = a12 Var(X1 ) + a22 Var(X2 ) + 2a1 a2 Cov (X1 , X2 )
1 2
‡Y2 = Var (Y ) = Var aT X + b = aT CX a

If X1 , X2 are uncorrelated, then

Var (Y ) = Var (a1 X1 + a2 X2 + b) = a12 Var (X1 ) + a22 Var (X2 )
Cross-covariance vector
C D C D
Cov (X1 , Y ) a1 Var(X1 ) + a2 Cov (X1 , X2 )
cXY = = = CX a
Cov (X2 , Y ) a1 Cov(X1 , X2 ) + a2 Var (X2 )

Dept. SSR (ETSIT-UPM) Random variables Course 2022/2023 22 / 47

Bidimensional random vector. One linear transformation

Proof: Variance of one linear transformation

Ë È 51 22 6
2
‡Y2 = Var (Y ) = E (Y ≠ µY ) =E T
a X +◆ T
b ≠ a µX ≠ ◆
b
Ë1 21 2È
=E aT X ≠ aT µX XT a ≠ µT
Xa
Ë È
= aT E (X ≠ µX ) (X ≠ µX )T a = aT CX a

Proof: Cross-covariance vector of one linear transformation

Ë È
cXY = E [(X ≠ µX )(Y ≠ µY )] = E (X ≠ µX )(XT a + ◆
b ≠ µT b)
Xa ≠◆
Ë È
= E (X ≠ µX )(X ≠ µX )T a = CX a

Dept. SSR (ETSIT-UPM) Random variables Course 2022/2023 23 / 47

Bidimensional random vector. Two transformations

Given the random vector X = [X1 , X2 ]T and g1 : R2 ‘æ R, g2 : R2 ‘æ R,

Y = [Y1 , Y2 ]T = [g1 (X1 , X2 ), g2 (X1 , X2 )]T is a bidimensional RV
Case study: two linear transformations

Y1 = g1 (X1 , X2 ) = a11 X1 + a12 X2 + b1

Y2 = g2 (X1 , X2 ) = a21 X1 + a22 X2 + b2

using vector notation:

Y = AX + b

with C D C D
a11 a12 b1
A= b=
a21 a22 b2

Dept. SSR (ETSIT-UPM) Random variables Course 2022/2023 24 / 47

Bidimensional random vector. Two linear transformations

Moments of Y = AX + b

µY = E (Y) = E (AX + b) = AE (X) + b = AµX + b

Ë È
CY = E (Y ≠ µY )(Y ≠ µY )T
Ë È
= E (AX + ◆ b)(AX + ◆
b ≠ AµX ≠ ◆ b)T
b ≠ AµX ≠ ◆
Ë È
= AE (X ≠ µX )(X ≠ µX )T AT = ACX AT
Ë È Ë È
CXY = E (X ≠ µX )(Y ≠ µY )T = E (X ≠ µX )(AX + ◆ b)T
b ≠ AµX ≠ ◆
Ë È
= E (X ≠ µX )(X ≠ µX )T AT = CX AT
Ë È Ë È
CYX = E (Y ≠ µY )(X ≠ µX )T = E (AX + ◆ b)(X ≠ µX )T
b ≠ AµX ≠ ◆
Ë È
= AE (X ≠ µX )(X ≠ µX )T = ACX = CT
XY

Dept. SSR (ETSIT-UPM) Random variables Course 2022/2023 25 / 47

Bivariate Gaussian random variable I

A bivariate Gaussian RV is a bidimensional random vector whose

components X1 and X2 are jointly Gaussian RVs. The joint pdf is:
Ë! "2 !x "! x " !x "2 È
1 1 x1 ≠µ1 1 ≠µ1 2 ≠µ2 2 ≠µ2
≠ ≠2fl +
2(1≠fl2 )
f (x1 , x2 ) = e
‡1 ‡1 ‡2 ‡2

2fi ‡12 ‡22 (1 ≠ fl2 )

where

µ1 © µX1 = E (X1 ) µ2 © µX2 = E (X2 )

‡12 © ‡X2 1 = Var (X1 ) ‡22 © ‡X2 2 = Var (X2 )
Cov (X1 , X2 )
fl © flX 1 X 2 = 
‡12 ‡22

Dept. SSR (ETSIT-UPM) Random variables Course 2022/2023 26 / 47

Bivariate Gaussian random variable II

Using vector notation X = [X1 , X2 ]T can be denoted as X ≥ N (µX , CX )

and the joint pdf is
5 6
1 1
f (x) =  exp ≠ (x ≠ µX )T C≠1
X (x ≠ µX )
2fi det (CX ) 2

where x = [x1 , x2 ]T and parameters µX and CX are the mean vector and
covariance matrix.
Theorem: If X ≥ N (µX , CX ) and Y = AX + b, we obtain:
Y ≥ N (µY , CY ), with µY = AµX + b, CY = ACX AT

Dept. SSR (ETSIT-UPM) Random variables Course 2022/2023 27 / 47

Bivariate Gaussian random variable III

The marginal and conditional pdfs are:

(x1 ≠µ1 )2
Œ
1 ≠
2‡ 2
fX1 (x1 ) = fX1 ,X2 (x1 , x2 )dx2 = Ò e 1
≠Œ 2fi‡12
(x2 ≠µ2 )2
Œ
1 ≠
2‡ 2
fX2 (x2 ) = fX1 ,X2 (x1 , x2 )dx1 = Ò e 2
≠Œ 2fi‡22
2
(x1 ≠µX1 |X2 )
fX ,X (x1 , x2 ) 1 ≠
2‡ 2
fX1 (x1 | x2 ) = 1 2 =Ò e X1 |X2
fX2 (x2 ) 2fi‡X2 1 |X2

Dept. SSR (ETSIT-UPM) Random variables Course 2022/2023 28 / 47

Bivariate Gaussian random variable IV

where
‡1
µX1 |X2 = E (X1 |X2 = x2 ) = µ1 + fl (x2 ≠ µ2 )
‡2
1 2
‡X2 1 |X2 = Var (X1 |X2 = x2 ) = ‡12 1 ≠ fl2

Notice that E (X1 |X2 = x2 ) is a linear function of x2 (regression line).

Therefore, for a bivariate Gaussian RV, we have:
The conditional and marginal distributions are also Gaussian.
If their components are uncorrelated (fl = 0), they are also
independent (uncorrelatedness is equivalent to independence).

Dept. SSR (ETSIT-UPM) Random variables Course 2022/2023 29 / 47

N-dimensional random vector. Characterization I

The concepts of a two-dimensional random vector are easily extended to

an n-dimensional random vector.

X = [X1 , X2 , . . . , XN ] T

Joint cumulative distribution (CDF), density (pdf) and probability

mass (pmf) functions:

FX (x) © F (x1 , x2 , . . . , xN ) = P (X1 Æ x1 , X2 Æ x2 , . . . , XN Æ xN ) , ’x œ RN

ˆ n FX (x)
fX (x) © f (x1 , x2 , . . . , xN ) = , ’x œ RN
ˆx1 ˆx2 · · · ˆxN
pX (x) © p(x1 , x2 , . . . , xN ) = P (X1 = x1 , X2 = x2 , . . . , XN = xN ) , ’x œ X

where x = [x1 , x2 , . . . , xN ]T

Dept. SSR (ETSIT-UPM) Random variables Course 2022/2023 30 / 47

N-dimensional random vector. Characterization II

Marginal pdfs and pmfs:

fxi (xi ) = f (x1 , x2 , . . . , xN )dx1 · · · dxi≠1 dxi+1 · · · dxN

RN≠1
ÿ
pxi (xi ) = p(x1 , x2 , . . . , xN )
xj œ Xj
j”=i

Conditional pdfs and pmfs:

f (x1 , x2 , . . . , xN )
fX1 (x1 | x2 , x3 , . . . , xN ) =
f (x2 , x3 , . . . , xN )
p(x1 , x2 , . . . , xN )
pX1 (x1 | x2 , x3 , . . . , xN ) =
p(x2 , x3 , . . . , xN )

Dept. SSR (ETSIT-UPM) Random variables Course 2022/2023 31 / 47

N-dimensional random vector. Moments I

Mean vector and correlation and covariance matrices:

µX = E (X) = [E (X1 ), E (X2 ), . . . , E (XN )]T

RX = E (XXT )
S T
E (X12 ) E (X1 X2 ) ··· E (X1 XN )
W
W E (X2 X1 ) E (X22 ) ··· E (X2 XN ) X
X
=W
W .. .. .. .. X
X
U . . . . V
E (XN X1 ) E (XN X2 ) · · · E (XN2 )
Ë È
CX = E (X ≠ µX )(X ≠ µX )T = RX ≠ µX µT
X
S T
Var(X1 ) Cov(X1 , X2 ) ··· Cov(X1 , XN )
W X
W Cov(X2 , X1 ) Var(X2 ) ··· Cov(X2 , XN ) X
=W
W .. .. .. .. X
X
U . . . . V
Cov(XN , X1 ) Cov(XN , X2 ) · · · Var(XN )
Dept. SSR (ETSIT-UPM) Random variables Course 2022/2023 32 / 47
N-dimensional random vector. Moments II

Conditional moments

E (X1 | X2 = x2 , X3 = x3 , . . . , XN = xN ) =
I Œ
x f (x | x2 , x3 , . . . , xN )dx1
≠Œ 1 X1 1
continuous
= q
x1 œ X x1 pX1 (x1 | x2 , x3 , . . . , xN )
1
discrete

Var (X1 | X2 = x2 , X3 = x3 , . . . , XN = xN ) =
E (X12 | X2 = x2 , X3 = x3 , . . . , XN = xN )≠E 2 (X1 | X2 = x2 , , X3 = x3 , . . . , XN = xN )

Dept. SSR (ETSIT-UPM) Random variables Course 2022/2023 33 / 47

Matrices of cross-correlations and cross-covariances

Given X = [X1 , X2 , . . . , XN ]T and Y = [Y1 , Y2 , . . . , YM ]T , their

cross-correlation and cross-covariance matrices are:
S T
E (X1 Y1 ) E (X1 Y2 ) ··· E (X1 YM )
W X
W E (X2 Y1 ) E (X2 Y2 ) ··· E (X2 YM ) X
RXY = E (XYT ) = W
W .. .. .. .. X = RT
X YX
U . . . . V
E (XN Y1 ) E (XN Y2 ) · · · E (XN YM )
Ë È
CXY = E (X ≠ µX )(Y ≠ µY )T
S T
Cov(X1 , Y1 ) Cov(X1 , Y2 ) ··· Cov(X1 , YM )
W X
W Cov(X2 , Y1 ) Cov(X2 , Y2 ) ··· Cov(X2 , YM ) X
=W .. .. .. X = CT
W .. X YX
U . . . . V
Cov(XN , Y1 ) Cov(XN , Y2 ) · · · Cov(XN , YM )

Dept. SSR (ETSIT-UPM) Random variables Course 2022/2023 34 / 47

Independence, uncorrelatedness and orthogonality

X1 , X2 , . . . , XN are independent iff:

Continuous RV
T
f (x1 , x2 , . . . , xN ) = fX1 (x1 )fX2 (x2 ) · · · fXN (xN ) ’ [x1 , x2 , . . . , xN ] œ RN

Discrete RV
T
p(x1 , x2 , . . . , xN ) = pX1 (x1 )pX2 (x2 ) · · · pXN (xN ) ’ [x1 , x2 , . . . , xN ] œ X

X1 , X2 , . . . , Xn are uncorrelated iff E (Xi Xj ) = E (Xi ) E (Xj ) , ’i ”= j,

or equivalently: CXi Xj = 0, ’i ”= j
X1 , X2 , . . . , Xn are orthogonal iff E (Xi Xj ) = 0, ’i ”= j or, equivalently:
RXi Xj = 0, ’i ”= j

Dept. SSR (ETSIT-UPM) Random variables Course 2022/2023 35 / 47

N-dimensional random vector. One transformation

Y = g(X1 , X2 , . . . , XN ) © g(X)

Moments
If X1 , X2 , . . . , XN are continuous Rvs:

E [g(X)] = g(x1 , x2 , . . . , xN )fX (x1 , x2 , . . . , xN )dx1 dx2 · · · dxN

X

If X1 , X2 , . . . , XN are discrete RVs:

ÿ
E [g(X)] = g(x1 , x2 , . . . , xN )pX (x1 , x2 , . . . , xN )
(x1 ,x2 ,...,xN )œ X

Dept. SSR (ETSIT-UPM) Random variables Course 2022/2023 36 / 47

N-dimensional random vector. One linear transformation

Case study: one linear transformation

Y = g(X) = a1 X1 +a2 X2 +· · ·+aN XN +b = aT X+b = XT a+b

where a = [a1 , a2 , . . . , aN ]T

E (aT X + b) = aT µX + b = µT
Xa + b
N
ÿ N ÿ
ÿ N
Var(aT X + b) = ai2 Var(Xi ) + ai aj Cov(Xi , Xj )
i=1 i=1 j=1
j”=i

= aT CX a

If X1 , X2 , . . . , XN are uncorrelated:
Var(a1 X1 + a2 X2 + · · · + aN XN + b) = a12 Var(X1 ) + a22 Var(X2 ) + · · · + aN2 Var(XN )

Dept. SSR (ETSIT-UPM) Random variables Course 2022/2023 37 / 47

N-dimensional random vector. M linear transformations

Y = g(X)
where X = [X1 , X2 , . . . , XN ]T , Y = [Y1 , Y2 , . . . , YM ]T and
g = [g1 (·), g2 (·), . . . , gM (·)]T
Case study: M linear transformations

Y = AX + b

where A is an M ◊ N matrix and b is an M ◊ 1 vector.

Moments of Y = AX + b

µY = AµX + b
CY = ACX AT
CXY = CX AT
CYX = ACX = CT
XY

Dept. SSR (ETSIT-UPM) Random variables Course 2022/2023 38 / 47

N-dimensional random vector. Multivariate Gaussian I

A multivariate Gaussian RV X = [X1 , X2 , . . . , XN ]T has the joint pdf:

5 6
1 1
fX (x) = N/2 
exp ≠ (x ≠ µX )T C≠1
X (x ≠ µX )
(2fi) det (CX ) 2

and is written as X s N (µX , CX ), where µX is the mean vector and CX is

the covariance matrix.
The components of X are jointly Gaussian RVs.
Theorem: If X s N (µX , CX ) and Y = AX + b, then Y s N (µY , CY ),
with µY = AµX + b, CY = ACX AT

Dept. SSR (ETSIT-UPM) Random variables Course 2022/2023 39 / 47

N-dimensional random vector. Multivariate Gaussian II
If X s N (µ, C) is split in two vectors with dimensions M and N ≠ M:
S T
X1
W M◊1 X
X=U V X1 = [X1 , X2 , . . . , XM ]T X2 = [XM+1 , XM+2 , . . . , XN ]T
X2
(N≠M)◊1

1 Vectors X1 and X2 are multivariate Gaussian: X1 s N (µ1 , C11 ),

X2 s N (µ2 , C22 ) with
S T S T
µ1 C11 C12
W M◊1 X W M◊M M◊(N≠M) X
µ=U V C=U V
µ2 C21 C22
(N≠M)◊1 (N≠M)◊M (N≠M)◊(N≠M)

2 The conditional distribution of X1 given X2 is multivariate Gaussian:

1 2
(X1 | X2 = x2 ) ≥ N µ1 + CT T ≠1
21 C22 (x2 ≠ µ2 ), C11 ≠ C21 C22 C21
≠1

Note that E (X1 | X2 = x2 ) = µ1 + CT 21 C22 (x2 ≠ µ2 ) is a linear

≠1

function of x2 (regression hyperplane)

Dept. SSR (ETSIT-UPM) Random variables Course 2022/2023 40 / 47
Exercise 1

Let us consider
! "
that we have a (non observable) Gaussian random variable
Y ≥ N 0, ‡Y2 from whom we obtain two other observed random
variables X1 , X2 defined as follows:

X1 = AY + B
X2 = AY 2 + D
! " ! " ! "
where A ≥ N µA , ‡A2 , B ≥ N 0, ‡B2 and D ≥ N 0, ‡D 2 are also

Gaussian random variables, independent among them and also with

respect to Y . The objective of this exercise is to calculate two constants
–, — defining the new random variable5 Ŷ = –X16 + —X2 to be obtained
1 22
minimizing the cost function › = E Y ≠ Ŷ . For that purpose, let us
proceed this way:

Dept. SSR (ETSIT-UPM) Random variables Course 2022/2023 41 / 47

Exercise 1 a)

# $ # $
a) Calculate the following statistics: E X12 , E X22 , E [YX1 ], E [YX2 ] ,
E [X1 X2 ]. Verify that E [X1 X2 ] = 0 , E [YX2 ] = 0.
d› d›
b) Calculate the equations d– = 0 and d— = 0 expressing the result just in
terms of those statistics found in a) (it is not necessary to substitute).
c) Calculate –, — expressing the results just in terms of previous statistics
found in a). Assume that E [X1 X2 ] = 0 , E [YX2 ] = !
0. "
Note : If Z is a Gaussian
# 2$
random #variable
$
Z ≥#N $0, ‡Z2 , we have the
following results: E Z = ‡Z , E Z = 0, E Z 4 = 3‡Z4 .
2 3

Dept. SSR (ETSIT-UPM) Random variables Course 2022/2023 42 / 47

Exercise 1 a)
# $ Ë È # $ # $
E X12 = E (AY + B)2 = E A2 Y 2 + E B 2 + 2E [AYB] =
# $ # $ # $ *0 *0 ! "
E A2 E Y 2 + E B 2 + 2E [A] E [Y ]E [B] = µ2A + ‡A2 ‡Y2 + ‡B2
# $ Ë! "2 È # $ # $ # $
E X22 = E AY 2 + D = E A2 Y 4 + E D 2 + 2E AY 2 D =
# $ # $ # $ # $ *0 ! "
E A2 E Y 4 + E D 2 + 2E [A] E Y 2 E [D] = 3 µ2A + ‡A2 ‡Y4 + ‡D
2
# $
E [YX1 ] = E [Y (AY + B)] = E AY 2 + E [YB] =
# $ *0 *0
E [A] E Y 2 + E [Y ]E [B] = µA ‡Y2
# ! 2
"$ # $
E [YX2 ] = E Y AY + D = E AY 3 + E [YD] =
# *0 *0
$ 0
⇠
:
E ⇠Y⇠
E [A]⇠ 3 + E [Y ]E [D] = 0
# ! 2+D
"$
E [X1 X2
# 2 3$
] = E (AY + B) #
AY $
=
2
E A Y + E [AYD] + E BAY + E [BD] =
# $ # ⇠ $ 0
:
⇠ *0 *0 *0 # $ *0 *0
E A2 ⇠ E ⇠Y 3 + E [A] E [Y ]E [D] + E [B]E [A] E Y 2 + E [B]E [D] = 0

Dept. SSR (ETSIT-UPM) Random variables Course 2022/2023 43 / 47

Exercise 1 b)
Taking into account that Ŷ = –X1 + —X2 , we get:
; 51 C D
ˆ› ˆ 22 6< 1 2 ˆ Ŷ
= E Y ≠ Ŷ = ≠2E Y ≠ Ŷ
ˆ– ˆ– ˆ–
= ≠2E [(Y ≠ –X1 ≠ —X2 ) X1 ]
; 51 C D
ˆ› ˆ 22 6< 1 2 ˆ Ŷ
= E Y ≠ Ŷ = ≠2E Y ≠ Ŷ
ˆ— ˆ— ˆ—
= ≠2E [(Y ≠ –X1 ≠ —X2 ) X2 ]

Therefore
ˆ› Ë È
= 0 =∆ E [YX1 ] ≠ –E X12 ≠ —E [X1 X2 ] = 0
ˆ–
ˆ› Ë È
= 0 =∆ E [YX2 ] ≠ –E [X1 X2 ] ≠ —E X22 = 0
ˆ—

Dept. SSR (ETSIT-UPM) Random variables Course 2022/2023 44 / 47

Exercise 1 c)

Taking into account that E [X1 X2 ] = 0 and E [YX2 ] = 0, we find:

ˆ› :0
⇠
⇠2⇠ ⇠⇠:0 Ë È
2
= 0 =∆ ⇠
E⇠[YX ] ≠ –⇠
E⇠[X⇠
1 X2 ] ≠ —E X2 = 0 =∆ — = 0
ˆ—
ˆ› Ë È
⇠⇠:0
= 0 =∆ E [YX1 ] ≠ –E X12 ≠ —⇠ E⇠
[X⇠1 X2 ] = 0 =∆
ˆ–
E [YX1 ] µA ‡Y2
=∆ – = # 2 $ =∆ – = ! 2 "
E X1 µA + ‡A2 ‡Y2 + ‡B2

Dept. SSR (ETSIT-UPM) Random variables Course 2022/2023 45 / 47

Exercise 3

Assume that X is a random observable vector of dimension N ◊ 1, H is a

known matrix of dimension N ◊ p, Y is a random vector of dimension
p ◊ 1, with zero mean and covariance matrix CY , and Z is a random
vector of dimension N ◊ 1, Z ≥ N (0,CZ ) and independent of Y. Next
figure shows the dependence between these variables.

Calculate the following statistics µX , CXY , CX .

Dept. SSR (ETSIT-UPM) Random variables Course 2022/2023 46 / 47

Exercise 3
Solution: As X = HY + Z

*0 *0
µX = E (X) = E (HY + Z) = HE (Y) + E (Z) = 0
Ë È Ë È
CXY = E (X ≠ µX ) (Y ≠ µY )T = E XYT = RXY
Ë È Ë È
CX = E (X ≠ µX ) (X ≠ µX )T = E XXT = RX

The matrices of correlation and cross-correlation are:

Ë È Ë È Ë È Ë È 0
*
RXY = E XYT = E (HY + Z) YT = HE YYT + E ZYT = HCY
Ë È Ë È
RX = E XXT = E (HY + Z) (HY + Z)T
Ë È Ë È 0
* Ë È 0
* Ë È
T T T T
= HE YY H + HE YZ + E ZY HT + E ZZT
= HCY HT + CZ

Dept. SSR (ETSIT-UPM) Random variables Course 2022/2023 47 / 47