MATH 60093 Monte Carlo Modeling

Generating Multivariate Normal

A multivariate normal distribution X = (X1 ; :::; Xd ) N ( ; ) is speci…ed by its mean vector = ( 1 ; :::; d )
and covariance matrix = ( ij )d d where ij = cov (Xi ; Xj ) = E [(Xi i ) (Xj j )] and each component
Xi has distribution N i ; i2 with i2 = ii :
To qualify as a covariance matrix, must be symmetric 1 and positive semide…nite 2 : If is positive
3
de…nite then the normal distribution N ( ; ) has density
1 1 T 1
fX (x) = d=2 1=2
exp (x ) (x ) ; x 2 Rd :
(2 ) j j 2
with j j the determinant of : The standard d-dimensional normal N (0; I) with I the d d identity matrix
is the special case
1 1 T
fX (x) = d=2
exp x x :
(2 ) 2
In specifying a multivariate normal distribution, it is sometimes convenient to specify the marginal
standard deviations i ; i = 1; :::; d and the correlations ij of Xi and Xj from which the entries ij of the
covariance matrix can be determined by
ij = i j ij :

In matrix form, it is expressed as

0 10 10 1
1 11 12 1d 1
B C B 21 CB C
B 2 CB 22 2d CB 2 C
=B .. C B .. .. .. .. CB .. C:
@ . A@ . . . . A@ . A
d 1d 2d dd d

An important property of the multivariate normal distribution is the linear transformation property.
Proposition 1 (Linear Transformation Property) Any linear transformation of a normal vector is
again normal.
X N ( ; ) =) AX N A ; A AT
for any d-vector ; and d d matrix ; and any k d matrix A; for any k:
By the linear transformation property, we know that if Z N (0; I) and X = + AZ; then X
N ; AAT : Once we generate independent standard normal random variables Z1 ; :::; Zd and assemble them
into a vector Z N (0; I) ; then the problem of sampling X from the multivariate normal N ( ; ) reduces
to …nding a matrix A for with AAT = :

Cholesky Factorization
Among all such matrix A such that AAT = ; a lower triangular matrix is particularly convenient because
it reduces the calculation of + AZ to the following:
X1 = 1 + a11 z1
X2 = 2 + a21 z1 + a22 z2
..
.
Xd = d + ad1 z1 + ad2 z2 + + add zd
A Cholesky factorization of is a representation of as AAT where A is a lower triangular matrix. If is
positive de…nite then it has a Cholesky factorization and the matrix A is unique up to changes in sign.
1 = T:
2 xT x 0 or equivalently all eigenvalues of are non-negative
3 xT x > 0 or equivalently all eigenvalues of are positive.

1
Example 2 Consider a 2 2 covariance matrix ; represented as
2
1 1 2
= 2 :
1 2 2

Assuming 1 > 0 and 2 > 0; the Cholesky factor is

1 0p
A= 2
;
2 1 2

as is easily veri…ed by evaluating AAT : Thus, we can sample from a bivariate normal distribution N ( ; )
by setting

X1 = 1 + 1 z1
p
X2 = + z1 + 1 2z
2 2 2 2

with z1 ; z2 are independent standard normals.

For the general case where is a d d covariance matrix, we need to solve

0 10 1
a11 a11 a21 ad1
B a21 a22 CB a22 ad2 C
B CB C
B .. .. . . C B . . .. C=
@ . . . A @ . . A
ad1 ad2 add add

Traversing the ij by looping over j = 1; :::; d and then i = j; :::; d produces the equations

a211 = 11
a21 a11 = 21
..
.
ad1 a11 = 11

a221 + a2dd = 22

a2d1 + + a2dd = dd :

Exactly one new entry of the matrix A appears in each equation, making it possible to solve for the individual
entries sequentially. More compactly, from the basic identity
j
X
ij = aik ajk , j i;
k=1

we get
Pj 1
ij k=1 aik ajk
aij = ,j<i
ajj
and v
u i 1
u X
aii = t ii a2ik :
k=1

The following algorithm provide a simple recursion to …nd the Cholesky factor.

2
Algorithm 3 Input: Symmetric positive definite matrix d d matrix :
Output: Lower triangular matrix A with AAT = :

A 0 (d d zero matrix)
for j = 1; :::; d
for i = j; :::; d
vi ij
for k = 1; :::; j 1
vi vi ajk aik
p
aij vi = vj
return A:

Eigenvector Factorization
The equation AAT = can also be solved by diagonalizing : As a symmetric d d matrix, has d
real eigenvalues 1 ; :::; d ; and because must be positive de…nite or semide…nite, the i are non-negative.
Furthermore, has an associated orthonormal set of eigenvectors fv1 ; :::; vd g ; i.e. vectors satisfying

viT vi = 1
viT vj = 0 for j 6= i; i; j = 1; :::; d:

and
vi = i vi :

It follows that = V V T ; where V is the orthogonal matrix4 with columns v1 ; :::; vd and is the diagonal
matrix with diagonal entries 1 ; :::; d : Hence, if we choose
1=2
A=V
0 p 1
1 p
B C
B 2 C
=V B .. C
@ . A
p
d

then
AAT = V V T = :
Methods for calculating V and are included in many mathematical software.libraries.

4V VT =I