Lecture Notes 8

Random Processes in Linear Systems

• Linear System with Random process Input

• LTI System with WSS Process Input

• Process Linear Estimation

◦ Infinite smoothing filter
◦ Spectral Factorization
◦ Wiener Filter

EE 278: Random Processes in Linear Systems Page 8 – 1

 Linear System with Random Process Input

• Consider a linear system with (time-varying) impulse response h(t, t − τ ) driven

by a random process input X(t)

X(t) h(t, t − τ ) Y (t)

• The output of the system is

! ∞
Y (t) = h(t, t − τ )X(τ ) dτ
−∞

• We wish to specify the output random process Y (t)

• It is difficult to obtain a complete specification of the output process in general
• Important special case: If X(t) is a GRP, the output process Y (t) is also a GRP
(since the integral above can be approximated by a sum and thus the output
process is obtained via a linear transformation of X(t))

EE 278: Random Processes in Linear Systems Page 8 – 2

• We focus on finding the mean and autocorrelation functions of Y (t) in terms of
the mean and autocorrelation functions of the input process X(t) and the
impulse response of the system h(t, t − τ )
We are also interested in finding the crosscorrelation function between X(t) and
Y (t) defined as
RXY (t1, t2) = E(X(t1)Y (t2))
Note that unlike RX (t1, t2), RXY (t1, t2) is not necessarily symmetric in t1 and
t2 . However, RXY (t1, t2) = RY X (t2, t1)
• To find the mean of Y (t), consider
"! ∞ # ! ∞
E(Y (t)) = E h(t, t − τ )X(τ ) dτ = h(t, t − τ ) E(X(τ )) dτ
−∞ −∞

• The crosscorrelation function between Y (t) and X(t) is

RY X (t1, t2) = E(Y (t1)X(t2))
"! ∞ #
=E h(t1, t1 − τ )X(τ )X(t2) dτ
−∞
! ∞
= h(t1, t1 − τ )RX (τ, t2) dτ
−∞

EE 278: Random Processes in Linear Systems Page 8 – 3

• The autocorrelation function of Y (t) is
RY (t1, t2) = E(Y (t1)Y (t2))
"! ∞ #
=E h(t2, t2 − τ )X(τ )Y (t1) dτ
−∞
! ∞
= h(t2 , t2 − τ )RY X (t1, τ ) dτ
−∞
! ∞! ∞
= h(t2, t2 − τ2)h(t1, t1 − τ1)RX (τ1, τ2) dτ1 dτ2
−∞ −∞

The average power is

E(Y 2(t)) = RY (t, t)
• Example (Integrator): Let X(t) be a white noise process with autocorrelation
function RX (τ ) = (N/2)δ(τ ) and let the linear system be an ideal integrator,
i.e., ! t
Y (t) = X(τ ) dτ
0
Find the mean and autocorrelation functions and the average power of the
integrator output Y (t), for t > 0

EE 278: Random Processes in Linear Systems Page 8 – 4

 This example is motivated by several applications:
◦ Noise in an image sensor pixel: the white noise models the photodetector shot
noise, which is integrated with the signal over a capacitor before sampling
◦ Noise in a voltage controlled oscillator (for phase locked loops)
• Solution: The mean is
! t
E(Y (t)) = E(X(τ )) dτ = 0
0
To obtain the autocorrelation function and average power for this case, we can
specialize the previous results to
! t1
N
RY X (t1, t2) = δ(t2 − τ ) dτ
0 2
⎧
⎨ N , for t ≤ t
2 1
= 2
⎩0, otherwise

EE 278: Random Processes in Linear Systems Page 8 – 5

 ⎧
⎨ N t2,
⎪ for t2 ≤ t1
RY (t1, t2) = N2
⎩ t1 otherwise
⎪
2
N
= min{t1, t2}
2

N
E(Y 2(t)) = RY (t, t) =
t
2
Note that the average power grows linearly with t (as for the random walk)
• If in addition X(t) is a GRP, then Y (t) is also a GRP and is referred to as the
Wiener process

EE 278: Random Processes in Linear Systems Page 8 – 6

 LTI System with WSS Process Input

• Consider a linear time invariant (LTI) system with real impulse response h(t) and
transfer function H(f ) = F(h(t)), driven by WSS process X(t), −∞ < t < ∞

X(t) h(t) Y (t)

! ∞
• We want to characterize its output Y (t) = X(t) ∗ h(t) = X(τ )h(t − τ )dτ
−∞

• It
( )turns out (not
( surprisingly) that if the system is stable, i.e.,
( ∞
( −∞ h(t) dt ( = |H(0)| < ∞, then X(t) and Y (t) are jointly WSS, which
(
means that:
◦ X(t) and Y (t) are WSS, and
◦ Their crosscorrelation function RXY (t1, t2) is time invariant, i.e.,
RXY (t1, t2) = E(X(t1)Y (t2)) = RXY (t1 + τ, t2 + τ ) for all τ

EE 278: Random Processes in Linear Systems Page 8 – 7

• Relabel RXY (t1, t2) for jointly WSS X(t), Y (t) as RXY (τ ), where τ = t1 − t2
RXY (τ ) = RXY (t2 + τ, t2) = RXY (t2 + (t1 − t2), t2) = RXY (t1, t2)
Again RXY (τ ) is not necessarily even. However,
RXY (τ ) = RY X (−τ )

• Example: Let Θ ∼ U[0, 2π]. Consider two processes

X(t) = α cos(ωt + Θ) and Y (t) = α sin(ωt + Θ)
These processes are jointly WSS, since each is WSS (in fact SSS) and
* 2 +
RXY (t1, t2) = E α cos(ωt1 + Θ) sin(ωt2 + Θ)
2 ! 2π *
α +
= sin(ω(t1 + t2) + 2θ) − sin(ω(t1 − t2)) dθ
4π 0
α2
= − sin(ω(t1 − t2))
2
• We define the cross power spectral density for jointly WSS processes X(t), Y (t)
as
SXY (f ) = F(RXY (τ ))

EE 278: Random Processes in Linear Systems Page 8 – 8

• Example: Let Y (t) = X(t) + Z(t), where X(t) and Z(t) are zero mean
uncorrelated WSS processes. Show that Y (t) and X(t) are jointly WSS, and
find RXY (τ ) (in terms of RX and RZ ) and SXY (f ) (in terms of SX and SZ )
Solution: First, we show that Y (t) is WSS, since it is zero mean and
* +
RY (t1, t2) = E (X(t1) + Z(t1))(X(t2) + Z(t2))
, - , -
= E X(t1)X(t2) + E Z(t1)Z(t2)
(X(t), Z(t) zero mean, uncorrelated)
= RX (τ ) + RZ (τ )
Taking the Fourier transform of both sides, SY (f ) = SX (f ) + SZ (f )
To show that Y (t) and X(t) are jointly WSS, we need to show that their
crosscorrelation function is time invariant
* +
RXY (t1, t2) = E X(t1)(X(t2) + Z(t2))
, - , -
= E X(t1)X(t2) + E X(t1)Z(t2)
= RX (t1, t2) + 0 (X(t), Z(t) zero mean, uncorrelated)
= RX (τ )
Taking the Fourier transform, SXY (f ) = SX (f )

EE 278: Random Processes in Linear Systems Page 8 – 9

 Output Mean, Autocorrelation, and PSD

Theorem: Let X(t), t ∈ R, be a WSS process input to a stable LTI system with
real impulse response h(t) and transfer function H(f ). Then the input X(t) and
output Y (t) are jointly WSS with:

1. E(Y (t)) = H(0) E(X(t))

2. RY X (τ ) = h(τ ) ∗ RX (τ )
3. RY (τ ) = h(τ ) ∗ RX (τ ) ∗ h(−τ )
RY X (τ )
RX (τ ) h(τ ) h(−τ ) RY (τ )

4. SY X (f ) = H(f )SX (f )
5. SY (f ) = |H(f )|2SX (f )
SY X (f )
SX (f ) H(f ) H(−f ) SY (f )

EE 278: Random Processes in Linear Systems Page 8 – 10

Remark: For a discrete time WSS process X(n) and a stable LTI system h(n),
X(n) and the output process Y (n) are jointly WSS and we can similarly find
RY (n), . . .
Proof: Note that here the LTI system is in steady state

1. To find the mean of Y (t), consider

"! ∞ #
E(Y (t)) = E X(τ )h(t − τ ) dτ
−∞
! ∞
= E(X(τ )) h(t − τ ) dτ
−∞
! ∞
= E(X(t)) h(t − τ ) dτ = E(X(t))H(0)
−∞

2. To find the crosscorrelation function between Y (t) and X(t), consider

, -
RY X (τ ) = E Y (t + τ )X(t)
"! ∞ #
=E h(α)X(t + τ − α)X(t) dα
−∞

EE 278: Random Processes in Linear Systems Page 8 – 11

 ! ∞
= h(α)RX (τ − α) dα
−∞

= h(τ ) ∗ RX (τ )

3. To find the autocorrelation function of Y (t), consider

RY (τ ) = E(Y (t + τ )Y (t))
" ! ∞ #
= E Y (t + τ ) h(α)X(t − α) dα
−∞
! ∞
= h(α)RY X (τ + α) dα
−∞

= RY X (τ ) ∗ h(−τ )

4. Follows by taking the Fourier transform of RY X (τ )

5. Follows by taking the Fourier transform of RY (τ )

EE 278: Random Processes in Linear Systems Page 8 – 12

 SX (f ) is the Power Spectral Density

• We can use the above results to show that SX (f ) is indeed the power spectral
density of X(t); i.e., the average power in any frequency band [f1, f2] is
! f2
2 SX (f ) df
f1

• To show this we pass X(t) through an ideal band-pass filter

X(t) h(t) Y (t)

H(f )

f
−f2 −f1 f1 f2

EE 278: Random Processes in Linear Systems Page 8 – 13

• Now the average power of X(t) in the band [f1, f2] is
! ∞
E(Y 2(t)) = SY (f ) df
−∞
! ∞
= |H(f )|2SX (f ) df
−∞
! −f1 ! f2
= SX (f ) df + SX (f ) df
−f2 f1
! f2
=2 SX (f ) df
f1

• This also shows that SX (f ) ≥ 0 for all f

EE 278: Random Processes in Linear Systems Page 8 – 14

 KT /C Noise

• The noise in a resistor R (in ohms) due to thermal noise is modeled as a WGN
voltage source V (t) in series with R. The psd of V (t) is
SV (f ) = 2kT R V2/Hz for all f , where k is Boltzmann’s constant and T is the
temperature in degrees K

V (t)
R

• Now let’s find the average output noise power for an RC circuit

C
V (t) V0(t)

EE 278: Random Processes in Linear Systems Page 8 – 15

• First we find the transfer function for the circuit
1 1
H(f ) = ⇒ |H(f )|2 =
1 + i2πf RC 1 + (2πf RC)2
• Now we write the output psd in terms of the input psd as
2 1
SVo = SV (f )|H(f )| = 2kT R , −∞ < f < ∞
1 + (2πf RC)2
• Thus the average output power is
! ∞
E(Vo2(t)) = SVo (f )df
−∞
! ∞
2kT R 1
= d(2πf RC)
2πRC −∞ 1 + (2πf RC)2
! ∞
kT 1
= 2
dx
πC −∞ 1 + x
(+∞
kT ( kT kT
= arctan x(( = π = ,
πC −∞
πC C

which is independent of R!

EE 278: Random Processes in Linear Systems Page 8 – 16

 Autoregressive Moving Average Process

• Let Xn , −∞ < n < ∞, be a discrete time white noise process with average
power N
The autoregressive moving average (ARMA) process of order (p, q), Yn ,
−∞ < n < ∞, is defined as
p
. q
.
Yn = − αk Yn−k + βlXn−l
k=1 l=0

where β0 = 1, α1, . . . , αp and β1, . . . , βq are fixed parameters

• This process can be viewed as the output of an LTI system with transfer function
/q
1 + l=1 βle−i2πf l 1
H(f ) = , |f | <
1 + pk=1 αk e−i2πf k
/
2

Therefore, the PSD of Yn is SY (f ) = |H(f )|2N for |f | < 1/2

EE 278: Random Processes in Linear Systems Page 8 – 17

• Moving average (MA) process of order q: Let α1 = · · · = αp = 0, then Yn is
simply a weighted sum of the q + 1 most recent Xn samples with weights
(1, β1, . . . , βq ), i.e.,
q
.
Yn = βlXn−l , and the transfer function of the LTI system is
l=0
q
. 1
H(f ) = 1 + βl e−i2πf l , |f | <
2
l=1

• Communication channel with intersymbol interference: The Xn process

represents the transmitted information symbols and 1, β1, . . . , βq are the
coefficients of the channel impulse response
The process Yn is the interference-impaired received symbols
• Autoregressive (AR) process of order p: Let β1 = · · · = βn = 0. Then
p
.
Yn = − αk Yn−k + Xn, and the transfer function of the LTI system is
k=1
1 1
H(f ) = /p k
, |f | <
1+ k=1 αk e
−i2πf 2

EE 278: Random Processes in Linear Systems Page 8 – 18

• Modeling the human speech generation process: The process Xn is generated
by the vocal cords. The vocal tract is modeled as a series of coupled lossless
acoustic tubes parameterized by (α1, . . . , αp )
The process Yn is the uttered speech signal after it passes through the vocal
tract
• For p = 1, we obtain the first-order autoregressive process
Yn = −α1Yn−1 + Xn,
1 1
H(f ) = , |f | <
1 + α1e−i2πf 2
h(n) = (−α1)n u(n)
This transfer function is stable iff ∞
/
n=−∞ |h(n)| < ∞, i.e., iff |α1| < 1
If Xn is Gaussian, we obtain a stationary version of the Gauss–Markov process
discussed in Lecture Notes 6 with α = −α1

EE 278: Random Processes in Linear Systems Page 8 – 19

 Sampling Theorem for Bandlimited WSS Processes

• Recall the Nyquist sampling theorem for bandlimited deterministic signals:

◦ Let x(t) be a signal with Fourier transform X(f ) such that X(f ) = 0 for
f∈/ [−B, B]
◦ We sample the signal at rate 1/T to obtain the sampled signal
yn = x(nT ) for n = . . . , −2, −1, 0, 1, 2, . . .
The Fourier transform of the sequence yn ,
.∞
Y (f ) = X(f − n/T ),
n=−∞

is periodic with period 1/T

◦ To recover the signal, we pass yn through an ideal low pass filter of
bandwidth 1/T . The Fourier transform of the reconstructed signal is
X̂(f ) = Y (f ) · ⊓(f T )

◦ Hence if the sampling rate 1/T ≥ 2B, X̂(f ) = X(f ) and the signal can be
reconstructed perfectly from its samples

EE 278: Random Processes in Linear Systems Page 8 – 20

• It turns out that a similar result holds for sampling of bandlimited WSS random
processes
• Sampling theorem for WSS processes:
◦ Let X(t) be a continuous time WSS process with zero mean and
autocorrelation function RX (τ ) and PSD SX (f ) = 0 for f ∈
/ [−B, B]
◦ We sample X(t) at rate 1/T to obtain the sampled (discrete time) process
Yn = X(nT ) with
µY (n) = E(Yn) = E(X(nT )) = 0,
RY (n1, n2) = E(Yn1 Yn2 ) = E(X(n1T )X(n2T )) = RX ((n1 − n2)T )
Hence Yn is WSS with zero mean and autocorrelation function
RY (n) = RX (nT )

The PSD of Yn ,
∞
.
SY (f ) = SX (f − n/T ),
n=−∞
is periodic with period 1/T

EE 278: Random Processes in Linear Systems Page 8 – 21

 ◦ As for the deterministic signal case, to reconstruct the RP X(t), we pass Yn
through an ideal low pass filter
The resulting reconstruction process X̂(t) is WSS with PSD
SX̂ (f ) = SY (f )| ⊓ (f T )|2

◦ Hence if the sampling rate 1/T ≥ 2B, SX̂ (f ) = SX (f )

◦ We show that this implies that the reconstruction process X̂(t) = X(t) for
every t with probability one. Specifically, we show that if 1/T ≥ 2B,
2
* +
E (X(t) − X̂(t)) = 0 for every t

◦ Proof: We know that if 1/T ≥ 2B, SX̂ (f ) = SX (f ), which implies that

RX̂ (τ ) = RX (τ )
Moreover, 0τ 1
RX̂X (τ ) = sinc ∗ RX (τ )
T
Now, consider
2
* +
E (X(t) − X̂(t)) = RX (0) + RX̂ (0) − 2RX̂X (0) = 2RX (0) − 2RX (0) = 0
Hence, X̂(t) = X(t) w.p.1 for every t

EE 278: Random Processes in Linear Systems Page 8 – 22

 Process Linear Estimation

• Let X(t) and Y (t) be zero mean jointly WSS processes with known
autocorrelation and crosscorrelation functions RX (τ ), RY (τ ), and RXY (τ )
• We observe the random process Y (α) for t − a ≤ α ≤ t + b (−a ≤ b) and wish
to find the*MMSE linear estimate of the signal X(t), i.e., X̂(t) such that the
MSE = E (X(t) − X̂(t))2 is minimized
+

• The linear estimate is of the form

! a
X̂(t) = h(τ )Y (t − τ ) dτ
−b

• By the orthogonality principle, the MMSE linear estimate must satisfy

, -
X(t) − X̂(t) ⊥ Y (t − τ ) , −b ≤ τ ≤ a
or * +
E (X(t) − X̂(t))Y (t − τ ) = 0 , −b ≤ τ ≤ a

EE 278: Random Processes in Linear Systems Page 8 – 23

 Thus, for −b ≤ τ ≤ a, we must have
* + * +
RXY (τ ) = E X(t)Y (t − τ ) = E X̂(t)Y (t − τ )
"! a #
=E h(α)Y (t − α)Y (t − τ ) dα
−b
! a
= h(α)RY (τ − α) dα
−b

So, to find h(α) we need to solve an infinite set of integral equations

• Solving these equations analytically is not possible in general. However, it can
be done for two important special cases:
◦ Infinite smoothing: when a, b → ∞
◦ Filtering: when a → ∞ and b = 0 (Wiener–Hopf equations)

EE 278: Random Processes in Linear Systems Page 8 – 24

 Infinite Smoothing Filter

• When a, b → ∞, the integral equations for the MMSE linear estimate become
! ∞
RXY (τ ) = h(α)RY (τ − α) dα , −∞ < τ < +∞
−∞

In other words,
RXY (τ ) = h(τ ) ∗ RY (τ )
• The Fourier transform convolution theorem gives the transfer function for the
optimal infinite smoothing filter:
SXY (f )
SXY (f ) = H(f )SY (f ) ⇒ H(f ) =
SY (f )

EE 278: Random Processes in Linear Systems Page 8 – 25

• The minimum MSE is
2
* +
MSE = E (X(t) − X̂(t))
* + * +
= E (X(t) − X̂(t))X(t) − E (X(t) − X̂(t))X̂(t)
* +
= E (X(t) − X̂(t))X(t) (by orthogonality)
2
* + * +
= E (X(t) − E X(t)X̂(t)

To evaluate the second term, consider

RX X̂ (τ ) = E(X(t + τ )X̂(t))
0 ! ∞ 1
= E X(t + τ ) h(α)Y (t − α) dα
−∞
! ∞
= h(α)RXY (τ + α) dα = RXY (τ ) ∗ h(−τ )
−∞

Therefore
∞
|SXY (f )|2 ∞
! !
E(X(t)X̂(t)) = RX X̂ (0) = H(−f )SXY (f ) df = df ,
−∞ −∞ SY (f )

EE 278: Random Processes in Linear Systems Page 8 – 26

 and the minimum MSE is
2 2
* + * + , -
E (X(t) − X̂(t)) = E (X(t) − E X(t)X̂(t)
|SXY (f )|2
! ∞ ! ∞
= SX (f ) df − df
−∞ −∞ SY (f )
|SXY (f )|2
! ∞" #
= SX (f ) − df
−∞ S Y (f )

• Example (Additive White Noise Channel): Let X(t) and Z(t) be zero mean
uncorrelated WSS processes with
2
P
|f | ≤ B
SX (f ) = 2
0 otherwise
N
SZ (f ) = for all f
2
Here the signal X is bandlimited white noise, and Z is white noise
Find the optimal infinite smoothing filter for estimating X(t) given
Y (τ ) = X(τ ) + Z(τ ) , −∞ < τ < +∞
and the MSE for the estimate produced by this filter

EE 278: Random Processes in Linear Systems Page 8 – 27

 The power spectral densities of X and Z are shown below

SX (f )
P
2

f
−B SZ (f ) B

N
2

f
H(f )
P
P +N

f
−B B

EE 278: Random Processes in Linear Systems Page 8 – 28

• The transfer function of the optimal infinite smoothing filter is given by
SXY (f )
H(f ) =
SY (f )
SX (f )
=
SX (f ) + SZ (f )
P
2
P +N |f | ≤ B
=
0 otherwise

The MMSE is given by

∞
|SXY (f )|2
∞
! !
MSE = SX (f ) df − df
−∞ −∞ SY (f )
! +B ! +B
P (P/2)2
= df − df
−B 2 −B P/2 + N/2

P 2/4
= PB − 2B
(P + N )/2
NP B
=
N +P

EE 278: Random Processes in Linear Systems Page 8 – 29

 Spectral Factorization
• It can be shown that the power spectral density SX (f ) of WSS process X(t)
has a square root, i.e., a transfer function H(f ) such that
SX (f ) = H(f )H ∗(f ) = |H(f )|2
This is similar to the square root of a covariance (correlation) matrix for a
random vector discussed in Lecture notes 4
• As for the random vector case, the square root of a PSD, H(f ), and its inverse
1/H(f ) can be used for coloring and whitening of WSS processes, e.g.,
◦ Coloring:

X(t) H(f ) Y (t)

SX (f ) = 1 SY (f ) = S(f )

EE 278: Random Processes in Linear Systems Page 8 – 30

 ◦ Whitening:

Y (t) 1/H(f ) X(t)

SY (f ) = S(f ) SX (f ) = 1
Here X(t) is the innovation process of Y (t)
• It turns out that under certain conditions, the PSD S(f ) of a WSS process has
a causal square root, that is, S +(f ) such that S(f ) = S +(f )S −(f ), where
S −(f ) = (S +(f ))∗ is an anticausal filter (note the similarity to the square root
for correlation matrix via Cholesky decomposition)
• In particular, if S(f ) is a rational PSD for a continuous time WSS process, i.e.,
(2πif + a1)(2πif + a2) . . . (2πif + am)
S(f ) = c ,
(2πif + b1)(2πif + b2) . . . (2πif + bn)
then it can be factorized into product of causal and anticausal square roots
Proof: Since S(f ) is real and nonnegative, S ∗(f ) = S(f ), if the denominator
has factor (2πif + b), Re(b) > 0, then it must have factor (−2πif + b∗).
Similarly, if numerator has factor (2πif + a), Re(a) > 0, then it must have
factor (−2πif + a∗)

EE 278: Random Processes in Linear Systems Page 8 – 31

 Then we can express any rational PSD as S(f ) = S +(f )S −(f ), where S +(f ) is
a causal square root that consists of the f factors and S −(f ) is an anti-causal
square root consisting of the −f factors
• Example: Consider the PSD
4π 2f 2 + 3
S(f ) = 2 2
4π f + 1
The causal square root of S(f ) is
√ √
i2πf + 3 −i2πf + 3
S +(f ) = and S −(f ) =
i2πf + 1 −i2πf + 1
The corresponding impulse responses are:
√ √
+
h (t) = δ(t) + ( 3 − 1)e u(t), h (t) = δ(t) + ( 3 − 1)etu(−t)
−t −

• For a discrete time WSS process a rational PSD is of the form

(a1 − e−i2πf )(a∗1 − ei2πf ) . . . (am − e−i2πf )(a∗m − ei2πf )
S(f ) = c
(b1 − e−i2πf )(b∗1 − ei2πf ) . . . (bm − e−i2πf )(b∗m − ei2πf )
and can be expressed also as S(f ) = S +(f )S −(f ) (the e−i2πf terms are causal
and the ei2πf terms are noncausal)

EE 278: Random Processes in Linear Systems Page 8 – 32

• Example: Consider the PSD for a discrete time process
3
S(f ) =
5 − 4 cos(2πf )
The causal square root is
√ √
3 3
S +(f ) = and S −
(f ) =
2 − e−i2πf 2 − ei2πf
Spectral factorization theorem: In general, a PSD S(f ) has a causal square root
if it satisfies the Paley-Wiener condition
! ∞
S(f )
log 2
df > −∞ for continuous time process
−∞ 1 + 4π f
! 1/2
log S(f ) df > −∞ for discrete time process
−1/2

• These condition are not always satisfied. For example they are not satisfied for
bandlimited processes
• Remark: We assume throughout that F −1[S +(f )] is real; hence
S −(f ) = S +(−f )

EE 278: Random Processes in Linear Systems Page 8 – 33

 Wiener Filter

• Again let X(t) and Y (t) be jointly WSS random processes. consider the linear
estimation of process X(t) from observations Y (α), t − a ≤ α ≤ t + b
• When a → ∞ and b = 0, the equations for the MMSE linear estimate, called
Wiener–Hopf equations, are
! ∞
RXY (τ ) = h(α)RY (τ − α) dα , 0 ≤ τ < ∞
0
! ∞
= h(α)RY (τ − α) dα , 0 ≤ τ < ∞
−∞

where h(t) is a causal impulse response

• Notation: A real-valued function h(t) can be expressed as
h(t) = [h(t)]+ + [h(t)]−,
where [h(t)]+ = h(t) for t ≥ 0 and [h(t)]+ = 0 for t < 0 is the positive (causal)
part of h(t), and [h(t)]− = h(t) − [h(t)]+ is the negative (anticausal) part

EE 278: Random Processes in Linear Systems Page 8 – 34

 Taking the Fourier transform, we have
H(f ) = [H(f )]+ + [H(f )]−,
where [H(f )]+ and [H(f )]− are the FT of the positive and negative parts of
h(t), respectively
Example: Let
4π 2f 2 + 3
S(f ) = 2 2
4π f + 1
We can write
i2πf + 2 1
S(f ) = +
i2πf + 1 −i2πf + 1
The first term is [S(f )]+ and the second is [S(f )]− . The corresponding impulse
responses are
[R(t)]+ = δ(t) + e−tu(t)
[R(t)]− = etu(−t)
Compare to the causal square root factors

EE 278: Random Processes in Linear Systems Page 8 – 35

• Now, back to the linear estimation problem. First assume that the observation
process Y (τ ) is white, i.e., RY (τ ) = δ(τ ), then the Wiener–Hopf equations
reduce to
RXY (τ ) = h(τ ) , 0 ≤ τ < ∞,
i.e., h(τ ) = [RXY (τ )]+
and the corresponding transfer function is
! ∞
H(f ) = RXY (τ )e−2πif τ dτ,
0
i.e., H(f ) = [SXY (f )]+
• For general SY (f ) with causal square root SY+(f ), we first whiten the process to
obtain Ỹ (τ ) with RỸ (τ ) = δ(τ ), then convolve with [RX Ỹ (τ )]+

1 Ỹ (t)
Y (t) +
SY (f ) [SX Ỹ (f )]+ X̂(t)

EE 278: Random Processes in Linear Systems Page 8 – 36

• Now to find RX Ỹ (τ ), let g(t) = F −1[1/SY+(f )] and consider
RX Ỹ (τ ) = E(X(t + τ )Ỹ (t))
= E(X(t + τ )Y (t) ∗ g(t))
= RXY (τ ) ∗ g(−τ )
Taking the Fourier Transform we have
SXY (f )
SX Ỹ (f ) = −
SY (f )
Hence, 3 4
SXY (f )
[SX Ỹ (f )]+ =
SY−(f ) +
The transfer function of the Wiener filter is then given by
3 4
1 SXY (f )
H(f ) = +
SY (f ) SY−(f ) +

EE 278: Random Processes in Linear Systems Page 8 – 37

• To find the MMSE, we follow similar steps to the infinite smoothing case to
obtain
( SXY (f ) (2
! ∞" (3 4 ( #
MSE = SX (f ) − (( −
( df
−∞ SY (f ) +(
• Example: Consider a continuous-time RP X(t) with
2
SX (f ) =
1 + 4π 2f 2
and the noisy observation Y (t) = X(t) + Z(t), where Z(t) is white noise
uncorrelated with X(t) with SZ (f ) = 1
To compute the Wiener filter, we first factor the PSD
4π 2f 2 + 3
SY (f ) = SX (f ) + SZ (f ) = 2 2
4π f + 1
to obtain
√
i2πf + 3
SY+(f ) = ,
i2πf + 1
√
− −i2πf + 3
SY (f ) =
−i2πf + 1

EE 278: Random Processes in Linear Systems Page 8 – 38

 The crosspower spectral density SXY (f ) = SX (f ), hence
SXY (f ) 2 −i2πf + 1
= 2 2
· √
SY (f )
−
1 + 4π f −i2πf + 3
2
= √
(i2πf + 1)(−i2πf + 3)
√ √
3−1 3−1
= + √
i2πf + 1 −i2πf + 3
The first term is causal and the second term is anticausal. Therefore,
3 4 √
SXY (f ) 3−1
=
SY−(f ) + i2πf + 1
Hence, the Wiener filter is
√
3−1
H(f ) = √
i2πf + 3
√ √
− 3t
h(t) = ( 3 − 1) e u(t)

EE 278: Random Processes in Linear Systems Page 8 – 39

 The MSE is
√
2 3−2
! ∞ √
MSE = 2 2
df = 3 − 1
−∞ 1 + 4π f

• Example: Consider a discrete-time RP X(t) with

3
SX (f ) =
5 − 4 cos(2πf )
and the noisy observation Y (t) = X(t) + Z(t), where Z(t) is white noise,
independent of X(t), with SZ (f ) = 1
Again we factor the PSD
8 − 4 cos(2πf )
SY (f ) = SX (f ) + SZ (f ) =
5 − 4 cos(2πf )
to obtain
√
√
5
+ 2 + 3 − e−i2πf
SY (f ) = 4 − 2 3 ·
2 − e−i2πf
√
√ 2 + 3 − ei2πf
5
−
SY (f ) = 4 − 2 3 ·
2 − ei2πf

EE 278: Random Processes in Linear Systems Page 8 – 40

 The crosspower spectral density is SXY (f ) = SX (f ) and
SXY (f ) 3 1
= 6 √ · √
SY (f )
−
4 − 2 3 (2 − e −i2πf )(2 + 3 − e−i2πf )
6 √
5 √
12 − 6 3 3 − 3 3/2
= + √
2−e −i2πf
2 + 3 − ei2πf
The first term is causal and the second term is anticausal. Therefore
3 4 6 √
SXY (f ) 12 − 6 3
=
SY−(f ) + 2 − e−i2πf
Hence, the Wiener filter is
√
3
H(f ) = √
2 + 3 + e−i2πf
√ √ n
h(n) = (2 3 − 3) (2 − 3) u(n)

EE 278: Random Processes in Linear Systems Page 8 – 41

 Wiener Filter Versus Kalman Filter
• Both the Kalman and Wiener filters are MMSE linear estimates of a process
from causal observations
• There are several differences, however
Kalman filter Wiener filter
Xn, Yn state space model Xn, Yn jointly WSS

not necessarily WSS

time domain filter frequency domain filter

recursive non-recursive

• Remarks:
◦ A continuous time counterpart to the Kalman filter exists and is known as the
Kalman-Bucy filter
◦ If Xn, Yn are jointly WSS and can be described via a state space model, then
the Kalman filter gives a recursive way to compute the Wiener filter

EE 278: Random Processes in Linear Systems Page 8 – 42