Appendix A.

Detection and
Estimation in Additive
Gaussian Noise
Kyungchun Lee
Dept. of EIE, SeoulTech
2013 Fall
Information and Communication Engineering
Outline
Gaussian random variables
Detection in Gaussian noise
Estimation in Gaussian noise (Brief introduction)
2
[Review] Probability and Random Variables
Random experiment
On any trial of the experiment, the outcome is unpredictable.
For a large number of trials of the experiment, the outcomes exhibit
statistical regularity. That is, a definite average pattern of outcomes
is observed if the experiment is repeated a large number of times.
E.g., tossing of a coin: possible outcomes are heads and tails

3
Introduction to Analog & Digital Communications, S. Haykin and M. Moher
[Review] Relative-Frequency Approach
Suppose that in n trials, of the experiment, event A occurs n
A

times. Then, we say that the relative frequency of the event A
is n
A
/n.
The relative frequency is a nonnegative real number less than or equal
to one.


The experiment exhibits statistical regularity if, for any sequence of n
trials, the relative frequency n
A
/n converges to a limit as n becomes
large. We define this limit


as the probability of event A.



4
1 0 s s
n
n
A
|
.
|

\
|
=

n
n
A
A
n
lim ] [ P
[Review] Sample Space
With each possible outcome of the experiment, we associate
a point called the sample point.
The totality of all sample point
Sample space S
The entire sample space S is called the sure event.
The null set | is called the null or impossible event.
A single sample point is called an elementary event.


5
[Review] Sample Space
Example) Throw of a die
Six possible outcomes: 1, 2, 3, 4, 5, 6
Sample space: {1, 2, 3, 4, 5 , 6}
The elementary event of a six shows corresponds to the sample
point {6}.
The event of an even number shows corresponds to the subset {2, 4,
6} of the sample space.

6
[Review] A formal definition of probability
A probability system consists of the triple:
1. A sample space S of elementary events (outcomes).
2. A class of events that are subsets of S.
3. A probability measure P[A] assigned to each event A in the class ,
which has the following properties (axioms of probability):
7
[ ] 1 P S =
c
c
P[ ] P[ ] P[ ] A B A B = +
0 [ ] 1 P A s s
-
-
- If AUB is the union of
two mutually exclusive
events in the class ,
then
c
[Review] Random Variables
Random Variables
A function whose domain is a sample space and whose range is a set
of real numbers is called a random variable of the experiment.
E.g., mapping a head to 1 and mapping a tail to 0

8
[Review] Random Variables
If the outcome of the experiment is s, we denote the random
variable as X(s) or just X.
We denote a particular outcome of a random experiment by
x; that is, X(s
k
)=x

The random variables may be discrete and take only a finite
number of values.
Alternatively, random variables may be continuous.
E.g., The amplitude of a noise voltage at a particular instant in
time
9
[Review] Probability Distribution Function
Probability Distribution Function
The probability that the random variable X takes any value less than or
equal to x

More often called Cumulative Distribution Function (CDF)

Properties of Probability Distribution Function
The distribution function is bounded between zero and one.
The distribution function is a monotone nondecreasing
function of x; that is,

10
) 7 . 8 ( ] [ P ) ( x X x F
X
s =
) (x F
X
) (x F
X
2 1 2 1
if ) ( ) ( x x x F x F
X X
s s
[Review] Probability density function
If x is a continuous-valued random variable and is
differentiable with respect to x, we can define the probability
density function as


Properties of Probability Density Function
Since the distribution function is monotone nondecreasing, it follows
that the density function is nonnegative for all values of x.
The distribution function may be recovered from the density function
by integration, as shown by


The above property implies that total area under the curve of the
density function is unity
11
) ( ) ( x F
x
x f
X X
c
c
=
) (x F
X
}

=
x
X X
ds s f x F ) ( ) (
[Review] Probability density function

12
[Review] Probability density function

13
[Review] Independent
The two random variables , X and Y, are statistically
independent if the outcome of X does not affect the outcome
Y.

By setting ,



Simple notation:

14
] [ P ] [ P ] , [ P B Y A X B Y A X e e = e e
) ( ) ( ) , (
,
y F x F y x F
Y X Y X
=
( , ], ( , ] A x B x = =
P[ , ] P[ ]P[ ] X Y X Y =
[Review] Multiple Random Variables
Joint Distribution Function
The probability that the random variable X is less than or equal to a
specified value x and that the random variable Y is less than or equal
to a specified value y


More often called Joint Cumulative Distribution Function (Joint CDF)

Suppose is continuous everywhere, we obtain the
joint probability density function


Nonnegatve everywhere
The total volume is unity.
15
] , [ P ) , (
,
y Y x X y x F
Y X
s s =
y x
y x F
y x f
Y X
Y X
c c
c
=
) , (
) , (
,
2
,
,
( , )
X Y
F x y
16
[Review] Mean
Mean
Statistical averages or expectations
For a discrete random variable X, the mean, , is the
weighted sum of the possible outcomes.



For a continuous random variable, the analogous
definition of the expected value is
X

] [ P
] [ E
x X x
X
X
X
= =
=

dx x xf X
X
}

= ) ( ] [ E
[Review] Covariance
Covariance
The covariance of two random variables, X and Y






For the two continuous random variables,






17
)] )( [( E ) , ( Cov
Y X
Y X Y X =
} }

= dxdy y x xyf XY
Y X
) , ( ] [ E
,
Cov( , ) E[ ] E[ ] E[ ]
E[ ]
E[ ]
Y X X Y
X Y X Y X Y
X Y
X Y XY X Y
XY
XY



= +
= +
=
[Review] Covariance
If the two random variables happen to be independent





The covariance of independent random variables is zero.
In this case, we say that the random variables are uncorrelated.
However, zero covariance does not, in general, imply independence.

18
E[ ] ( ) ( )
( ) ( )
E[ ]E[ ]
X Y
X X
X Y
XY xyf x f y dxdy
xf x dx yf y dy
X Y




=
=
= =
} }
} }
A.1 Gaussian random variables
A.1.1 Scalar real Gaussian random variables
A standard Gaussian random variable w





A (general) Gaussian random variable x


19
Gaussian random variables
Q-function
The tail of the Gaussian random variable

20
A.1.2 Gaussian random variables
Linear combinations of independent Gaussian random
variables are still Gaussian.
If are independent and


then




21
(A.6)
Real Gaussian random vectors
A standard Gaussian random vector :



where


22
Real Gaussian random vectors
Property) An orthogonal transformation
preserves the magnitude of a vector.
If w is standard Gaussian, then Ow is also standard Gaussian.
(Isotropic property)


23
Real Gaussian random vectors
Gaussian random vector
Linear transformation of a standard Gaussian random vector plus a
constant vector


Property 1) A standard Gaussian random vector is also
Gaussian (with and ).

Property 2) Any linear combination of the elements of a
Gaussian random vector is a Gaussian random variable.


This directly follows from (A.6).
24
n
e9 c where
(A.10)
Real Gaussian random vectors
Property 3) If A is invertible, then the probability density
function of x is expressed as



Proof omitted.
Covariance matrix of x


25
Real Gaussian random vectors
A few inferences form property 3
Consider two matrices A and AO used to define two Gaussian random
vectors as in (A.10). When O is orthogonal, the covariance matrices of
both these random vectors are the same, equal to AA
t
; so the two
random vectors must be distributed identically.
A Gaussian random vector is composed of independent Gaussian
random variables exactly when the covariance matrix K is diagonal.
The component random variables are uncorrelated (zero
covariance). Such a random vector is also called a white Gaussian
random vector.
26
A.1.3 Complex Gaussian random vectors
Complex random vector

where are real random vectors

Complex Gaussian random vector
is a real Gaussian random vector

The mean and covariance of the complex random vector



()*: Hermitian transpose (transpose of a matrix with each element
replaced by its complex conjugate)
27
Complex Gaussian random vectors
In wireless communication we are almost exclusively
interested in complex random vectors that have the circular
symmetry property:


For a circular symmetric random vector, the covariance matrix K fully
specifies the first- and second-order statistics

A circular symmetric Gaussian random vector with covariance
matrix K is denoted as
28
Complex Gaussian random vectors
Property 1) A complex Gaussian random variable
with independent and identically distributed (i.i.d.) zero-mean
Gaussian real and imaginary components is circular
symmetric.
The phase of w is uniform over the range and independent of
the magnitude , which has a Rayleigh distribution.
29
Complex Gaussian random vectors
A standard circular symmetric Gaussian random vector w
denoted by has the density function



If w is and A is a complex matrix, then x = Aw is also
circular symmetric Gaussian, with covariance matrix K = AA

,
i.e.,
If A is invertible, the density function of x can be expressed as

30
Complex Gaussian random vectors
Property 2) For a standard circular symmetric Gaussian
random vector w, we have


when U is a complex orthogonal matrix (called a unitary
matrix and characterized by the property )


31
(Isotropic property)
A.2 Detection in Gaussian noise
A.2.1 Scalar detection
The received signal

ue{u
A
, u
B
}


The detection problem
Making a decision on whether u
A
or u
B
was transmitted based on the
observation y

What is the optimal detector?
Provides the least probability of making an erroneous decision
Chooses the symbol that is most likely to have been transmitted given
the received signal y, i.e., u
A
is chosen if
32
33
[Review] Conditional Probability
Conditional Probability
Example) Tossing two dice.
X: The number showing on the first die
Y: The sum of the two dice

Conditional probability of Y given X
The probability mass function of Y given that X has occurred:



where is the joint probability of the two random variables.



Bayes rule




] [ P
] , [ P
] | [ P
X
Y X
X Y =
P[ , ] P[ | ]P[ ]
= P[ | ]P[ ]
X Y Y X X
X Y Y
=
] [ P
] [ P ] | [ P
] | [ P
X
Y Y X
X Y =
P[ , ] X Y
[Review] Conditional Probability
Example)
A blood test is 95% effective in detecting the viral infection when it is,
in fact, present.
However, the test also yields false positive result for 1% of the healthy
persons tested.
0.5% of the population has the infection.
If a person is tested to be positive, would you decide that he has the
infection?




34
[positive|I] 0.95 P =
[positive |no I] 0.01 P = I: Infection
[Review] Conditional Probability
The probability that a person has the infection, given that his
test result is positive:






35
| |
[I, positive]
[I|positive]
[positive]
[positive|I] [I]
[positive|I] [I] [positive|no I] no I
(0.95) (0.005)
(0.95) (0.005) (0.01) (0.995)
0.323 0.5
P
P
P
P P
P P P P
=

=
+

=
+
= <
Scalar detection
Since the two symbols u
A
, u
B
are equally likely to have been
transmitted, Bayes rule lets us simplify this to the maximum
likelihood (ML) receiver.
Choose the transmit symbol that makes the observation y most likely.

ML decision rule
We choose u
A
if



and u
B
otherwise.
36
(A.30)
Scalar detection
The ML rule in (A.30) further simplifies:

Choosing the nearest neighboring transmit symbol
37
Scalar detection
The error probability



Only depends on the distance between the two transmit symbols u
A

and u
B

38
Detection in a vector space
Consider detecting the transmit vector u equally likely to be
u
A
or u
B
(both elements of )
Received vector

where

ML decision rule
We choose u
A
if



which simplifies to

39
n
9
(A.35)
Detection in a vector space

40
Detection in a vector space
Suppose u
A
is transmitted, so

Then an error occurs when the event in (A.35) does not occur,
i.e.,

Therefore, the error probability is equal to


Since

the error probability can be rewritten as





41
Detection in a vector space: An alternative view
We can write the transmit vector as


where the information is in the scalar


We observe that the transmit symbol (a scalar x) is only in a
specific direction:

The components of the received vector y in the directions orthogonal
to v contain purely noise irrelevant for detection.

42
1 2 xe
Detection in a vector space: An alternative view
Projecting the received vector along the signal direction v
provides all the necessary information for detection:
43
sufficient statistic obtained
by a matched filter
A.2.3 Detection in a complex vector space
The transmit symbol u is equally likely to be one of two
complex vectors u
A
, u
B
.

The received signal model

where

As in the real case, we write


The signal direction
44
Detection in a complex vector space
Decision statistic

where

Since x is real (1/2), we can further extract a sufficient
statistic by looking only at the real component:

where

The error probability

45
A.3 Estimation in Gaussian noise
A.3.1 Scalar estimation
Consider a zero-mean real signal x embedded in independent
additive real Gaussian noise


We wish to come up with an estimate of .

Mean Squared Error (MSE)
46
x
x
Scalar estimation
The estimate that yields the smallest MSE is the classical
conditional mean

Orthogonality principle: the error is independent of the observation
47
Linear estimator
To simplify the analysis, one studies the restricted class of
linear estimates that minimize the MSE.
When x is a Gaussian random variable with zero mean, the
conditional mean operator is actually linear.
Assuming that x is zero mean, linear estimates are of the form

By the orthogonality principle,


The corresponding minimum MSE (MMSE)
48
Summary

49
Summary

50
Summary

51
Summary

52
Summary

53
Summary

54
Summary

55