Massachusetts Institute of Technology

Department of Electrical Engineering and Computer Science

6.437 Inference and Information
Spring 2018

Problem Set 1

Issued: Tuesday, February 6, 2018 Due: Tuesday, February 13, 2018

Problem 1.1

(a) Let x and y be independent identically distributed geometric random variables

with parameter p. That is, x and y have common probability mass function

p(n) = p(1 − p)n , n = 0, 1, . . . .

Let s = x + y .

(i) Find and sketch ps (s).

(ii) Find and sketch px|s (x|s) vs. x, with s viewed as a known integer param-
eter.
(iii) Find the conditional mean of x given s = s, which is defined as
∞
X
E [x | s = s] = x px|s (x|s).
x=−∞

(iv) The conditional mean of x given s (s viewed as a random variable) is

∞
X
µx|s = E [x|s] = x px|s (x|s).
x=−∞

Since µx|s is a function of the random variable s, it too is a random variable.

Find the probability mass function for µx|s .

(b) Let x and y be independent identically distributed random variables with com-
mon density function (
1 0≤α≤1
p(α) = .
0 otherwise
Let s = x + y .

(i) Find and sketch ps (s).

(ii) Find and sketch px|s (x|s) vs. x, with s viewed as a known real parameter.
 (iii) The conditional mean of x given s = s is
Z +∞
E [x | s = s] = x px|s (x|s) dx.
−∞

Find E [x | s = 0.5].
(iv) The conditional mean of x given s (s viewed as a random variable) is
Z +∞
µx|s = E [x|s] = x px|s (x|s) dx.
−∞

Since µx|s is a function of the random variable s, it too is a random variable.

Find the probability density function for µx|s .

Problem 1.2
A dart is thrown at random at a wall. Let (x, y ) denote the Cartesian coordinates
of the point in the wall pierced by the dart. Suppose that x and y are statistically
independent Gaussian random variables, each with mean zero and variance σ 2 , i.e.,

α2
 
1
px (α) = py (α) = √ exp − 2 .
2πσ 2 2σ

(a) Find the probability that the dart will fall within the σ-radius circle centered
at the point (0, 0).

(b) Find the probability that the dart will hit in the first quadrant (x ≥ 0, y ≥ 0).

(c) Find the conditional probability that the dart will fall within the σ-radius circle
centered at (0, 0) given that the dart hits in the first quadrant.

(d) Let r = (x 2 + y 2 )1/2 , and Θ = tan−1 (y /x) be the polar coordinates associated
with (x, y ). Find P ( 0 ≤ r ≤ r, 0 ≤ Θ ≤ Θ) and obtain pr ,Θ (r, Θ).
This observation leads to a widely used algorithm for generating Gaussian ran-
dom variables.

Problem 1.3 (practice)

(a) Consider the random variables x, y whose joint density function is given by (see
Fig. 1.3-1) (
2 if x, y ≥ 0 and x + y ≤ 1
px,y (x, y) = .
0 otherwise

2
 y

1
px,y (x,y)=2

0 1 x
Figure 1.3-1

(i) Compute the covariance matrix

 
λx λxy
Λ= .
λxy λy

(ii) Knowledge of y generally gives us information about the random variable

x (and vice versa). We want to estimate x based on knowledge of y . In
particular, we seek an estimate of x that is an affine function of y , i.e.,

x̂ = x̂(y ) = ay + b,

where a and b are constants. Select a and b so that the expected mean-
square error between x and its estimate x̂(y ), i.e.,

E (x̂ − x)2 ,
 

is minimized.

(b) Consider the random variables x, y whose joint density function is given by (see
Fig. 1.3-2) (
1 0 ≤ x, y ≤ 1
px,y (x, y) = .
0 otherwise

Repeat steps (i) and (ii) of part (a), and compare your results to those in part
(a).

3
 y px,y (x,y)=1

0 1 x
Figure 1.3-2

Problem 1.4
The traditional shell game is an example of a confidence trick, but in this problem
we consider a shell game that is legitimate and for which probabilistic analysis is the
important skill. Suppose there are three shells. Your opponent hides a pea under
one of the three shells at random and remembers which one; the other two shells
have nothing under them. A priori, you do not know which shell has the pea. If you
ultimately guess the correct shell, you win $1. Otherwise you lose $1.

(a) If the rules of the game are simply that you pick a shell and turn it over to
reveal what is under it, what is your strategy and your expected winnings?

(b) Suppose after you pick your shell, you don’t turn it over right away. Instead,
your opponent turns over one of the remaining shells that has nothing under it.
Given what your opponent turned over, what is the probability that you will
win, i.e., that your pick has the pea under it?

(c) Suppose further that after following the process of part (b) you are allowed to
switch your selection to the remaining shell your opponent chose not to turn
over. How does your probability of winning change, if at all, by switching your
selection. Should you switch your selection? Explain.

Problem 1.5
In the binary communication system shown in Figure 1.5-1, messages m = 0 and
m = 1 occur with a priori probabilities 1/4 and 3/4 respectively. Suppose that we
observe r ,
r = n + m,
where n is a random variable that is statistically independent of whether message
m = 0 or m = 1 occurs.

4
 +
m r
source receiver ^
m
m=0 or 1 received ^
m=0 or 1
message signal
n
Figure 1.5-1

(a) Say n is a continuous valued random variable, and has the pdf shown in Figure
1.5-2.

pn (n)

2/3
n
-3/4 3/4
Figure 1.5-2
Find the minimum probability-of-error detector, and compute the associated
probability of error, P (m̂ 6= m).

(b) Suppose that the receiver does not know the a priori probabilities, so it decides
to use a maximum likelihood (ML) detector. Find the ML detector and the
associated probability of error. Is the ML detector unique? Justify your answer.
If your answer is no, find a different ML detector and the associated probability
of error.

(c) Now say n is discrete, and takes on the values -1, 0, 1 with probabilities 1/8, 3/4,
1/8 respectively. Find the minimum probability of error detector, and compute
the associated probability of error, P (m̂ 6= m).

Problem 1.6
A disease has two varieties: the “0” strain and the “1” strain, with a priori proba-
bilities p0 and p1 respectively.

(a) Initially, a rather noisy test was developed to determine which strain is present
for patients who are known to have one of the two varieties. The output of
the test is the sample value y1 of a random variable y1 . Given that the strain
is “0” (H = H0 ), y1 = 5 + z1 , and given that the strain is “1” (H = H1 ),
y1 = 1 + z1 . The measurement noise z1 is independent of H and is Gaussian
with z1 ∼ N (0, σ 2 ). Find the MAP decision rule, i.e., determine the set of
observations y1 for which the decision is Ĥ = H1 . Let

E = (Ĥ = H0 ∩ H = H1 ) ∪ (Ĥ = H1 ∩ H = H0 )

5
 denote the event that there is a decision error. Determine P (E|H = H0 ) and
P (E|H = H1 ) in terms of Q (·).

(b) A budding medical researcher determines that the test is making too many
errors. A new measurement procedure is devised with two observation random
variables y1 and y2 , y1 being the same as in part (a). Under hypothesis H0 ,
y2 = 5 + z1 + z2 , and under hypothesis H1 , y2 = 1 + z1 + z2 . Assume that z2 is
independent of both z1 and H and that z2 ∼ N (0, σ 2 ). Find the MAP decision
rule for Ĥ in terms of the joint observation (y1 , y2 ), and find P (E|H = H0 ) and
P (E|H = H1 ). (Hint: Find py2 |y1 ,H (y2 |y1 , H0 ) and py2 |y1 ,H (y2 |y1 , H1 ).)

(c) Explain in layman’s terms why the medical researcher should learn more about
probability.

(d) Now, suppose that z2 in part (b) is uniformly distributed between 0 and 1 rather
than being Gaussian. It is still true that z2 is independent of both z1 and H.
Find the MAP decision rule for Ĥ in terms of the joint observation (y1 , y2 ) and
find P (E|H = H0 ) and P (E|H = H1 ).

(e) Finally, suppose that z1 is also uniformly distributed between 0 and 1. Again,
find the MAP decision rule and error probabilities.