LECTURE NOTES

on

PROBABILITY and STATISTICS

Eusebius Doedel
 TABLE OF CONTENTS

SAMPLE SPACES 1
Events 5
The Algebra of Events 6
Axioms of Probability 9
Further Properties 10
Counting Outcomes 13
Permutations 14
Combinations 21

CONDITIONAL PROBABILITY 45
Independent Events 63

DISCRETE RANDOM VARIABLES 71

Joint distributions 82
Independent random variables 91
Conditional distributions 97
Expectation 101
Variance and Standard Deviation 108
Covariance 110
SPECIAL DISCRETE RANDOM VARIABLES 118
The Bernoulli Random Variable 118
The Binomial Random Variable 120
The Poisson Random Variable 130

CONTINUOUS RANDOM VARIABLES 142

Joint distributions 150
Marginal density functions 153
Independent continuous random variables 158
Conditional distributions 161
Expectation 163
Variance 169
Covariance 175
Markovs inequality 181
Chebyshevs inequality 184

SPECIAL CONTINUOUS RANDOM VARIABLES 187

The Uniform Random Variable 187
The Exponential Random Variable 191
The Standard Normal Random Variable 196
The General Normal Random Variable 201
The Chi-Square Random Variable 206
THE CENTRAL LIMIT THEOREM 211

SAMPLE STATISTICS 246

The Sample Mean 252
The Sample Variance 257
Estimating the Variance of a Normal Distribution 266
Samples from Finite Populations 274
The Sample Correlation Coefficient 282
Maximum Likelihood Estimators 288
Hypothesis Testing 305

LEAST SQUARES APPROXIMATION 335

Linear Least Squares 335
General Least Squares 343

RANDOM NUMBER GENERATION 362

The Logistic Equation 363
Generating Random Numbers 378
Generating Uniformly Distributed Random Numbers 379
Generating Random Numbers using the Inverse Method 392

SUMMARY TABLES AND FORMULAS 403

 SAMPLE SPACES

DEFINITION :
The sample space is the set of all possible outcomes of an experiment.

EXAMPLE : When we flip a coin then sample space is

S = {H , T },
where
H denotes that the coin lands Heads up
and
T denotes that the coin lands Tails up.

For a fair coin we expect H and T to have the same chance of

occurring, i.e., if we flip the coin many times then about 50 % of the
outcomes will be H.
We say that the probability of H to occur is 0.5 (or 50 %) .
The probability of T to occur is then also 0.5.

1
EXAMPLE :

When we roll a fair die then the sample space is

S = {1, 2, 3, 4, 5, 6}.

1
The probability the die lands with k up is 6
, (k = 1, 2, , 6).

When we roll it 1200 times we expect a 5 up about 200 times.

The probability the die lands with an even number up is

1 1 1 1
+ + = .
6 6 6 2

2
EXAMPLE :
When we toss a coin 3 times and record the results in the sequence
that they occur, then the sample space is

S = { HHH , HHT , HT H , HT T , T HH , T HT , T T H , T T T } .

Elements of S are vectors , sequences , or ordered outcomes .

We may expect each of the 8 outcomes to be equally likely.

1
Thus the probability of the sequence HT T is 8
.

The probability of a sequence to contain precisely two Heads is

1 1 1 3
+ + = .
8 8 8 8

3
EXAMPLE : When we toss a coin 3 times and record the results
without paying attention to the order in which they occur, e.g., if we
only record the number of Heads, then the sample space is
n o
S = {H, H, H} , {H, H, T } , {H, T, T } , {T, T, T } .
The outcomes in S are now sets ; i.e., order is not important.
Recall that the ordered outcomes are
{ HHH , HHT , HT H , HT T , T HH , T HT , T T H , T T T } .

Note that
{H, H, H} corresponds to one of the ordered outcomes,
{H, H, T } ,, three ,,
{H, T, T } ,, three ,,
{T, T, T } ,, one ,,

Thus {H, H, H} and {T, T, T } each occur with probability 18 ,

while {H, H, T } and {H, T, T } each occur with probability 38 .

4
Events

In Probability Theory subsets of the sample space are called events.

EXAMPLE : The set of basic outcomes of rolling a die once is

S = {1, 2, 3, 4, 5, 6},

so the subset E = { 2 , 4 , 6 } is an example of an event.

If a die is rolled once and it lands with a 2 or a 4 or a 6 up then we

say that the event E has occurred.

We have already seen that the probability that E occurs is

1 1 1 1
P (E) = + + = .
6 6 6 2

5
The Algebra of Events

Since events are sets, namely, subsets of the sample space S, we can
do the usual set operations :

If E and F are events then we can form

Ec the complement of E
EF the union of E and F
EF the intersection of E and F

We write E F if E is a subset of F .

REMARK : In Probability Theory we use

Ec instead of E ,

EF instead of E F ,

EF instead of E F .

6
If the sample space S is finite then we typically allow any subset of
S to be an event.

EXAMPLE : If we randomly draw one character from a box con-

taining the characters a, b, and c, then the sample space is
S = {a , b , c} ,
and there are 8 possible events, namely, those in the set of events
n o
E = { } , {a} , {b} , {c} , {a, b} , {a, c} , {b, c} , {a, b, c} .

If the outcomes a, b, and c, are equally likely to occur, then

1 1 1
P ({ }) = 0 , P ({a}) = , P ({b}) = , P ({c}) = ,
3 3 3
2 2 2
P ({a, b}) = , P ({a, c}) = , P ({b, c}) = , P ({a, b, c}) = 1 .
3 3 3
For example, P ({a, b}) is the probability the character is an a or a b.

7
We always assume that the set E of allowable events includes the
complements, unions, and intersections of its events.

EXAMPLE : If the sample space is

S = {a , b , c , d} ,

and we start with the events

n o
E0 = {a} , {c, d} ,

then this set of events needs to be extended to (at least)

n o
E = { } , {a} , {c, d} , {b, c, d} , {a, b} , {a, c, d} , {b} , {a, b, c, d} .

EXERCISE : Verify E includes complements, unions, intersections.

8
Axioms of Probability

A probability function P assigns a real number (the probability of E)

to every event E in a sample space S.

P () must satisfy the following basic properties :

0 P (E) 1 ,

P (S) = 1 ,

For any disjoint events Ei , i = 1, 2, , n, we have

P (E1 E2 En ) = P (E1 ) + P (E2 ) + P (En ) .

9
Further Properties

PROPERTY 1 :

P (E E c ) = P (E) + P (E c ) = 1 . ( Why ? )

Thus
P (E c ) = 1 P (E) .

EXAMPLE :
What is the probability of at least one H in four tosses of a coin?

SOLUTION : The sample space S will have 16 outcomes. (Which?)

1 15
P (at least one H) = 1 P (no H) = 1 = .
16 16

10
PROPERTY 2 :
P (E F ) = P (E) + P (F ) P (EF ) .

PROOF (using the third axiom) :

P (E F ) = P (EF ) + P (EF c ) + P (E c F )

= [P (EF ) + P (EF c )] + [P (EF ) + P (E c F )] P (EF )

= P(E) + P(F) - P(EF) . ( Why ? )

NOTE :
Draw a Venn diagram with E and F to see this !

The formula is similar to the one for the number of elements :

n(E F ) = n(E) + n(F ) n(EF ) .

11
So far our sample spaces S have been finite.
S can also be countably infinite, e.g., the set Z of all integers.
S can also be uncountable, e.g., the set R of all real numbers.

EXAMPLE : Record the low temperature in Montreal on January

8 in each of a large number of years.
We can take S to be the set of all real numbers, i.e., S = R.
(Are there are other choices of S ?)
What probability would you expect for the following events to have?

(a) P ({}) (b) P ({x : < x < })

(How does this differ from finite sample spaces?)

We will encounter such infinite sample spaces many times

12
Counting Outcomes

We have seen examples where the outcomes in a finite sample space

S are equally likely , i.e., they have the same probability .

Such sample spaces occur quite often.

Computing probabilities then requires counting all outcomes and

counting certain types of outcomes .

The counting has to be done carefully!

We will discuss a number of representative examples in detail.

Concepts that arise include permutations and combinations.

13
Permutations

Here we count of the number of words that can be formed

from a collection of items (e.g., letters).

(Also called sequences , vectors , ordered sets .)

The order of the items in the word is important;

e.g., the word acb is different from the word bac .

The word length is the number of characters in the word.

NOTE :
For sets the order is not important. For example, the set {a,c,b} is
the same as the set {b,a,c} .

14
EXAMPLE : Suppose that four-letter words of lower case alpha-
betic characters are generated randomly with equally likely outcomes.
(Assume that letters may appear repeatedly.)

(a) How many four-letter words are there in the sample space S ?
SOLUTION : 264 = 456, 976 .

(b) How many four-letter words are there are there in S that start
with the letter s ?
SOLUTION : 263 .

(c) What is the probability of generating a four-letter word that

starts with an s ?
SOLUTION :
263 1
4
= = 0.038 .
26 26
Could this have been computed more easily?

15
EXAMPLE : How many re-orderings (permutations) are there of
the string abc ? (Here letters may appear only once.)
SOLUTION : Six, namely, abc , acb , bac , bca , cab , cba .

If these permutations are generated randomly with equal probability

then what is the probability the word starts with the letter a ?
SOLUTION :
2 1
= .
6 3

EXAMPLE : In general, if the word length is n and all characters

are distinct then there are n! permutations of the word. ( Why ? )

If these permutations are generated randomly with equal probability

then what is the probability the word starts with a particular letter ?

SOLUTION :
(n 1)! 1
= . ( Why ? )
n! n

16
EXAMPLE : How many

words of length k

can be formed from

a set of n (distinct) characters ,

(where k n ) ,

when letters can be used at most once ?

SOLUTION :
n (n 1) (n 2) (n (k 1))
= n (n 1) (n 2) (n k + 1)
n!
= ( Why ? )
(n k)!

17
EXAMPLE : Three-letter words are generated randomly from the
five characters a , b , c , d , e , where letters can be used at most
once.

(a) How many three-letter words are there in the sample space S ?
SOLUTION : 5 4 3 = 60 .

(b) How many words containing a , b are there in S ?

SOLUTION : First place the characters
a,b
i.e., select the two indices of the locations to place them.
This can be done in
3 2 = 6 ways . ( Why ? )
There remains one position to be filled with a c , d or an e .
Therefore the number of words is 3 6 = 18 .

18
(c) Suppose the 60 solutions in the sample space are equally likely .

What is the probability of generating a three-letter word that

contains the letters a and b ?

SOLUTION :
18
= 0.3 .
60

19
EXERCISE :

Suppose the sample space S consists of all five-letter words

having distinct alphabetic characters .

How many words are there in S ?

How many special words are in S for which only the second
and the fourth character are vowels, i.e., one of {a, e, i, o, u, y} ?

Assuming the outcomes in S to be equally likely, what is the

probability of drawing such a special word?

20
Combinations

Let S be a set containing n (distinct) elements.

Then
a combination of k elements from S ,
is
any selection of k elements from S ,

where order is not important .

(Thus the selection is a set .)

NOTE : By definition a set always has distinct elements .

21
EXAMPLE :

There are three combinations of 2 elements chosen from the set

S = {a , b , c} ,

namely, the subsets

{a, b} , {a, c} , {b, c} ,

whereas there are six words of 2 elements from S ,

namely,

ab , ba , ac , ca , bc , cb .

22
In general, given
a set S of n elements ,

the number of possible subsets of k elements from S equals

 
n n!
.
k k! (n k)!

 
n
REMARK : The notation is referred to as
k
n choose k .

 
n n! n!
NOTE : = = = 1,
n n! (n n)! n! 0!

since 0! 1 (by convenient definition !) .

23
PROOF :

First recall that there are

n!
n (n 1) (n 2) (n k + 1) =
(n k)!

possible sequences of k distinct elements from S .

However, every sequence of length k has k! permutations of itself,

and each of these defines the same subset of S.

Thus the total number of subsets is

 
n! n
.
k! (n k)! k

24
EXAMPLE :
In the previous example, with 2 elements chosen from the set

{a , b , c} ,

we have n = 3 and k = 2 , so that there are

3!
= 6 words ,
(3 2)!
namely
ab , ba , ac , ca , bc , cb ,
while there are
 
3 3! 6
= = 3 subsets ,
2 2! (3 2)! 2
namely
{a, b} , {a, c} , {b, c} .

25
EXAMPLE : If we choose 3 elements from {a , b , c , d} , then
n = 4 and k = 3 ,
so there are
4!
= 24 words, namely :
(4 3)!

abc , abd , acd , bcd ,

acb , adb , adc , bdc ,
bac , bad , cad , cbd ,
bca , bda , cda , cdb ,
cab , dab , dac , dbc ,
cba , dba , dca , dcb ,
while there are
 
4 4! 24
= = 4 subsets ,
3 3! (4 3)! 6
namely,
{a, b, c} , {a, b, d} , {a, c, d} , {b, c, d} .

26
EXAMPLE :

(a) How many ways are there to choose a committee of 4 persons

from a group of 10 persons, if order is not important?
SOLUTION :
 
10 10!
= = 210 .
4 4! (10 4)!

(b) If each of these 210 outcomes is equally likely then what is the
probability that a particular person is on the committee?
SOLUTION :
   
9 10 84 4
/ = = . ( Why ? )
3 4 210 10

Is this result surprising?

27
(c) What is the probability that a particular person is not on the
committee?
SOLUTION :
   
9 10 126 6
/ = = . ( Why ? )
4 4 210 10

Is this result surprising?

(d) How many ways are there to choose a committee of 4 persons

from a group of 10 persons, if one is to be the chairperson?
SOLUTION :
     
10 9 9 9!
= 10 = 10 = 840 .
1 3 3 3! (9 3)!

QUESTION : Why is this four times the number in (a) ?

28
EXAMPLE : Two balls are selected at random from a bag with
four white balls and three black balls, where order is not important.

What would be an appropriate sample space S ?

SOLUTION : Denote the set of balls by

B = {w1 , w2 , w3 , w4 , b1 , b2 , b3 } ,
where same color balls are made distinct by numbering them.

Then a good choice of the sample space is

S = the set of all subsets of two balls from B ,
because the wording selected at random suggests that each such
subset has the same chance to be selected.

The number of outcomes in S (which are sets of two balls) is then

 
7
= 21 .
2

29
EXAMPLE : ( continued )
(Two balls are selected at random from a bag with four white balls
and three black balls.)
What is the probability that both balls are white?
SOLUTION :    
4 7 6 2
/ = = .
2 2 21 7

What is the probability that both balls are black?

SOLUTION :    
3 7 3 1
/ = = .
2 2 21 7

What is the probability that one is white and one is black?

SOLUTION :      
4 3 7 43 4
/ = = .
1 1 2 21 7
(Could this have been computed differently?)

30
EXAMPLE : ( continued )
In detail, the sample space S is
n
{w1 , w2 }, {w1 , w3 }, {w1 , w4 }, | {w1 , b1 }, {w1 , b2 }, {w1 , b3 },
{w2 , w3 }, {w2 , w4 }, | {w2 , b1 }, {w2 , b2 }, {w2 , b3 },
{w3 , w4 }, | {w3 , b1 }, {w3 , b2 }, {w3 , b3 },
| {w4 , b1 }, {w4 , b2 }, {w4 , b3 },

{b1 , b2 }, {b1 , b3 }, o
S has 21 outcomes, each of which is a set . {b2 , b3 }
1
We assumed each outcome of S has probability 21
.
The event both balls are white contains 6 outcomes.
The event both balls are black contains 3 outcomes.
The event one is white and one is black contains 12 outcomes.
What would be different had we worked with sequences ?

31
EXERCISE :

Three balls are selected at random from a bag containing

2 red , 3 green , 4 blue balls .

What would be an appropriate sample space S ?

What is the the number of outcomes in S ?

What is the probability that all three balls are red ?

What is the probability that all three balls are green ?

What is the probability that all three balls are blue ?

What is the probability of one red, one green, and one blue ball ?

32
EXAMPLE : A bag contains 4 black balls and 4 white balls.
Suppose one draws two balls at the time, until the bag is empty.
What is the probability that each drawn pair is of the same color?

SOLUTION : An example of an outcome in the sample space S is

n o
{w1 , w3 } , {w2 , b3 } , {w4 , b1 } , {b2 , b4 } .

The number of such doubly unordered outcomes in S is

    
1 8 6 4 2 1 8! 6! 4! 2! 1 8!
= = = 105 (Why?)
4! 2 2 2 2 4! 2! 6! 2! 4! 2! 2! 2! 0! 4! (2!)4

The number of such outcomes with pairwise the same color is

     
1 4 2 1 4 2
= 3 3 = 9. ( Why ? )
2! 2 2 2! 2 2

Thus the probability each pair is of the same color is 9/105 = 3/35 .

33
EXAMPLE : ( continued )
The 9 outcomes of pairwise the same color constitute the event
(
n o
{w1 , w2 } , {w3 , w4 } , {b1 , b2 } , {b3 , b4 } ,
n o
{w1 , w3 } , {w2 , w4 } , {b1 , b2 } , {b3 , b4 } ,
n o
{w1 , w4 } , {w2 , w3 } , {b1 , b2 } , {b3 , b4 } ,
n o
{w1 , w2 } , {w3 , w4 } , {b1 , b3 } , {b2 , b4 } ,
n o
{w1 , w3 } , {w2 , w4 } , {b1 , b3 } , {b2 , b4 } ,
n o
{w1 , w4 } , {w2 , w3 } , {b1 , b3 } , {b2 , b4 } ,
n o
{w1 , w2 } , {w3 , w4 } , {b1 , b4 } , {b2 , b3 } ,
n o
{w1 , w3 } , {w2 , w4 } , {b1 , b4 } , {b2 , b3 } ,
)
n o
{w1 , w4 } , {w2 , w3 } , {b1 , b4 } , {b2 , b3 } .

34
EXERCISE :

How many ways are there to choose a committee of 4 persons

from a group of 6 persons, if order is not important?
Write down the list of all these possible committees of 4 persons.
If each of these outcomes is equally likely then what is the
probability that two particular persons are on the committee?

EXERCISE :

Two balls are selected at random from a bag with three white balls
and two black balls.

Show all elements of a suitable sample space.

What is the probability that both balls are white?

35
EXERCISE :

We are interested in birthdays in a class of 60 students.

What is a good sample space S for this purpose?

How many outcomes are there in S ?

What is the probability of no common birthdays in this class?

What is the probability of common birthdays in this class?

36
EXAMPLE :
How many nonnegative integer solutions are there to

x1 + x2 + x3 = 17 ?

SOLUTION :
Consider seventeen 1s separated by bars to indicate the possible
values of x1 , x2 , and x3 , e.g.,

111|111111111|11111 .

The total number of positions in the display is 17 + 2 = 19 .

The total number of nonnegative solutions is now seen to be

 
19 19! 19 18
= = = 171 .
2 (19 2)! 2! 2

37
EXAMPLE :
How many nonnegative integer solutions are there to the inequality

x1 + x2 + x3 17 ?

SOLUTION :
Introduce an auxiliary variable (or slack variable )

x4 17 (x1 + x2 + x3 ) .

Then
x1 + x2 + x3 + x4 = 17 .

Use seventeen 1s separated by 3 bars to indicate the possible values

of x1 , x2 , x3 , and x4 , e.g.,

111|11111111|1111|11 .

38
 111|11111111|1111|11 .

The total number of positions is

17 + 3 = 20 .

The total number of nonnegative solutions is therefore

 
20 20! 20 19 18
= = = 1140 .
3 (20 3)! 3! 32

39
EXAMPLE :
How many positive integer solutions are there to the equation
x1 + x2 + x3 = 17 ?

SOLUTION : Let
x1 = x1 + 1 , x2 = x2 + 1 , x3 = x3 + 1 .

Then the problem becomes :

How many nonnegative integer solutions are there to the equation
x1 + x2 + x3 = 14 ?
111|111111111|11
The solution is
 
16 16! 16 15
= = = 120 .
2 (16 2)! 2! 2

40
EXAMPLE :
What is the probability the sum is 9 in three rolls of a die ?

SOLUTION : The number of such sequences of three rolls with

sum 9 is the number of integer solutions of
x1 + x2 + x3 = 9 ,
with
1 x1 6 , 1 x2 6 , 1 x3 6 .
Let
x1 = x1 + 1 , x2 = x2 + 1 , x3 = x3 + 1 .

Then the problem becomes :

How many nonnegative integer solutions are there to the equation
x1 + x2 + x3 = 6 ,
with
0 x1 , x2 , x3 5 .

41
EXAMPLE : ( continued )
Now the equation
x1 + x2 + x3 = 6 , ( 0 x1 , x2 , x3 5 ) ,

1|111|11
has  
8
= 28 solutions ,
2

from which we must subtract the 3 impossible solutions

(x1 , x2 , x3 ) = (6, 0, 0) , (0, 6, 0) , (0, 0, 6) .
111111|| , |111111| , ||111111

Thus the probability that the sum of 3 rolls equals 9 is

28 3 25
= = 0.116 .
63 216

42
EXAMPLE : ( continued )

The 25 outcomes of the event the sum of the rolls is 9 are

{ 126 , 135 , 144 , 153 , 162 ,

216 , 225 , 234 , 243 , 252 , 261 ,

315 324 , 333 , 342 , 351 ,

414 , 423 , 432 , 441 ,

513 , 522 , 531 ,

612 , 621 }.

The lexicographic ordering of the outcomes (which are sequences)

in this event is used for systematic counting.

43
EXERCISE :

How many integer solutions are there to the inequality

x1 + x2 + x3 17 ,

if we require that

x1 1 , x2 2 , x3 3 ?

EXERCISE :

What is the probability that the sum is less than or equal to 9

in three rolls of a die ?

44
 CONDITIONAL PROBABILITY

Giving more information can change the probability of an event.

EXAMPLE :
If a coin is tossed two times then what is the probability of two
Heads?
ANSWER : 1
.
4

EXAMPLE :
If a coin is tossed two times then what is the probability of two Heads,
given that the first toss gave Heads ?
ANSWER : 1
.
2

45
NOTE :

Several examples will be about playing cards .

A standard deck of playing cards consists of 52 cards :

Four suits :
Hearts , Diamonds (red ) , and Spades , Clubs (black) .

Each suit has 13 cards, whose denomination is

2 , 3 , , 10 , Jack , Queen , King , Ace .

The Jack , Queen , and King are called face cards .

46
EXERCISE :

Suppose we draw a card from a shuffled set of 52 playing cards.

What is the probability of drawing a Queen ?

What is the probability of drawing a Queen, given that the card

drawn is of suit Hearts ?

What is the probability of drawing a Queen, given that the card

drawn is a Face card ?

What do the answers tell us?

(Well soon learn the events Queen and Hearts are independent .)

47
The two preceding questions are examples of conditional probability .

Conditional probability is an important and useful concept.

If E and F are events, i.e., subsets of a sample space S , then

P (E|F ) is the conditional probability of E , given F ,

defined as
P (EF )
P (E|F ) .
P (F )
or, equivalently

P (EF ) = P (E|F ) P (F ) ,

(assuming that P (F ) is not zero).

48
 P (EF )
P (E|F )
P (F )

E
S S E
F
F

Suppose that the 6 outcomes in S are equally likely.

What is P (E|F ) in each of these two cases ?

49
 P (EF )
P (E|F )
P (F )

S E S F

Suppose that the 6 outcomes in S are equally likely.

What is P (E|F ) in each of these two cases ?

50
EXAMPLE : Suppose a coin is tossed two times.

The sample space is

S = {HH , HT , T H , T T } .

Let E be the event two Heads , i.e.,

E = {HH} .

Let F be the event the first toss gives Heads , i.e.,

F = {HH , HT } .
Then
EF = {HH} = E ( since E F ) .

We have
1
P (EF ) P (E) 4 1
P (E|F ) = = = 2 = .
P (F ) P (F ) 4
2

51
EXAMPLE :
Suppose we draw a card from a shuffled set of 52 playing cards.
What is the probability of drawing a Queen, given that the card
drawn is of suit Hearts ?
ANSWER :
1
P (QH) 52 1
P (Q|H) = = 13 = .
P (H) 52
13

What is the probability of drawing a Queen, given that the card

drawn is a Face card ?
ANSWER :
4
P (QF ) P (Q) 52 1
P (Q|F ) = = = 12 = .
P (F ) P (F ) 52
3

(Here Q F , so that QF = Q .)

52
The probability of an event E is sometimes computed more easily

if we condition E on another event F ,

namely, from

P (E) = P ( E(F F c ) ) ( Why ? )

= P ( EF EF c ) = P (EF ) + P (EF c ) ( Why ? )

and

P (EF ) = P (E|F ) P (F ) , P (EF c ) = P (E|F c ) P (F c ) ,

we obtain this basic formula

P (E) = P (E|F ) P (F ) + P (E|F c ) P (F c ) .

53
EXAMPLE :

An insurance company has these data :

The probability of an insurance claim in a period of one year is

4 percent for persons under age 30

2 percent for persons over age 30

and it is known that

30 percent of the targeted population is under age 30.

What is the probability of an insurance claim in a period of one year

for a randomly chosen person from the targeted population?

54
SOLUTION :

Let the sample space S be all persons under consideration.

Let C be the event (subset of S) of persons filing a claim.

Let U be the event (subset of S) of persons under age 30.

Then U c is the event (subset of S) of persons over age 30.

Thus
P (C) = P (C|U ) P (U ) + P (C|U c ) P (U c )
4 3 2 7
= +
100 10 100 10
26
= = 2.6% .
1000

55
EXAMPLE :

Two balls are drawn from a bag with 2 white and 3 black balls.

There are 20 outcomes (sequences) in S . ( Why ? )

What is the probability that the second ball is white ?

SOLUTION :

Let F be the event that the first ball is white.

Let S be the event that the second second ball is white.

Then
c c 1 2 2 3 2
P (S) = P (S|F ) P (F ) + P (S|F ) P (F ) = + = .
4 5 4 5 5

QUESTION : Is it surprising that P (S) = P (F ) ?

56
EXAMPLE : ( continued )
Is it surprising that P (S) = P (F ) ?

ANSWER : Not really, if one considers the sample space S :

n
w 1 w2 , w1 b 1 , w1 b 2 , w1 b 3 ,

w2 w1 , w2 b 1 , w2 b 2 , w2 b 3 ,

b 1 w1 , b 1 w2 , b1 b2 , b1 b3 ,

b 2 w1 , b 2 w2 , b2 b1 , b2 b3 ,
o
b 3 w1 , b 3 w2 , b3 b1 , b3 b2 ,

where outcomes (sequences) are assumed equally likely.

57
EXAMPLE :

Suppose we draw two cards from a shuffled set of 52 playing cards.

What is the probability that the second card is a Queen ?

ANSWER :

P (2nd card Q) =

P (2nd card Q|1st card Q) P (1st card Q)

+ P (2nd card Q|1st card not Q) P (1st card not Q)

3 4 4 48 204 4 1
= + = = = .
51 52 51 52 51 52 52 13

QUESTION : Is it surprising that P (2nd card Q) = P (1st card Q) ?

58
A useful formula that inverts conditioning is derived as follows :

Since we have both

P (EF ) = P (E|F ) P (F ) ,

and
P (EF ) = P (F |E) P (E) .

If P (E) 6= 0 then it follows that

P (EF ) P (E|F ) P (F )
P (F |E) = = ,
P (E) P (E)
and, using the earlier useful formula, we get
P (E|F ) P (F )
P (F |E) = ,
P (E|F ) P (F ) + P (E|F c ) P (F c )

which is known as Bayes formula .

59
EXAMPLE : Suppose 1 in 1000 persons has a certain disease.
A test detects the disease in 99 % of diseased persons.
The test also detects the disease in 5 % of healthly persons.
With what probability does a positive test diagnose the disease?

SOLUTION : Let
D diseased , H healthy , + positive.
We are given that
P (D) = 0.001 , P (+|D) = 0.99 , P (+|H) = 0.05 .

By Bayes formula
P (+|D) P (D)
P (D|+) =
P (+|D) P (D) + P (+|H) P (H)

0.99 0.001
= = 0.0194 (!)
0.99 0.001 + 0.05 0.999

60
EXERCISE :
Suppose 1 in 100 products has a certain defect.

A test detects the defect in 95 % of defective products.

The test also detects the defect in 10 % of non-defective products.

With what probability does a positive test diagnose a defect?

EXERCISE :
Suppose 1 in 2000 persons has a certain disease.

A test detects the disease in 90 % of diseased persons.

The test also detects the disease in 5 % of healthly persons.

With what probability does a positive test diagnose the disease?

61
More generally, if the sample space S is the union of disjoint events
S = F1 F2 Fn ,
then for any event E
P (E|Fi ) P (Fi )
P (Fi |E) = .
P (E|F1 ) P (F1 ) + P (E|F2 ) P (F2 ) + + P (E|Fn ) P (Fn )

EXERCISE :
Machines M1 , M2 , M3 produce these proportions of a article

Production : M1 : 10 % , M2 : 30 % , M3 : 60 % .

The probability the machines produce defective articles is

Defects : M1 : 4 % , M2 : 3 % , M3 : 2 % .

What is the probability a random article was made by machine M1 ,

given that it is defective?

62
Independent Events

Two events E and F are independent if

P (EF ) = P (E) P (F ) .

In this case
P (EF ) P (E) P (F )
P (E|F ) = = = P (E) ,
P (F ) P (F )

(assuming P (F ) is not zero).

Thus

knowing F occurred doesnt change the probability of E .

63
EXAMPLE : Draw one card from a deck of 52 playing cards.
Counting outcomes we find
12 3
P (Face Card) = 52
= 13
,

13 1
P (Hearts) = 52
= 4
,

3
P (Face Card and Hearts) = 52
,

3
P (Face Card|Hearts) = 13
.
We see that
3
P (Face Card and Hearts) = P (Face Card) P (Hearts) (= ).
52
Thus the events Face Card and Hearts are independent.

Therefore we also have

3
P (Face Card|Hearts) = P (Face Card) (= ).
13

64
EXERCISE :

Which of the following pairs of events are independent?

(1) drawing Hearts and drawing Black ,

(2) drawing Black and drawing Ace ,

(3) the event {2, 3, , 9} and drawing Red .

65
EXERCISE : Two numbers are drawn at random from the set
{1, 2, 3, 4}.

If order is not important then what is the sample space S ?

Define the following functions on S :

X( {i, j} ) = i + j , Y ( {i, j} ) = |i j| .

Which of the following pairs of events are independent?

(1) X = 5 and Y = 2 ,

(2) X = 5 and Y = 1 .

REMARK :
X and Y are examples of random variables . (More soon!)

66
EXAMPLE : If E and F are independent then so are E and F c .

PROOF : E = E(F F c ) = EF EF c , where

EF and EF c are disjoint .
Thus
P (E) = P (EF ) + P (EF c ) ,
from which

P (EF c ) = P (E) P (EF )

= P (E) P (E) P (F ) (since E and F independent)

= P (E) ( 1 P (F ) )

= P (E) P (F c ) .

EXERCISE :
Prove that if E and F are independent then so are E c and F c .

67
NOTE : Independence and disjointness are different things !

E
S S E
F
F

Independent, but not disjoint. Disjoint, but not independent.

(The six outcomes in S are assumed to have equal probability.)

If E and F are independent then P (EF ) = P (E) P (F ) .

If E and F are disjoint then P (EF ) = P ( ) = 0 .

If E and F are independent and disjoint then one has zero probability !

68
Three events E , F , and G are independent if

P (EF G) = P (E) P (F ) P (G) .

and
P (EF ) = P (E) P (F ) .
P (EG) = P (E) P (G) .
P (F G) = P (F ) P (G) .

EXERCISE : Are the three events of drawing

(1) a red card ,

(2) a face card ,

(3) a Heart or Spade ,

independent ?

69
EXERCISE :

A machine M consists of three independent parts, M1 , M2 , and M3 .

Suppose that
9
M1 functions properly with probability 10
,

9
M2 functions properly with probability 10
,

8
M3 functions properly with probability 10
,
and that

the machine M functions if and only if its three parts function.

What is the probability for the machine M to function ?

What is the probability for the machine M to malfunction ?

70
 DISCRETE RANDOM VARIABLES

DEFINITION : A discrete random variable is a function X(s) from

a finite or countably infinite sample space S to the real numbers :
X() : S R.

EXAMPLE : Toss a coin 3 times in sequence. The sample space

is
S = {HHH , HHT , HT H , HT T , T HH , T HT , T T H , T T T } ,
and examples of random variables are
X(s) = the number of Heads in the sequence ; e.g., X(HT H) = 2 ,

Y (s) = The index of the first H ; e.g., Y (T T H) = 3 ,

0 if the sequence has no H , i.e., Y (T T T ) = 0 .

NOTE : In this example X(s) and Y (s) are actually integers .

71
Value-ranges of a random variable correspond to events in S .

EXAMPLE : For the sample space

S = {HHH , HHT , HT H , HT T , T HH , T HT , T T H , T T T } ,
with
X(s) = the number of Heads ,
the value
X(s) = 2 , corresponds to the event {HHT , HT H , T HH} ,
and the values
1 < X(s) 3 , correspond to {HHH , HHT , HT H , T HH} .

NOTATION : If it is clear what S is then we often just write

X instead of X(s) .

72
 Value-ranges of a random variable correspond to events in S ,
and
events in S have a probability .
Thus
Value-ranges of a random variable have a probability .

EXAMPLE : For the sample space

S = {HHH , HHT , HT H , HT T , T HH , T HT , T T H , T T T } ,
with X(s) = the number of Heads ,
we have
6
P (0 < X 2) = .
8

QUESTION : What are the values of

P (X 1) , P (X 0) , P (X 1) , P (X 2) , P (X 3) , P (X 4) ?

73
NOTATION : We will also write pX (x) to denote P (X = x) .

EXAMPLE : For the sample space

S = {HHH , HHT , HT H , HT T , T HH , T HT , T T H , T T T } ,
with
X(s) = the number of Heads ,
we have
1
pX (0) P ( {T T T } ) = 8

3
pX (1) P ( {HT T , T HT , T T H} ) = 8

3
pX (2) P ( {HHT , HT H , T HH} ) = 8

1
pX (3) P ( {HHH} ) = 8
where
pX (0) + pX (1) + pX (2) + pX (3) = 1 . ( Why ? )

74
 E3 X(s)
HHH
S
E2 3
HHT

HTH
2
THH

HTT
E1 1
THT

TTH

E0 0
TTT

Graphical representation of X .

The events E0 , E1 , E2 , E3 are disjoint since X(s) is a function !

(X : S R must be defined for all s S and must be single-valued.)

75
The graph of pX .

76
DEFINITION :
pX (x) P (X = x) ,
is called the probability mass function .

DEFINITION :
FX (x) P (X x) ,
is called the (cumulative) probability distribution function .

PROPERTIES :
FX (x) is a non-decreasing function of x . ( Why ? )
FX () = 0 and FX () = 1 . ( Why ? )
P (a < X b) = FX (b) FX (a) . ( Why ? )

NOTATION : When it is clear what X is then we also write

p(x) for pX (x) and F (x) for FX (x) .

77
EXAMPLE : With X(s) = the number of Heads , and
S = {HHH , HHT , HT H , HT T , T HH , T HT , T T H , T T T } ,
1 3 3 1
p(0) = 8
, p(1) = 8
, p(2) = 8
, p(3) = 8
,
we have the probability distribution function

F (1) P (X 1) = 0
1
F ( 0) P (X 0) = 8
4
F ( 1) P (X 1) = 8
7
F ( 2) P (X 2) = 8
F ( 3) P (X 3) = 1
F ( 4) P (X 4) = 1
We see, for example, that
P (0 < X 2) = P (X = 1) + P (X = 2)

7 1 6
= F (2) F (0) = 8
8
= 8
.

78
The graph of the probability distribution function FX .

79
EXAMPLE : Toss a coin until Heads occurs.
Then the sample space is countably infinite , namely,
S = {H , T H , T T H , T T T H , } .

The random variable X is the number of rolls until Heads occurs :

X(H) = 1 , X(T H) = 2 , X(T T H) = 3 ,
Then
1 1
p(1) = 2
, p(2) = 4
, p(3) = 18 , ( Why ? )
and n n
X X 1 1
F (n) = P (X n) = p(k) = k
= 1 n ,
k=1 k=1
2 2
and, as should be the case,
n
X X 1
p(k) = lim p(k) = lim (1 n ) = 1 .
k=1
n
k=1
n 2

NOTE : The outcomes in S do not have equal probability !

EXERCISE : Draw the probability mass and distribution functions.

80
 X(s) is the number of tosses until Heads occurs

REMARK : We can also take S Sn as all ordered outcomes of

length n. For example, for n = 4,

S4 = { HHHH , HHHT , HHT H , HHT T ,

HT HH , HT HT , HT T H , HT T T ,

T HHH , T HHT , T HT H , T HT T ,

T T HH , T T HT , T T T H , TTTT }.

where for each outcome the first Heads is marked as H .

1
Each outcome in S4 has equal probability 2n (here 24 = 16
) , and
1 1 1 1
pX (1) = 2
, pX (2) = 4
, pX (3) = 8
, pX (4) = 16
,
independent of n .

81
Joint distributions
The probability mass function and the probability distribution function
can also be functions of more than one variable.

EXAMPLE : Toss a coin 3 times in sequence. For the sample space

S = {HHH , HHT , HT H , HT T , T HH , T HT , T T H , T T T } ,
we let
X(s) = # Heads , Y (s) = index of the first H (0 for T T T ) .
Then we have the joint probability mass function
pX,Y (x, y) = P (X = x , Y = y) .
For example,
pX,Y (2, 1) = P (X = 2 , Y = 1)

= P ( 2 Heads , 1st toss is Heads)

2 1
= 8
= 4
.

82
EXAMPLE : ( continued ) For
S = {HHH , HHT , HT H , HT T , T HH , T HT , T T H , T T T } ,
X(s) = number of Heads, and Y (s) = index of the first H ,

we can list the values of pX,Y (x, y) :

Joint probability mass function pX,Y (x, y)

y=0 y=1 y=2 y=3 pX (x)
1 1
x=0 8 0 0 0 8
1 1 1 3
x=1 0 8 8 8 8
2 1 3
x=2 0 8 8 0 8
1 1
x=3 0 8 0 0 8
1 4 2 1
pY (y) 8 8 8 8 1

NOTE :
The marginal probability pX is the probability mass function of X.
The marginal probability pY is the probability mass function of Y .

83
EXAMPLE : ( continued )
X(s) = number of Heads, and Y (s) = index of the first H .

y=0 y=1 y=2 y=3 pX (x)

1 1
x=0 8 0 0 0 8
1 1 1 3
x=1 0 8 8 8 8
2 1 3
x=2 0 8 8 0 8
1 1
x=3 0 8 0 0 8
1 4 2 1
pY (y) 8 8 8 8 1

For example,

X = 2 corresponds to the event {HHT , HT H , T HH} .

Y = 1 corresponds to the event {HHH , HHT , HT H , HT T } .
(X = 2 and Y = 1) corresponds to the event {HHT , HT H} .

QUESTION : Are the events X = 2 and Y = 1 independent ?

84
 X(s)
Y(s)
E 31
HHH
S
E 13 3
TTH
E11
HTT
1
E12 THT

E 21 HTH
2
HHT

E 22 THH

0
E 00
TTT

The events Ei,j { s S : X(s) = i , Y (s) = j } are disjoint .

QUESTION : Are the events X = 2 and Y = 1 independent ?

85
DEFINITION :
pX,Y (x, y) P (X = x , Y = y) ,
is called the joint probability mass function .

DEFINITION :
FX,Y (x, y) P (X x , Y y) ,
is called the joint (cumulative) probability distribution function .

NOTATION : When it is clear what X and Y are then we also

write

p(x, y) for pX,Y (x, y) ,

and
F (x, y) for FX,Y (x, y) .

86
EXAMPLE : Three tosses : X(s) = # Heads, Y (s) = index 1st H .
Joint probability mass function pX,Y (x, y)
y=0 y=1 y=2 y=3 pX (x)
1 1
x=0 8 0 0 0 8
1 1 1 3
x=1 0 8 8 8 8
2 1 3
x=2 0 8 8 0 8
1 1
x=3 0 8 0 0 8
1 4 2 1
pY (y) 8 8 8 8 1

Joint distribution function FX,Y (x, y) P (X x, Y y)

y=0 y=1 y=2 y=3 FX ()
1 1 1 1 1
x=0 8 8 8 8 8
1 2 3 4 4
x=1 8 8 8 8 8
1 4 6 7 7
x=2 8 8 8 8 8
1 5 7
x=3 8 8 8 1 1
1 5 7
FY () 8 8 8 1 1

Note that the distribution function FX is a copy of the 4th column,

and the distribution function FY is a copy of the 4th row. ( Why ? )

87
In the preceding example :
Joint probability mass function pX,Y (x, y)
y=0 y=1 y=2 y=3 pX (x)
1 1
x=0 8 0 0 0 8
1 1 1 3
x=1 0 8 8 8 8
2 1 3
x=2 0 8 8 0 8
1 1
x=3 0 8 0 0 8
1 4 2 1
pY (y) 8 8 8 8 1

Joint distribution function FX,Y (x, y) P (X x, Y y)

y=0 y=1 y=2 y=3 FX ()
1 1 1 1 1
x=0 8 8 8 8 8
1 2 3 4 4
x=1 8 8 8 8 8
1 4 6 7 7
x=2 8 8 8 8 8
1 5 7
x=3 8 8 8 1 1
1 5 7
FY () 8 8 8 1 1

QUESTION : Why is
P (1 < X 3 , 1 < Y 3) = F (3, 3) F (1, 3) F (3, 1) + F (1, 1) ?

88
EXERCISE :
Roll a four-sided die (tetrahedron) two times.
(The sides are marked 1 , 2 , 3 , 4 .)
Suppose each of the four sides is equally likely to end facing down.
Suppose the outcome of a single roll is the side that faces down ( ! ).

Define the random variables X and Y as

X = result of the first roll , Y = sum of the two rolls.

What is a good choice of the sample space S ?

How many outcomes are there in S ?
List the values of the joint probability mass function pX,Y (x, y) .
List the values of the joint cumulative distribution function FX,Y (x, y) .

89
EXERCISE :

Three balls are selected at random from a bag containing

2 red , 3 green , 4 blue balls .

Define the random variables

R(s) = the number of red balls drawn,

and
G(s) = the number of green balls drawn .

List the values of

the joint probability mass function pR,G (r, g) .
the marginal probability mass functions pR (r) and pG (g) .
the joint distribution function FR,G (r, g) .
the marginal distribution functions FR (r) and FG (g) .

90
Independent random variables

Two discrete random variables X(s) and Y (s) are independent if

P (X = x , Y = y) = P (X = x) P (Y = y) , for all x and y ,

or, equivalently, if their probability mass functions satisfy

pX,Y (x, y) = pX (x) pY (y) , for all x and y ,

or, equivalently, if the events

Ex X 1 ({x}) and Ey Y 1 ({y}) ,
are independent in the sample space S , i.e.,
P (Ex Ey ) = P (Ex ) P (Ey ) , for all x and y .
NOTE :
In the current discrete case, x and y are typically integers .
X 1 ({x}) { s S : X(s) = x } .

91
 X(s)
Y(s)
E 31
HHH
S
E 13 3
TTH
E11
HTT
1
E12 THT

E 21 HTH
2
HHT

E 22 THH

0
E 00
TTT

Three tosses : X(s) = # Heads, Y (s) = index 1st H .

What are the values of pX (2) , pY (1) , pX,Y (2, 1) ?

Are X and Y independent ?

92
RECALL :

X(s) and Y (s) are independent if for all x and y :

pX,Y (x, y) = pX (x) pY (y) .

EXERCISE :

Roll a die two times in a row.

Let
X be the result of the 1st roll ,
and
Y the result of the 2nd roll .

Are X and Y independent , i.e., is

pX,Y (k, ) = pX (k) pY (), for all 1 k, 6 ?

93
EXERCISE :

Are these random variables X and Y independent ?

Joint probability mass function pX,Y (x, y)

y=0 y=1 y=2 y=3 pX (x)
x=0 1
8 0 0 0 1
8
x=1 0 1
8
1
8
1
8
3
8
x=2 0 2
8
1
8 0 3
8
x=3 0 1
8 0 0 1
8
pY (y) 1
8
4
8
2
8
1
8 1

94
EXERCISE : Are these random variables X and Y independent ?

Joint probability mass function pX,Y (x, y)

y = 1 y = 2 y = 3 pX (x)
x=1 1
3
1
12
1
12
1
2
x=2 2
9
1
18
1
18
1
3
x=3 1
9
1
36
1
36
1
6
pY (y) 2
3
1
6
1
6 1

Joint distribution function FX,Y (x, y) P (X x, Y y)

y = 1 y = 2 y = 3 FX (x)
x=1 1
3
5
12
1
2
1
2
x=2 5
9
25
36
5
6
5
6
x=3 2
3
5
6 1 1
FY (y) 2
3
5
6 1 1

QUESTION : Is FX,Y (x, y) = FX (x) FY (y) ?

95
PROPERTY :
The joint distribution function of independent random variables
X and Y satisfies
FX,Y (x, y) = FX (x) FY (y) , for all x, y .

PROOF :

FX,Y (xk , y ) = P (X xk , Y y )
P P
= ik j pX,Y (xi , yj )
P P
= ik j pX (xi ) pY (yj ) (by independence)
P P
= ik { pX (xi ) j pY (yj ) }
P P
= { ik pX (xi ) } { j pY (yj ) }

= FX (xk ) FY (y ) .

96
Conditional distributions

Let X and Y be discrete random variables with joint probability

mass function
pX,Y (x, y) .

For given x and y , let

Ex = X 1 ({x}) and Ey = Y 1 ({y}) ,
be their corresponding events in the sample space S.

Then
P (Ex Ey ) pX,Y (x, y)
P (Ex |Ey ) = .
P (Ey ) pY (y)

Thus it is natural to define the conditional probability mass function

pX,Y (x, y)
pX|Y (x|y) P (X = x | Y = y) = .
pY (y)

97
 X(s)
Y(s)
E 31
HHH
S
E 13 3
TTH
E11
HTT
1
E12 THT

E 21 HTH
2
HHT

E 22 THH

0
E 00
TTT

Three tosses : X(s) = # Heads, Y (s) = index 1st H .

What are the values of P (X = 2 | Y = 1) and P (Y = 1 | X = 2) ?

98
EXAMPLE : (3 tosses : X(s) = # Heads, Y (s) = index 1st H.)
Joint probability mass function pX,Y (x, y)
y = 0 y = 1 y = 2 y = 3 pX (x)
x=0 1
8 0 0 0 1
8
x=1 0 1
8
1
8
1
8
3
8
x=2 0 2
8
1
8 0 3
8
x=3 0 1
8 0 0 1
8
pY (y) 1
8
4
8
2
8
1
8 1
pX,Y (x,y)
Conditional probability mass function pX|Y (x|y) = pY (y)
.
y=0 y=1 y=2 y=3
x=0 1 0 0 0
x=1 0 2
8
4
8 1
x=2 0 4
8
4
8 0
x=3 0 2
8 0 0
1 1 1 1

pX,Y (x,y)
EXERCISE : Also construct the Table for pY |X (y|x) = pX (x)
.

99
EXAMPLE :
Joint probability mass function pX,Y (x, y)
y = 1 y = 2 y = 3 pX (x)
x=1 1
3
1
12
1
12
1
2
x=2 2
9
1
18
1
18
1
3
x=3 1
9
1
36
1
36
1
6
pY (y) 2
3
1
6
1
6 1

pX,Y (x,y)
Conditional probability mass function pX|Y (x|y) = pY (y)
.
y=1 y=2 y=3
x=1 1
2
1
2
1
2
x=2 1
3
1
3
1
3
x=3 1
6
1
6
1
6
1 1 1

QUESTION : What does the last Table tell us?

EXERCISE : Also construct the Table for P (Y = y|X = x) .

100
Expectation
The expected value of a discrete random variable X is
X X
E[X] xk P (X = xk ) = xk pX (xk ) .
k k

Thus E[X] represents the weighted average value of X .

( E[X] is also called the mean of X .)

EXAMPLE : The expected value of rolling a die is

1 1 1 1
P6 7
E[X] = 1 6 + 2 6 + + 6 6 = 6 k=1 k = 2
.

EXERCISE : Prove the following :

E[aX] = a E[X] ,
E[aX + b] = a E[X] + b .

101
EXAMPLE : Toss a coin until Heads occurs. Then
S = {H , T H , T T H , T T T H , } .

The random variable X is the number of tosses until Heads occurs :

X(H) = 1 , X(T H) = 2 , X(T T H) = 3 .
Then
n
1 1 1 X k
E[X] = 1 + 2 + 3 + = lim k
= 2.
2 4 8 n
k=1
2
Pn k
n k=1 k/2
1 0.50000000
2 1.00000000
3 1.37500000
10 1.98828125
40 1.99999999
REMARK :
Perhaps using Sn = {all sequences of n tosses} is better

102
The expected value of a function of a random variable is

X
E[g(X)] g(xk ) p(xk ) .
k

EXAMPLE :
The pay-off of rolling a die is $k2 , where k is the side facing up.

What should the entry fee be for the betting to break even?

SOLUTION : Here g(X) = X 2 , and

6
X
21 1 6(6 + 1)(2 6 + 1) 91
E[g(X)] = k = = = $15.17 .
k=1
6 6 6 6

103
The expected value of a function of two random variables is
XX
E[g(X, Y )] g(xk , y ) p(xk , y ) .
k

EXAMPLE : y=1 y=2 y=3 pX (x)

x=1 1
3
1
12
1
12
1
2
x=2 2
9
1
18
1
18
1
3
x=3 1
9
1
36
1
36
1
6
pY (y) 2
3
1
6
1
6 1

1 1 1 5
E[X] = 1 2
+ 2 3
+ 3 6
= 3
,
2 1 1 3
E[Y ] = 1 3
+ 2 6
+ 3 6
= 2
,
1 1 1
E[XY ] = 1 3
+ 2 12
+ 3 12

2 1 1
+ 2 9
+ 4 18
+ 6 18

1 1 1 5 ( So ? )
+ 3 9
+ 6 36
+ 9 36
= 2
.

104
PROPERTY :

If X and Y are independent then E[XY ] = E[X] E[Y ] .

PROOF :
P P
E[XY ] = k xk y pX,Y (xk , y )
P P
= k xk y pX (xk ) pY (y ) (by independence)
P P
= k{ xk pX (xk ) y pY (y )}
P P
= { k xk pX (xk )} { y pY (y )}

= E[X] E[Y ] .

EXAMPLE : See the preceding example !

105
PROPERTY : E[X + Y ] = E[X] + E[Y ] . ( Always ! )

PROOF :
P P
E[X + Y ] = k (xk + y ) pX,Y (xk , y )
P P P P
= k xk pX,Y (xk , y ) + k y pX,Y (xk , y )
P P P P
= k xk pX,Y (xk , y ) + k y pX,Y (xk , y )
P P P P
= k {xk pX,Y (xk , y )} + { y k pX,Y (xk , y )}
P P
= k {xk pX (xk )} + {y pY (y )}

= E[X] + E[Y ] .

NOTE : X and Y need not be independent !

106
EXERCISE :
Probability mass function pX,Y (x, y)
y = 6 y = 8 y = 10 pX (x)
x=1 1
5 0 1
5
2
5
x=2 0 1
5 0 1
5
x=3 1
5 0 1
5
2
5
pY (y) 2
5
1
5
2
5 1

Show that

E[X] = 2 , E[Y ] = 8 , E[XY ] = 16

X and Y are not independent

Thus if
E[XY ] = E[X] E[Y ] ,

then it does not necessarily follow that X and Y are independent !

107
Variance and Standard Deviation
Let X have mean
= E[X] .

Then the variance of X is

X
2
V ar(X) E[ (X ) ] (xk )2 p(xk ) ,
k

which is the average weighted square distance from the mean.

We have
V ar(X) = E[X 2 2X + 2 ]

= E[X 2 ] 2E[X] + 2

= E[X 2 ] 22 + 2

= E[X 2 ] 2 .

108
The standard deviation of X is
p p p
(X) V ar(X) = E[ (X )2 ] = E[X 2 ] 2 .

which is the average weighted distance from the mean.

EXAMPLE : The variance of rolling a die is

6
X 1
V ar(X) = [k2 ] 2
k=1
6

1 6(6 + 1)(2 6 + 1) 7 2 35
= ( ) = .
6 6 2 12

The standard deviation is

r
35
= = 1.70 .
12

109
Covariance
Let X and Y be random variables with mean

E[X] = X , E[Y ] = Y .

Then the covariance of X and Y is defined as

X
Cov(X, Y ) E[ (XX ) (Y Y ) ] = (xk X ) (y Y ) p(xk , y ) .
k,

We have
Cov(X, Y ) = E[ (X X ) (Y Y ) ]

= E[XY X Y Y X + X Y ]

= E[XY ] X Y Y X + X Y

= E[XY ] E[X] E[Y ] .

110
We defined

Cov(X, Y ) E[ (X X ) (Y Y ) ]
X
= (xk X ) (y Y ) p(xk , y )
k,

= E[XY ] E[X] E[Y ] .

NOTE :
Cov(X, Y ) measures concordance or coherence of X and Y :

If X > X when Y > Y and X < X when Y < Y then

Cov(X, Y ) > 0 .

If X > X when Y < Y and X < X when Y > Y then

Cov(X, Y ) < 0 .

111
EXERCISE : Prove the following :

V ar(aX + b) = a2 V ar(X) ,

Cov(X, Y ) = Cov(Y, X) ,

Cov(cX, Y ) = c Cov(X, Y ) ,

Cov(X, cY ) = c Cov(X, Y ) ,

Cov(X + Y, Z) = Cov(X, Z) + Cov(Y, Z) ,

V ar(X + Y ) = V ar(X) + V ar(Y ) + 2 Cov(X, Y ) .

112
PROPERTY :

If X and Y are independent then Cov(X, Y ) = 0 .

PROOF :

We have already shown ( with X E[X] and Y E[Y ] ) that

Cov(X, Y ) E[ (X X ) (Y Y ) ] = E[XY ] E[X] E[Y ] ,

and that if X and Y are independent then

E[XY ] = E[X] E[Y ] .

from which the result follows.

113
EXERCISE : ( already used earlier )
Probability mass function pX,Y (x, y)
y = 6 y = 8 y = 10 pX (x)
x=1 1
5 0 1
5
2
5
x=2 0 1
5 0 1
5
x=3 1
5 0 1
5
2
5
pY (y) 2
5
1
5
2
5 1
Show that
E[X] = 2 , E[Y ] = 8 , E[XY ] = 16
Cov(X, Y ) = E[XY ] E[X] E[Y ] = 0
X and Y are not independent

Thus if
Cov(X, Y ) = 0 ,

then it does not necessarily follow that X and Y are independent !

114
PROPERTY :

If X and Y are independent then

V ar(X + Y ) = V ar(X) + V ar(Y ) .

PROOF :

We have already shown (in an exercise !) that

V ar(X + Y ) = V ar(X) + V ar(Y ) + 2 Cov(X, Y ) ,

and that if X and Y are independent then

Cov(X, Y ) = 0 ,

from which the result follows.

115
EXERCISE :

Compute
E[X] , E[Y ] , E[X 2 ] , E[Y 2 ]

E[XY ] , V ar(X) , V ar(Y )

Cov(X, Y )
for

Joint probability mass function pX,Y (x, y)

y = 0 y = 1 y = 2 y = 3 pX (x)
x=0 1
8 0 0 0 1
8
x=1 0 1
8
1
8
1
8
3
8
x=2 0 2
8
1
8 0 3
8
x=3 0 1
8 0 0 1
8
pY (y) 1
8
4
8
2
8
1
8 1

116
EXERCISE :

Compute
E[X] , E[Y ] , E[X 2 ] , E[Y 2 ]

E[XY ] , V ar(X) , V ar(Y )

Cov(X, Y )
for

Joint probability mass function pX,Y (x, y)

y=1 y=2 y=3 pX (x)
x=1 1
3
1
12
1
12
1
2
x=2 2
9
1
18
1
18
1
3
x=3 1
9
1
36
1
36
1
6
pY (y) 2
3
1
6
1
6 1

117
 SPECIAL DISCRETE RANDOM VARIABLES

The Bernoulli Random Variable

A Bernoulli trial has only two outcomes , with probability

P (X = 1) = p,

P (X = 0) = 1p ,
e.g., tossing a coin, winning or losing a game, .
We have
E[X] = 1 p + 0 (1 p) = p ,

E[X 2 ] = 12 p + 02 (1 p) = p ,

V ar(X) = E[X 2 ] E[X]2 = p p2 = p(1 p) .

NOTE : If p is small then V ar(X)

=p.

118
EXAMPLES :

1
When p = 2
(e.g., for tossing a coin), we have
1 1
E[X] = p = 2
, V ar(X) = p(1 p) = 4
.

When rolling a die , with outcome k , (1 k 6) , let

X(k) = 1 if the roll resulted in a six ,
and
X(k) = 0 if the roll did not result in a six .
Then
1 5
E[X] = p = 6
, V ar(X) = p(1 p) = 36
.

When p = 0.01 , then

E[X] = 0.01 , V ar(X) = 0.0099

= 0.01 .

119
The Binomial Random Variable

Perform a Bernoulli trial n times in sequence .

Assume the individual trials are independent .

An outcome could be
100011001010 (n = 12) ,
with probability
P (100011001010) = p5 (1 p)7 . ( Why ? )

Let the X be the number of successes (i.e. 1s) .

For example,
X(100011001010) = 5 .
We have
 
12
P (X = 5) = p5 (1 p)7 . ( Why ? )
5

120
In general, for k successes in a sequence of n trials, we have
 
n
P (X = k) = pk (1 p)nk , (0 k n) .
k

EXAMPLE : Tossing a coin 12 times:

1
n = 12 , p = 2
k pX (k) FX (k)
0 1 / 4096 1 / 4096
1 12 / 4096 13 / 4096
2 66 / 4096 79 / 4096
3 220 / 4096 299 / 4096
4 495 / 4096 794 / 4096
5 792 / 4096 1586 / 4096
6 924 / 4096 2510 / 4096
7 792 / 4096 3302 / 4096
8 495 / 4096 3797 / 4096
9 220 / 4096 4017 / 4096
10 66 / 4096 4083 / 4096
11 12 / 4096 4095 / 4096
12 1 / 4096 4096 / 4096

121
 1
The Binomial mass and distribution functions for n = 12 , p = 2

122
For k successes in a sequence of n trials :
 
n
P (X = k) = pk (1 p)nk , (0 k n) .
k

EXAMPLE : Rolling a die 12 times:

1
n = 12 , p = 6
k pX (k) FX (k)
0 0.1121566221 0.112156
1 0.2691758871 0.381332
2 0.2960935235 0.677426
3 0.1973956972 0.874821
4 0.0888280571 0.963649
5 0.0284249838 0.992074
6 0.0066324966 0.998707
7 0.0011369995 0.999844
8 0.0001421249 0.999986
9 0.0000126333 0.999998
10 0.0000007580 0.999999
11 0.0000000276 0.999999
12 0.0000000005 1.000000

123
 1
The Binomial mass and distribution functions for n = 12 , p = 6

124
EXAMPLE :
In 12 rolls of a die write the outcome as, for example,

100011001010
where
1 denotes the roll resulted in a six ,
and
0 denotes the roll did not result in a six .

As before, let X be the number of 1s in the outcome.

Then X represents the number of sixes in the 12 rolls.

Then, for example, using the preceding Table :

P (X = 5)
= 2.8 % , P (X 5)
= 99.2 % .

125
EXERCISE : Show that from
 
n
P (X = k) = pk (1 p)nk ,
k
and  
n
P (X = k + 1) = pk+1 (1 p)nk1 ,
k+1
it follows that
P (X = k + 1) = ck P (X = k) ,
where nk p
ck = .
k+1 1p

NOTE : This recurrence formula is an efficient and stable algorithm

to compute the binomial probabilities :
P (X = 0) = (1 p)n ,

P (X = k + 1) = ck P (X = k) , k = 0, 1, , n 1 .

126
Mean and variance of the Binomial random variable :

By definition, the mean of a Binomial random variable X is

n n  
X X n
E[X] = k P (X = k) = k pk (1 p)nk ,
k
k=0 k=0

which can be shown to equal np .

An easy way to see this is as follows :

If in a sequence of n independent Bernoulli trials we let

Xk = the outcome of the kth Bernoulli trial , (Xk = 0 or 1 ) ,

then
X X1 + X2 + + Xn ,
is the Binomial random variable that counts the successes .

127
 X X1 + X2 + + Xn

We know that
E[Xk ] = p ,
so
E[X] = E[X1 ] + E[X2 ] + + E[Xn ] = np .

We already know that

V ar(Xk ) = E[Xk2 ] (E[Xk ])2 = p p2 = p(1 p) ,

so, since the Xk are independent , we have

V ar(X) = V ar(X1 ) + V ar(X2 ) + + V ar(Xn ) = np(1 p) .

NOTE : If p is small then V ar(X)

= np .

128
EXAMPLES :

For 12 tosses of a coin , with Heads is success, we have

1
so n = 12 , p=
2
E[X] = np = 6 , V ar(X) = np(1 p) = 3 .

For 12 rolls of a die , with six is success , we have

1
so n = 12 , p=
6
E[X] = np = 2 , V ar(X) = np(1 p) = 5/3 .

If n = 500 and p = 0.01 , then

E[X] = np = 5 , V ar(X) = np(1 p) = 4.95

= 5.

129
The Poisson Random Variable

The Poisson variable approximates the Binomial random variable :

k
 
n k nk
P (X = k) = p (1 p) = e ,
k k!
when we take

= n p ( the average number of successes ) .

This approximation is accurate if n is large and p small .

Recall that for the Binomial random variable

E[X] = n p , and V ar(X) = np(1 p)
= np when p is small.

Indeed, for the Poisson random variable we will show that

E[X] = and V ar(X) = .

130
A stable and efficient way to compute the Poisson probability

k
P (X = k) = e , k = 0, 1, 2, ,
k!

k+1
P (X = k + 1) = e ,
(k + 1)!

is to use the recurrence relation

P (X = 0) = e ,

P (X = k + 1) = P (X = k) , k = 0, 1, 2, .
k+1

NOTE : Unlike the Binomial random variable, the Poisson random

variable can have an arbitrarily large integer value k.

131
The Poisson random variable
k
P (X = k) = e , k = 0, 1, 2, ,
k!
has (as shown later) : E[X] = and V ar(X) = .

The Poisson distribution function is

k k
X X
F (k) = P (X k) = e = e ,
=0
! =0
!

with, as should be the case,

X
lim F (k) = e = e e = 1 .
k
=0
!

( using the Taylor series from Calculus for e ) .

132
The Poisson random variable
k
P (X = k) = e , k = 0, 1, 2, ,
k!
models the probability of k successes in a given time interval,
when the average number of successes is .

EXAMPLE : Suppose customers arrive at the rate of six per hour.

The probability that k customers arrive in a one-hour period is
60
P (k = 0) = e6 = 0.0024 ,
0!
61
P (k = 1) = e6 = 0.0148 ,
1!
62
P (k = 2) = e6 = 0.0446 .
2!
The probability that more than 2 customers arrive is
1 (0.0024 + 0.0148 + 0.0446)
= 0.938 .

133
 k
 
n
pBinomial (k) = pk (1 p)nk = pPoisson (k) = e
k k!

EXAMPLE : = 6 customers/hour.
For the Binomial take n = 12 , p = 0.5 (0.5 customers/5 minutes) ,
so that indeed np = .
k pBinomial pPoisson FBinomial FPoisson
0 0.0002 0.0024 0.0002 0.0024
1 0.0029 0.0148 0.0031 0.0173
2 0.0161 0.0446 0.0192 0.0619
3 0.0537 0.0892 0.0729 0.1512
4 0.1208 0.1338 0.1938 0.2850
5 0.1933 0.1606 0.3872 0.4456
6 0.2255 0.1606 0.6127 0.6063
7 0.1933 0.1376 0.8061 0.7439
8 0.1208 0.1032 0.9270 0.8472
9 0.0537 0.0688 0.9807 0.9160
10 0.0161 0.0413 0.9968 0.9573
11 0.0029 0.0225 0.9997 0.9799
12 0.0002 0.0112 1.0000 0.9911 Why not 1.0000 ?

Here the approximation is not so good

134
 k
 
n
pBinomial (k) = pk (1 p)nk = pPoisson (k) = e
k k!
EXAMPLE : = 6 customers/hour.
For the Binomial take n = 60 , p = 0.1 (0.1 customers/minute) ,
so that indeed np = .
k pBinomial pPoisson FBinomial FPoisson
0 0.0017 0.0024 0.0017 0.0024
1 0.0119 0.0148 0.0137 0.0173
2 0.0392 0.0446 0.0530 0.0619
3 0.0843 0.0892 0.1373 0.1512
4 0.1335 0.1338 0.2709 0.2850
5 0.1662 0.1606 0.4371 0.4456
6 0.1692 0.1606 0.6064 0.6063
7 0.1451 0.1376 0.7515 0.7439
8 0.1068 0.1032 0.8583 0.8472
9 0.0685 0.0688 0.9269 0.9160
10 0.0388 0.0413 0.9657 0.9573
11 0.0196 0.0225 0.9854 0.9799
12 0.0089 0.0112 0.9943 0.9911
13

Here the approximation is better

135
 1
n = 12 , p = 2
, =6 n = 200 , p = 0.01 , = 2

The Binomial (blue) and Poisson (red) probability mass functions.

For the case n = 200, p = 0.01, the approximation is very good !

136
For the Binomial random variable we found
E[X] = np and V ar(X) = np(1 p) ,

while for the Poisson random variable, with = np we will show

E[X] = np and V ar(X) = np .

Note again that

np(1 p)
= np , when p is small .

EXAMPLE : In the preceding two Tables we have

n=12 , p=0.5 n=60 , p=0.1
Binomial Poisson Binomial Poisson
E[X] 6.0000 6.0000 E[X] 6.0000 6.0000
V ar[X] 3.0000 6.0000 V ar[X] 5.4000 6.0000
[X] 1.7321 2.4495 [X] 2.3238 2.4495

137
FACT : (The Method of Moments)

By Taylor expansion of etX about t = 0 , we have

2 2 3 3
h t X t X i
(t) E[etX ] = E 1 + tX + + +
2! 3!
t2 t3
= 1 + t E[X] + E[X 2 ] + E[X 3 ] + .
2! 3!

It follows that
(0) = E[X] , (0) = E[X 2 ] . ( Why ? )

This sometimes facilitates computing the mean

= E[X] ,
and the variance
V ar(X) = E[X 2 ] 2 .

138
APPLICATION : The Poisson mean and variance :

X X k
(t) E[etX ] = etk P (X = k) = etk e
k=0 k=0
k!


X (et )k et t
= e = e e = e(e 1) .
k=0
k!

t
Here (t) = et e(e 1)

t 2 t (et 1)
 
(t) = (e ) + e e ( Check ! )

so that E[X] = (0) =

E[X 2 ] = (0) = ( + 1) = 2 +

V ar(X) = E[X 2 ] E[X]2 = .

139
EXAMPLE : Defects in a wire occur at the rate of one per 10 meter,
with a Poisson distribution :
k
P (X = k) = e , k = 0, 1, 2, .
k!
What is the probability that :

A 12-meter roll has at no defects?

ANSWER : Here = 1.2 , and P (X = 0) = e = 0.3012 .

A 12-meter roll of wire has one defect?

ANSWER : With = 1.2 , P (X = 1) = e = 0.3614 .

Of five 12-meter rolls two have one defect and three have none?
 
5
ANSWER : 0.30123 0.36142 = 0.0357 . ( Why ? )
3

140
EXERCISE :
Defects in a certain wire occur at the rate of one per 10 meter.
Assume the defects have a Poisson distribution.
What is the probability that :
a 20-meter wire has no defects?

a 20-meter wire has at most 2 defects?

EXERCISE :
Customers arrive at a counter at the rate of 8 per hour.
Assume the arrivals have a Poisson distribution.
What is the probability that :
no customer arrives in 15 minutes?

two customers arrive in a period of 30 minutes?

141
 CONTINUOUS RANDOM VARIABLES

DEFINITION : A continuous random variable is a function X(s)

from an uncountably infinite sample space S to the real numbers R ,

X() : S R.
EXAMPLE :
Rotate a pointer about a pivot in a plane (like a hand of a clock).
The outcome is the angle where it stops : 2 , where (0, 1] .
A good sample space is all values of , i.e. S = (0, 1] .
A very simple example of a continuous random variable is X() = .

Suppose any outcome, i.e., any value of is equally likely.

What are the values of
1
P (0 < 2
) , P ( 13 < 1
2
) , P ( = 1 )
2
?

142
The (cumulative) probability distribution function is defined as

FX (x) P (X x) .
Thus
FX (b) FX (a) P (a < X b) .

We must have

FX () = 0 and FX () = 1 ,
i.e.,
lim FX (x) = 0 ,
x
and
lim FX (x) = 1 .
x

Also, FX (x) is a non-decreasing function of x . ( Why ? )

NOTE : All the above is the same as for discrete random variables !

143
EXAMPLE : In the pointer example , where X() = , we have
the probability distribution function
F(theta)

1/2

1/3

theta
0 1/3 1/2 1
Note that
F ( 31 ) P (X 13 ) = 1
3
, F ( 21 ) P (X 12 ) = 1
2
,

P ( 31 < X 12 ) = F ( 12 ) F ( 13 ) = 1
2
1
3
= 1
6
.

QUESTION : What is P ( 31 X 21 ) ?

144
The probability density function is the derivative of the probability
distribution function :
d
fX (x) FX (x) FX (x) .
dx

EXAMPLE : In the pointer example

0, x0
FX (x) = x, 0<x1
1, 1<x

Thus
0, x0
fX (x) = FX (x) = 1, 0<x1
0, 1<x

NOTATION : When it is clear what X is then we also write

f (x) for fX (x) , and F (x) for FX (x) .

145
EXAMPLE : ( continued )

0, x0 0, x0
F (x) = x, 0<x1 , f (x) = 1, 0<x1
1, 1<x 0, 1<x

F(theta) f(theta)

1 1

1/2

1/3

theta theta
0 1/3 1/2 1 0 1/3 1/2 1

Distribution function Density function

NOTE :
1
1 1 1
Z
2
P( < X ) = f (x) dx = = the shaded area .
3 2 1
3
6

146
In general, from
f (x) F (x) ,
with
F () = 0 and F () = 1 ,

we have from Calculus the following basic identities :

Z Z
f (x) dx = F (x) dx = F () F () = 1 ,

Z x
f (x) dx = F (x) F () = F (x) = P (X x) ,

Z b
f (x) dx = F (b) F (a) = P (a < X b) ,
a

Z a
f (x) dx = F (a) F (a) = 0 = P (X = a) .
a

147
EXERCISE : Draw graphs of the distribution and density functions
 
0, x0 0, x0
F (x) = , f (x) = ,
1 ex , x > 0 ex , x > 0

and verify that

F () = 0 , F () = 1 ,
f (x) = F (x) ,
Rx
F (x) = 0
f (x) dx , ( Why is zero as lower limit OK ? )
R
0
f (x) dx = 1 ,

P (0 < X 1) = F (1) F (0) = F (1) = 1 e1

= 0.63 ,

P (X > 1) = 1 F (1) = e1
= 0.37 ,

P (1 < X 2) = F (2) F (1) = e1 e2

= 0.23 .

148
EXERCISE : For positive integer n, consider the density functions
n
cx (1 xn ) , 0 x 1
fn (x) =
0, otherwise

Determine the value of c in terms of n .

Draw the graph of fn (x) for n = 1, 2, 4, 8, 16 .

Determine the distribution function Fn (x) .

Draw the graph of Fn (x) for n = 1, 2, 3, 4, 8, 16 .

Determine P (0 X 21 ) in terms of n .

What happens to P (0 X 21 ) when n becomes large?

9
Determine P ( 10 X 1) in terms of n .
9
What happens to P ( 10 X 1) when n becomes large?

149
Joint distributions

A joint probability density function fX,Y (x, y) must satisfy

Z Z
fX,Y (x, y) dx dy = 1 ( Volume = 1 ).

The corresponding joint probability distribution function is

Z y Z x
FX,Y (x, y) = P (X x , Y y) = fX,Y (x, y) dx dy .

2 FX,Y (x, y)
By Calculus we have = fX,Y (x, y) .
xy

Also, Z d Z b
P (a < X b , c < Y d) = fX,Y (x, y) dx dy .
c a

150
EXAMPLE :

If
1 for x (0, 1] and y (0, 1] ,
fX,Y (x, y) =
0 otherwise ,

then, for x (0, 1] and y (0, 1] ,

Z y Z x
FX,Y (x, y) = P (X x , Y y) = 1 dx dy = xy .
0 0

Thus
FX,Y (x, y) = xy , for x (0, 1] and y (0, 1] .

For example
1 1 1 1 1
P( X , Y ) = FX,Y ( , ) = .
3 2 3 2 6

151
 0.9
1.5 0.8
0.7
1.0 0.6
f 0.5
0.5
F 0.4
0.3
0.2

0.1 0.1
0.2 0.1
0.2
0.3 0.3
0.4
y 0.4 0.5 0.3
0.2
0.1
0.1 0.2
0.3 0.4 0.7
0.6
0.5 x
0.5
0.6
0.7 0.5
0.4
x y 0.6 0.7 0.8 0.9
0.9
0.8
0.6
0.8 0.7
0.9 0.8
0.9

Also,
3 1
1 1 1 3 1
Z Z
4 2
P( X , Y ) = f (x, y) dx dy = .
3 2 4 4 1
4
1
3
12

EXERCISE : Show that we can also compute this as follows :

F ( 12 , 43 ) F ( 13 , 34 ) F ( 12 , 41 ) + F ( 13 , 41 ) = 1
12
.
and explain why !

152
Marginal density functions

The marginal density functions are

Z Z
fX (x) = fX,Y (x, y) dy , fY (y) = fX,Y (x, y) dx .

with corresponding marginal distribution functions

Z x Z x Z
FX (x) P (X x) = fX (x) dx = fX,Y (x, y) dy dx ,

Z y Z y Z
FY (y) P (Y y) = fY (y) dy = fX,Y (x, y) dx dy .

By Calculus we have
dFX (x) dFY (y)
= fX (x) , = fY (y) .
dx dy

153
EXAMPLE : If

1 for x (0, 1] and y (0, 1] ,
fX,Y (x, y) =
0 otherwise ,

then, for x (0, 1] and y (0, 1] ,

Z 1 Z 1
fX (x) = fX,Y (x, y) dy = 1 dy = 1 ,
0 0
Z 1 Z 1
fY (y) = fX,Y (x, y) dx = 1 dx = 1 ,
0 0
Z x
FX (x) = P (X x) = fX (x) dx = x ,
0
Z y
FY (y) = P (Y y) = fY (y) dy = y .
0
For example
1
P( X 3
) = FX ( 13 ) = 1
3
, P( Y 1
2
) = FY ( 21 ) = 1
2
.

154
 EXERCISE :
(1 ex )(1 ey )

for x 0 and y 0 ,
Let FX,Y (x, y) =
0 otherwise .
Verify that
2
exy

F for x 0 and y 0 ,
fX,Y (x, y) = =
xy 0 otherwise .

0.9
0.8
0.7
0.9
0.6 0.8
0.5 0.7
f 0.4 0.6
0.5
0.3
F 0.4
0.2 0.3
0.1 0.2
0.1

1 1
2 1
1 2
3 2 2
y 4
3 3
4 x y
3 x
4 4

Density function fX,Y (x, y) Distribution function FX,Y (x, y)

155
EXERCISE : ( continued )
FX,Y (x, y) = (1ex )(1ey ) , fX,Y (x, y) = exy , for x, y 0 .

Also verify the following :

F (0, 0) = 0 , F (, ) = 1 ,

RR
0 0
fX,Y (x, y) dx dy = 1 , ( Why zero lower limits ? )

R
fX (x) = 0
exy dy = ex ,

R
fY (y) = 0
exy dx = ey .

fX,Y (x, y) = fX (x) fY (y) . ( So ? )

156
EXERCISE : ( continued )

FX,Y (x, y) = (1ex )(1ey ) , fX,Y (x, y) = exy , for x, y 0 .

Also verify the following :

Rx R x x
FX (x) = 0 fX (x) dx = 0 e dx = 1 ex ,
Ry Ry
FY (y) = 0
fY (y) dy = 0
ey dy = 1 ey ,

FX,Y (x, y) = FX (x) FY (y) . ( So ? )

P (1 < x < ) = FX ()FX (1) = 1(1e1 ) = e1

= 0.37 ,
R1R2
P( 1 < x 2 , 0 < y 1 ) = 0 1
exy dx dy

= (e1 e2 )(1 e1 )
= 0.15 ,

157
Independent continuous random variables

Recall that two events E and F are independent if

P (EF ) = P (E) P (F ) .

Continuous random variables X(s) and Y (s) are independent if

P (X IX , Y IY ) = P (X IX ) P (Y IY ) ,
for all allowable sets IX and IY (typically intervals) of real numbers.

Equivalently, X(s) and Y (s) are independent if for all such sets
IX and IY the events
X 1 (IX ) and Y 1 (IY ) ,
are independent in the sample space S.

NOTE : X 1 (IX ) {s S : X(s) IX } ,

Y 1 (IY ) {s S : Y (s) IY } .

158
FACT : X(s) and Y (s) are independent if for all x and y
fX,Y (x, y) = fX (x) fY (y) .

EXAMPLE : The random variables with density function

 xy
e for x 0 and y 0 ,
fX,Y (x, y) =
0 otherwise ,
are independent because (by the preceding exercise)
fX,Y (x, y) = exy = ex ey = fX (x) fY (y) .

NOTE :
(1 ex )(1 ey ) for x 0 and y 0 ,

FX,Y (x, y) =
0 otherwise ,
also satisfies (by the preceding exercise)
FX,Y (x, y) = FX (x) FY (y) .

159
PROPERTY :
For independent continuous random variables X and Y we have
FX,Y (x, y) = FX (x) FY (y) , for all x, y .
PROOF :
FX,Y (x, y) = P (X x , Y y)
Rx Ry
=
fX,Y (x, y) dy dx
Rx Ry
=
fX (x) fY (y) dy dx (by independence)
Rx Ry
=
[ fX (x)
fY (y) dy ] dx
Rx Ry
= [
fX (x) dx ] [
fY (y) dy ]

= FX (x) FY (y) .

REMARK : Note how the proof parallels that for the discrete case !

160
Conditional distributions
Let X and Y be continuous random variables.
For given allowable sets IX and IY (typically intervals), let
Ex = X 1 (IX ) and Ey = Y 1 (IY ) ,
be their corresponding events in the sample space S .
P (Ex Ey )
We have P (Ex |Ey ) .
P (Ey )

The conditional probability density function is defined as

fX,Y (x, y)
fX|Y (x|y) .
fY (y)

When X and Y are independent then

fX,Y (x, y) fX (x) fY (y)
fX|Y (x|y) = = fX (x) ,
fY (y) fY (y)
(assuming fY (y) 6= 0 ).

161
EXAMPLE : The random variables with density function
xy
e for x 0 and y 0 ,
fX,Y (x, y) =
0 otherwise ,

have (by previous exercise) the marginal density functions

fX (x) = ex , fY (y) = ey ,
for x 0 and y 0 , and zero otherwise.

Thus for such x, y we have

fX,Y (x, y) exy
fX|Y (x|y) = = = ex = fX (x) ,
fY (y) ey
i.e., information about Y does not alter the density function of X .

Indeed, we have already seen that X and Y are independent .

162
Expectation

The expected value of a continuous random variable X is

Z
E[X] = x fX (x) dx ,

which represents the average value of X over many trials.

The expected value of a function of a random variable is

Z
E[g(X)] = g(x) fX (x) dx .

The expected value of a function of two random variables is

Z Z
E[g(X, Y )] = g(x, y) fX,Y (x, y) dy dx .

163
EXAMPLE :

For the pointer experiment

0, x0



fX (x) = 1, 0<x1




0, 1<x

we have
1
x2 1 1
Z Z
E[X] = x fX (x) dx = x dx =  = ,
0 2 0 2

and
1
x3 1 1
Z Z
2 2 2
E[X ] = x fX (x) dx = x dx =  = .
0 3 0 3

164
EXAMPLE : For the joint density function
xy
e for x > 0 and y > 0 ,
fX,Y (x, y) =
0 otherwise .

we have (by previous exercise) the marginal density functions

x y
e for x > 0 , e for y > 0 ,
fX (x) = and fY (y) =
0 otherwise , 0 otherwise .

Z 
Thus E[X] = x ex dx = [(x+1)ex ] = 1 . ( Check ! )

0 0
Z
Similarly E[Y ] = y ey dy = 1 ,
0
and Z Z
E[XY ] = xy exy dy dx = 1 . ( Check ! )
0 0

165
EXERCISE :
Prove the following for continuous random variables :

E[aX] = a E[X] ,

E[aX + b] = a E[X] + b ,

E[X + Y ] = E[X] + E[Y ] ,

and compare the proofs to those for discrete random variables.

EXERCISE :
A stick of length 1 is split at a randomly selected point X.
( Thus X is uniformly distributed in the interval [0, 1]. )
Determine the expected length of the piece containing the point 1/3.

166
PROPERTY : If X and Y are independent then

E[XY ] = E[X] E[Y ] .

PROOF :

R R
E[XY ] = R R
x y fX,Y (x, y) dy dx
R R
= R R
x y fX (x) fY (y) dy dx (by independence)
R R
= R
[ x fX (x) R
y fY (y) dy ] dx
R R
= [ R x fX (x) dx ] [ R y fY (y) dy ]

= E[X] E[Y ] .

REMARK : Note how the proof parallels that for the discrete case !

167
EXAMPLE : For xy
e for x > 0 and y > 0 ,
fX,Y (x, y) =
0 otherwise ,

we already found
fX (x) = ex , fY (y) = ey ,
so that
fX,Y (x, y) = fX (x) fY (y) ,

i.e., X and Y are independent .

Indeed, we also already found that

E[X] = E[Y ] = E[XY ] = 1 ,
so that
E[XY ] = E[X] E[Y ] .

168
Variance Z
Let = E[X] = x fX (x) dx

Then the variance of the continuous random variable X is

Z
V ar(X) E[ (X )2 ] (x )2 fX (x) dx ,

which is the average weighted square distance from the mean.

As in the discrete case, we have

V ar(X) = E[X 2 2X + 2 ]

= E[X 2 ] 2E[X] + 2 = E[X 2 ] 2 .

The standard deviation of X is

p p
(X) V ar(X) = E[X 2 ] 2 .
which is the average weighted distance from the mean.

169
 ex ,

x>0,
EXAMPLE : For f (x) =
0, x0,
we have
R
E[X] = = 0
x ex dx = 1 ( already done ! ) ,
R 
E[X 2 ] = x2 ex dx = [(x2 + 2x + 2)ex ] = 2,

0
0

V ar(X) = E[X 2 ] 2 = 2 12 = 1 ,
p
(X) = V ar(X) = 1 .

NOTE : The two integrals can be done by integration by parts .

EXERCISE :
Also use the Method of Moments to compute E[X] and E[X 2 ] .

170
EXERCISE : For the random variable X with density function

0, x 1
f (x) = c , 1 < x 1
0, x>1

Determine the value of c

Draw the graph of f (x)
Determine the distribution function F (x)
Draw the graph of F (x)
Determine E[X]
Compute V ar(X) and (X)
Determine P (X 12 )
Determine P (| X | 12 )

171
EXERCISE : For the random variable X with density function

x + 1 , 1 < x 0
f (x) = 1x , 0<x1
0, otherwise

Draw the graph of f (x)

R
Verify that f (x) dx = 1
Determine the distribution function F (x)
Draw the graph of F (x)
Determine E[X]
Compute V ar(X) and (X)
Determine P (X 13 )
Determine P (| X | 13 )

172
EXERCISE : For the random variable X with density function
3
(1 x2 ) , 1 < x 1

f (x) = 4
0, otherwise
Draw the graph of f (x)
R
Verify that f (x) dx = 1
Determine the distribution function F (x)
Draw the graph of F (x)
Determine E[X]
Compute V ar(X) and (X)
Determine P (X 0)
Compute P (X 23 )
Compute P (| X | 23 )

173
EXERCISE : Recall the density function
n
cx (1 xn ) , 0 x 1
fn (x) =
0, otherwise

considered earlier, where n is a positive integer, and where

(n+1)(2n+1)
c = n
.
Determine E[X] .

What happens to E[X] for large n ?

Determine E[X 2 ]

What happens to E[X 2 ] for large n ?

What happens to V ar(X) for large n ?

174
Covariance
Let X and Y be continuous random variables with mean
E[X] = X , E[Y ] = Y .

Then the covariance of X and Y is

Cov(X, Y ) E[ (X X ) (Y Y ) ]
Z Z
= (x X ) (y Y ) fX,Y (x, y) dy dx .

As in the discrete case, we have

Cov(X, Y ) = E[ (X X ) (Y Y ) ]

= E[XY X Y Y X + X Y ]

= E[XY ] E[X] E[Y ] .

175
As in the discrete case, we also have

PROPERTY 1 :
V ar(X + Y ) = V ar(X) + V ar(Y ) + 2 Cov(X, Y ) ,

and

PROPERTY 2 : If X and Y are independent then

Cov(X, Y ) = 0 ,

V ar(X + Y ) = V ar(X) + V ar(Y ) .

NOTE :
The proofs are identical to those for the discrete case !

As in the discrete case, if Cov(X, Y ) = 0 then X and Y are

not necessarily independent!

176
EXAMPLE : For
xy
e for x > 0 and y > 0 ,
fX,Y (x, y) =
0 otherwise ,

we already found
fX (x) = ex , fY (y) = ey ,
so that
fX,Y (x, y) = fX (x) fY (y) ,

i.e., X and Y are independent .

Indeed, we also already found

E[X] = E[Y ] = E[XY ] = 1 ,
so that
Cov(X, Y ) = E[XY ] E[X] E[Y ] = 0 .

177
EXERCISE :

Verify the following properties :

V ar(cX + d) = c2 V ar(X) ,

Cov(X, Y ) = Cov(Y, X) ,

Cov(cX, Y ) = c Cov(X, Y ) ,

Cov(X, cY ) = c Cov(X, Y ) ,

Cov(X + Y, Z) = Cov(X, Z) + Cov(Y, Z) ,

V ar(X + Y ) = V ar(X) + V ar(Y ) + 2 Cov(X, Y ) .

178
EXERCISE :

For the random variables X , Y with joint density function

45xy 2 (1 x)(1 y 2 ) , 0 x 1 , 0 y 1
f (x, y) =
0, otherwise

R1R1
Verify that 0 0
f (x, y) dy dx = 1 .

Determine the marginal density functions fX (x) and fY (y) .

Are X and Y independent ?

What is the value of Cov(X, Y ) ?

179
The joint probability density function fXY (x, y) .

180
Markovs inequality.
For a continuous nonnegative random variable X , and c > 0 ,
we have
E[X]
P (X c) .
c
PROOF :
Z Z c Z
E[X] = xf (x) dx = xf (x) dx + xf (x) dx
0 0 c
Z
xf (x) dx
c
Z
c f (x) dx ( Why ? )
c

= c P (X c) .
EXERCISE :
Show Markovs inequality also holds for discrete random variables.

181
Markovs inequality : For continuous nonnegative X , c > 0 :
E[X]
P (X c) .
c
x
EXAMPLE : For e for x > 0 ,
f (x) =
0 otherwise ,

we have
Z
E[X] = x ex dx = 1 ( already done ! )
0

Markovs inequality gives

E[X] 1
c=1 : P (X 1) = = 1 (!)
1 1

E[X] 1
c = 10 : P (X 10) = = 0.1
10 10

QUESTION : Are these estimates sharp ?

182
QUESTION : Are these estimates sharp ?
Markovs inequality gives
E[X] 1
c=1 : P (X 1) = = 1 (!)
1 1
E[X] 1
c = 10 : P (X 10) = = 0.1
10 10
The actual values are
Z
P (X 1) = ex dx = e1
= 0.37
1
Z
P (X 10) = ex dx = e10
= 0.000045
10

EXERCISE : Suppose the score of students taking an examination

is a random variable with mean 65 .
Give an upper bound on the probability that a students score is
greater than 75 .

183
Chebyshevs inequality: For (practically) any random variable X:
1
P( | X | k ) 2
,
k
p
where = E[X] is the mean, = V ar(X) the standard deviation.

PROOF : Let Y (X )2 , which is nonnegative.

By Markovs inequality
E[Y ]
P (Y c) .
c
Taking c = k2 2 we have

P ( | X | k ) = P ( (X )2 k2 2 ) = P ( Y k2 2 )

E[ Y ] V ar(X) 2 1
= = 2 2 = . QED !
k2 2 k2 2 k k 2

NOTE : This inequality also holds for discrete random variables.

184
EXAMPLE : Suppose the value of the Canadian dollar in terms of
the US dollar over a certain period is a random variable X with

mean = 0.98 and standard deviation = 0.05 .

What can be said of the probability that the Canadian dollar is valued
between $0.88US and $1.08US ,
that is,
between 2 and + 2 ?

SOLUTION : By Chebyshevs inequality we have

1
P( | X | 2 ) 2
= 0.25 .
2
Thus
P ( | X | < 2 ) > 1 0.25 = 0.75 ,
that is,
P ( $0.88US < Can$ < $1.08US ) > 75 % .

185
EXERCISE :
The score of students taking an examination is a random variable
with mean = 65 and standard deviation = 5 .

What is the probability a student scores between 55 and 75 ?

How many students would have to take the examination so that
the probability that their average grade is between 60 and 70
is at least 80% ?
HINT : Defining
1
X = (X1 + X2 + + Xn ) , ( the average grade )
n
we have 1
X = E[X] = n = = 65 ,
n
and, assuming independence,
2 2 25 5
V ar(X) = n 2 = = , and X = .
n n n n

186
 SPECIAL CONTINUOUS RANDOM VARIABLES

The Uniform Random Variable

1
ba , a < x b 0, xa
xa
f (x) = , F (x) = ba
, a<xb
0 , otherwise 1, x>b

f(x) F(x)
1
___ 1
ba

F(x1)

F(x2)

x x
a x1 x2 b a x1 x2 b

(Already introduced earlier for the special case a = 0, b = 1 .)

187
EXERCISE :

Show that the uniform density function

1

ba
, a<xb
f (x) =
0, otherwise

has mean
a+b
= ,
2

and standard deviation

ba
= .
2 3

188
 A joint uniform random variable :
1 (x a)(y c)
f (x, y) = , F (x, y) = ,
(b a)(d c) (b a)(d c)

for x (a, b], y (c, d].

0.9
1.5 0.8
0.7
1.0 0.6
f 0.5
0.5
F 0.4
0.3
0.2

0.1 0.1
0.2 0.1
0.2
0.3 0.3
0.4
y 0.4 0.5 0.3
0.2
0.1
0.1 0.2
0.3 0.4 0.7
0.6
0.5 x
0.5
0.6
0.7 0.5
0.4
x y 0.6 0.7 0.8 0.9
0.9
0.8
0.6
0.8 0.7
0.9 0.8
0.9

Here x [0, 1], y [0, 1] .

189
EXERCISE :

Consider the joint uniform density function

c for x2 + y 2 4 ,
f (x, y) =
0 otherwise .

What is the value of c ?

What is P (X < 0) ?

What is P (X < 0 , Y < 0) ?

What is f ( x | y = 1 ) ?

HINT : No complicated calculations are needed !

190
The Exponential Random Variable
x
e , x>0 1 ex , x > 0
f (x) = , F (x) =
0, x0 0, x0

with
R x 1
E[X] = = 0
x e dx =
( Check ! ) ,

2
R 2
E[X ] = 0
x2 ex dx = 2
( Check ! ) ,

1
V ar(X) = E[X 2 ] 2 = 2
,
p 1
(X) = V ar(X) =
.

NOTE : The two integrals can be done by integration by parts .

EXERCISE : (Done earlier for = 1) :

Also use the Method of Moments to compute E[X] and E[X 2 ] .

191
 2.0 1.0

0.8
1.5

0.6

F (x)
f (x)

1.0

0.4

0.5
0.2

0.0 0.0
0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0
x x

The Exponential density and distribution functions

f (x) = ex , F (x) = 1 ex ,

for = 0.25, 0.50, 0.75, 1.00 (blue), 1.25, 1.50, 1.75, 2.00 (red ).

192
PROPERTY : From

F (x) P (X x) = 1 ex , ( for x > 0 ) ,

we have
P (X > x) = 1 (1 ex ) = ex .

Also, for x > 0 ,

P (X > x + x , X > x)
P (X > x + x | X > x) =
P (X > x)
P (X > x + x) e(x+x)
= = = ex .
P (X > x) ex

CONCLUSION : P ( X > x + x | X > x )

only depends on x (and ), and not on x !

We say that the exponential random variable is memoryless .

193
EXAMPLE :

Let the density function f (t) model failure of a device,

f (t) = et , (taking = 1 ) ,

i.e., the probability of failure in the time-interval (0, t] is

Z t Z t
F (t) = f (t) dt = et dt = 1 et ,
0 0

with

F (0) = 0 , ( the device works at time 0 ).

and
F () = 1 , ( the device must fail at some time ).

194
EXAMPLE : ( continued ) F (t) = 1 et .

Let Et be the event that the device still works at time t :

P (Et ) = 1 F (t) = et .

The probability it still works at time t + 1 is

P (Et+1 ) = 1 F (t + 1) = e(t+1) .

The probability it still works at time t + 1, given it works at time t is

P (Et+1 Et ) P (Et+1 ) e(t+1) 1

P (Et+1 |Et ) = = = = ,
P (Et ) P (Et ) et e

which is independent of t !

QUESTION : Is such an exponential distribution realistic if

the device is your heart, and time t is measured in decades ?

195
The Standard Normal Random Variable

The standard normal random variable has density function

1 12 x2
f (x) = e , < x < ,
2
with mean
Z
= x f (x) dx = 0 , ( Check ! )

Since
Z
E[X 2 ] = x2 f (x) dx = 1 , ( more difficult )

we have

V ar(X) = E[X 2 ] 2 = 1 , and (X) = 1 .

196
 1 12 x2
f (x) = e
2
0.40

0.35

0.30

0.25
f (x)

0.20

0.15

0.10

0.05

0.00
4 3 2 1 0 1 2 3 4
x
The standard normal density function f (x) .

197
 x
1
Z
12 x2
(x) = F (x) = e dx
2

1.0

0.9

0.8

0.7

0.6
F (x)

0.5

0.4

0.3

0.2

0.1

0.0
4 3 2 1 0 1 2 3 4
x
The standard normal distribution function F (x)
( often denoted by (x) ) .

198
 The Standard Normal Distribution (z)
z (z) z (z)
0.0 .5000 -1.2 .1151
-0.1 .4602 -1.4 .0808
-0.2 .4207 -1.6 .0548
-0.3 .3821 -1.8 .0359
-0.4 .3446 -2.0 .0228
-0.5 .3085 -2.2 .0139
-0.6 .2743 -2.4 .0082
-0.7 .2420 -2.6 .0047
-0.8 .2119 -2.8 .0026
-0.9 .1841 -3.0 .0013
-1.0 .1587 -3.2 .0007

( For example, P (Z 2.0) = (2.0) = 2.28% )

QUESTION : How to get the values of (z) for positive z ?

199
EXERCISE :

Suppose the random variable X has the standard normal distribution.

What are the values of

P ( X 0.5 )

P ( X 0.5 )

P ( | X | 0.5 )

P ( | X | 0.5 )

P( 1 X 1 )

P ( 1 X 0.5 )

200
The General Normal Random Variable

The general normal density function is

1 21 (x)2 / 2
f (x) = e
2

where, not surprisingly,

E[X] = ( Why ? )

One can also show that

V ar(X) E[(X )2 ] = 2 ,

so that is in fact the standard deviation .

201
The standard normal (black) and the normal density functions
with = 1, = 0.5 (red ) and = 1.5, = 2.5 (blue).

202
To compute the mean of the general normal density function
1 21 (x)2 / 2
f (x) = e
2
consider
Z
E[X ] = (x ) f (x) dx

1
Z
12 (x)2 / 2
= (x ) e dx
2

2 1 2 2

= e 2 (x) /  = 0.

2

Thus the mean is indeed

E[X] = .

203
NOTE : If X is general normal we have the very useful formula :
X
P( c ) = (c) ,

i.e., we can use the Table of the standard normal distribution !

PROOF : For any constant c we have

+c
X 1
Z
12 (x)2 / 2
P( c) = P (X +c) = e dx .
2

Let y (x )/ , so that x = + y .

Then the new variable y ranges from to c , and

(x )2 / 2 = y 2 , dx = dy ,
so that c
X 1
Z
12 y 2
P( c) = e dy = (c) .
2
( the standard normal distribution )

204
EXERCISE : Suppose X is normally distributed with

mean = 1.5 and standard deviation = 2.5 .

Use the standard normal Table to determine :

P ( X 0.5 )

P ( X 0.5 )

P ( | X | 0.5 )

P ( | X | 0.5 )

205
The Chi-Square Random Variable

Suppose X1 , X2 , , Xn ,
are independent standard normal random variables.

Then 2n X12 + X22 + + Xn2 ,

is called the chi-square random variable with n degrees of freedom.

We will show that

E[2n ] = n , V ar(2n ) = 2n , (2n ) = 2n .

NOTE :
2
The in 2n is part of its name , while 2
in X12 , etc. is power 2 !

206
 0.5 1.0

0.4 0.8

0.3 0.6

F (x)
f (x)

0.2 0.4

0.1 0.2

0.0 0.0
0 2 4 6 8 10 0 2 4 6 8 10
x x

The Chi-Square density and distribution functions for n = 1, 2, , 10.

( In the Figure for F , the value of n increases from left to right. )

207
If n = 1 then
21 X12 , where X X1 is standard normal .
We can compute the moment generating function of 21 :
Z
t21 tX 2 1 tx2 12 x2
E[e ] = E[e ] = e e dx
2

1
Z
12 x2 (12t)
= e dx
2

Let
1 1
1 2t = 2 , or equivalently , .
1 2t
Then

1 1
Z
t21 12 x2 / 2
E[e ] = e dx = = .
2 1 2t
(integral of a normal density function)

208
Thus we have found that :

The moment generating function of 21 is

t21 1
(t) E[e ] = ,
1 2t

with which we can compute

E[21 ] = (0) = 1, ( Check ! )

E[(21 )2 ] = (0) = 3, ( Check ! )

V ar(21 ) = E[(21 )2 ] E[21 ]2 = 2.

209
We found that

E[21 ] = 1 , V ar(21 ) = 2 .

In the general case where

2n X12 + X22 + + Xn2 ,
we have

E[2n ] = E[X12 ] + E[X22 ] + + E[Xn2 ] = n ,

and since the Xi are assumed independent ,

V ar[2n ] = V ar[X12 ] + V ar[X22 ] + + V ar[Xn2 ] = 2n ,

and
(2n ) = 2n .

210
 0.16

0.14

0.12

0.10
f (x)

0.08

0.06

0.04

0.02

0.00
0 5 10 15 20 25
x
The Chi-Square density functions for n = 5, 6, , 15 .
( For large n they look like normal density functions ! )

211
 The 2n - Table
n = 0.975 = 0.95 = 0.05 = 0.025
5 0.83 1.15 11.07 12.83
6 1.24 1.64 12.59 14.45
7 1.69 2.17 14.07 16.01
8 2.18 2.73 15.51 17.54
9 2.70 3.33 16.92 19.02
10 3.25 3.94 18.31 20.48
11 3.82 4.58 19.68 21.92
12 4.40 5.23 21.03 23.34
13 5.01 5.89 22.36 24.74
14 5.63 6.57 23.69 26.12
15 6.26 7.26 25.00 27.49

This Table shows z,n values such that P (2n z,n ) = .

( For example, P (210 3.94) = 95% )
 THE CENTRAL LIMIT THEOREM

The density function of the Chi-Square random variable

2n X1 + X2 + + Xn ,
where

Xi = Xi2 , and Xi is standard normal, i = 1, 2, , n ,

starts looking like a normal density function when n gets large.

This remarkable fact holds much more generally !

It is known as the Central Limit Theorem (CLT).

212
RECALL :

If X1 , X2 , , Xn are independent, identically distributed ,

each having

mean , variance 2 , standard deviation ,

then
S X1 + X2 + + Xn ,
has
mean : S E[S] = n ( Why ? )

variance : V ar(S) = n 2 ( Why ? )

Standard deviation : S = n

NOTE : S gets bigger as n increases ( and so does | S | ).

213
THEOREM (The Central Limit Theorem) (CLT) :

Let X1 , X2 , , Xn be identical, independent random variables,

each having

mean , variance 2 , standard deviation .

Then for large n the random variable

S X1 + X2 + + Xn ,
2

( which has mean n , variance n , standard deviation n)

is approximately normal .

S n
NOTE : Thus is approximately standard normal .
n

214
EXAMPLE : Recall that
2n X12 + X22 + + Xn2 ,

where each Xi is standard normal, and (using moments) we found

2

n has mean n and standard deviation 2n .

The Table below illustrates the accuracy of the approximation

r
2 0n n
P ( n 0 ) = ( ) = ( ).
2n 2
pn pn
n 2 ( 2 )
2 1 0.1587
8 2 0.0228
18 3 0.0013

QUESTION : What is the exact value of P (2n 0) ? (!)

215
EXERCISE :

Use the approximation

xn
P( 2n x) = ( ),
2n

to compute approximate values of

P ( 232 24 )

P ( 232 40 )

P ( | 232 32 | 8 )

216
RECALL :

If X1 , X2 , , Xn are independent, identically distributed ,

each having

mean , variance 2 , standard deviation ,

then
1
X (X1 + X2 + + Xn ) ,
n
has
mean : X = E[X] = ( Why ? )

2 1
variance : X = n2
n 2 = 2 /n ( Why ? )

Standard deviation : X = / n

NOTE : X gets smaller as n increases.

217
COROLLARY (to the Central Limit Theorem) :

If X1 , X2 , , Xn be identical, independent random variables,

each having

mean , variance 2 , standard deviation ,

then for large n the random variable

1
X (X1 + X2 + + Xn ) ,
n
2
( which has mean , variance , standard deviation )
n n
is approximately normal .

X
NOTE : Thus is approximately standard normal .
/ n

218
EXAMPLE : Suppose X1 , X2 , , Xn are
identical , independent , uniform random variables ,
each having density function
1
f (x) = , for x [1, 1] , ( 0 otherwise ) ,
2
with 1
mean = 0 , standard deviation = ( Check ! )
3

Then for large n the random variable

1
X (X1 + X2 + + Xn ) ,
n
with
1
mean = 0 , standard deviation = ,
3n
is approximately normal , so that
x0

P (X x) = ( ) = ( 3n x ) .
1/ 3n

219
EXERCISE : In the preceding example
x0
P (X x)
= ( ) ( 3n x ) .
1/ 3n

Fill in the Table to illustrate the accuracy of this approximation :

n
P (X 1) = ( 3n )
3
12

( What is the exact value of P ( X 1 ) ? ! )

For n = 12 find the approximate value of P ( X 0.1 ) .

For n = 12 find the approximate value of P ( X 0.5 ) .

220
EXPERIMENT : ( a lengthy one ! )

We give a detailed computational example to illustrate :

The concept of density function .

The numerical construction of a density function

and (most importantly)

The Central Limit Theorem .

221
EXPERIMENT : ( continued )

Generate N uniformly distributed random numbers in [0, 1] .

Many programming languages have a function for this.

Call the random number values generated xi , i = 1, 2, , N .

Letting xi = 2xi 1 gives uniform random values in [1, 1] .

222
EXPERIMENT : ( continued )

-0.737 0.511 -0.083 0.066 -0.562 -0.906 0.358 0.359

0.869 -0.233 0.039 0.662 -0.931 -0.893 0.059 0.342
-0.985 -0.233 -0.866 -0.165 0.374 0.178 0.861 0.692
0.054 -0.816 0.308 -0.168 0.402 0.821 0.524 -0.475
-0.905 0.472 -0.344 0.265 0.513 0.982 -0.269 -0.506
0.965 0.445 0.507 0.303 -0.855 0.263 0.769 -0.455
-0.127 0.533 -0.045 -0.524 -0.450 -0.281 -0.667 -0.027
0.795 0.818 -0.879 0.809 0.009 0.033 -0.362 0.973
-0.012 -0.468 -0.819 0.896 -0.853 0.001 -0.232 -0.446
0.828 0.059 -0.071 0.882 -0.900 0.523 0.540 0.656
-0.749 -0.968 0.377 0.736 0.259 0.472 0.451 0.999
0.777 -0.534 -0.387 -0.298 0.027 0.182 0.692 -0.176
0.683 -0.461 -0.169 0.075 -0.064 -0.426 -0.643 -0.693
0.143 0.605 -0.934 0.069 -0.003 0.911 0.497 0.109
0.781 0.250 0.684 -0.680 -0.574 0.429 -0.739 -0.818
120 values of a uniform random variable in [1, 1] .

223
EXPERIMENT : ( continued )
2
Divide [1, 1] into M subintervals of equal size x = M
.

Let Ik denote the kth interval, with midpoint xk .

Let mk be the frequency count (# of random numbers in Ik ) .

mk
Let f (xk ) = N x
, (N is the total # of random numbers) .

Then 1 f (x) dx
R1 PM PM mk
= k=1 f (xk ) x = k=1 N x x = 1 ,
and f (xk ) approximates the value of the density function .

The corresponding distribution function is

Z x
f (x) dx
X
F (x ) = = f (xk ) x .
1 k=1

224
EXPERIMENT : ( continued )
Interval Frequency Sum f (x) F (x)
1 50013 50013 0.500 0.067
2 50033 100046 0.500 0.133
3 50104 150150 0.501 0.200
4 49894 200044 0.499 0.267
5 50242 250286 0.502 0.334
6 49483 299769 0.495 0.400
7 50016 349785 0.500 0.466
8 50241 400026 0.502 0.533
9 50261 450287 0.503 0.600
10 49818 500105 0.498 0.667
11 49814 549919 0.498 0.733
12 50224 600143 0.502 0.800
13 49971 650114 0.500 0.867
14 49873 699987 0.499 0.933
15 50013 750000 0.500 1.000
Frequency Table, showing the count per interval .
(N = 750, 000 random numbers, M = 15 intervals)

225
EXPERIMENT : ( continued )

1.0

0.8

0.6
f (x)

0.4

0.2

0.0
1.0 0.5 0.0 0.5 1.0
x
The approximate density function , f (xk ) = Nmx
k
for
N = 5, 000, 000 random numbers, and M = 200 intervals.

226
 EXPERIMENT : ( continued )

1.0 1.0

0.9

0.8 0.8

0.7

0.6 0.6

F (x)
f (x)

0.5

0.4 0.4

0.3

0.2 0.2

0.1

0.0 0.0
1.0 0.5 0.0 0.5 1.0 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00
x x

Approximate density function f (x) and distribution function F (x) ,

for the case N = 5, 000, 000 random numbers, and M = 200 intervals.

NOTE : F (x) appears smoother than f (x) . ( Why ? )

227
EXPERIMENT : ( continued )

Next ( still for the uniform random variable in [1, 1] ) :

Generate n random numbers ( n relatively small ) .

Take the average of the n random numbers.

Do the above N times, where (as before) N is very large .

Thus we deal with a random variable

1
X (X1 + X2 + + Xn ) .
n

228
EXPERIMENT : ( continued )

-0.047 0.126 -0.037 0.148 -0.130 -0.004 -0.174 0.191

0.198 0.073 -0.025 -0.070 -0.018 -0.031 0.063 -0.064
-0.197 -0.026 -0.062 -0.004 -0.083 -0.031 -0.102 -0.033
-0.164 0.265 -0.274 0.188 -0.067 0.049 -0.090 0.002
0.118 0.088 -0.071 0.067 -0.134 -0.100 0.132 0.242
-0.005 -0.011 -0.018 -0.048 -0.153 0.016 0.086 -0.179
-0.011 -0.058 0.198 -0.002 0.138 -0.044 -0.094 0.078
-0.011 -0.093 0.117 -0.156 -0.246 0.071 0.166 0.142
0.103 -0.045 -0.131 -0.100 0.072 0.034 0.176 0.108
0.108 0.141 -0.009 0.140 0.025 -0.149 0.121 -0.120
0.012 0.002 -0.015 0.106 0.030 -0.096 -0.024 -0.111
-0.147 0.004 0.084 0.047 -0.048 0.018 -0.183 0.069
-0.236 -0.217 0.061 0.092 -0.003 0.005 -0.054 0.025
-0.110 -0.094 -0.115 0.052 0.135 -0.076 -0.018 -0.121
-0.030 -0.146 -0.155 0.089 -0.177 0.027 -0.025 0.020

Values of X for the case N = 120 and n = 25 .

229
EXPERIMENT : ( continued )

For sample size n, (n = 1, 2, 5, 10, 25) , and M = 200 intervals :

Generate N values of X , where N is very large .

Let mk be the number of values of X in Ik .

mk
As before, let fn (xk ) = N x
.

Now fn (xk ) approximates the density function of X .

230
EXPERIMENT : ( continued )
Interval Frequency Sum f (x) F (x)
1 0 0 0.00000 0.00000
2 0 0 0.00000 0.00000
3 0 0 0.00000 0.00000
4 11 11 0.00011 0.00001
5 1283 1294 0.01283 0.00173
6 29982 31276 0.29982 0.04170
7 181209 212485 1.81209 0.28331
8 325314 537799 3.25314 0.71707
9 181273 719072 1.81273 0.95876
10 29620 748692 0.29620 0.99826
11 1294 749986 0.01294 0.99998
12 14 750000 0.00014 1.00000
13 0 750000 0.00000 1.00000
14 0 750000 0.00000 1.00000
15 0 750000 0.00000 1.00000
Frequency Table for X, showing the count per interval .
(N = 750, 000 values of X, M = 15 intervals, sample size n = 25)

231
 EXPERIMENT : ( continued )
3.5 1.0

0.9
3.0
0.8

2.5 0.7

0.6
2.0

F (x)
f (x)

0.5

1.5
0.4

1.0 0.3

0.2
0.5
0.1

0.0 0.0
1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.0
x x

The approximate density functions fn (x) , n = 1, 2, 5, 10, 25 ,

and the corresponding distribution functions Fn (x) .
( N = 5, 000, 000 values of X , M = 200 intervals )

232
EXPERIMENT : ( continued )
Recall that for uniform random variables Xi on [1, 1]
1
X (X1 + X2 + + Xn ) ,
n
is approximately normal , with
1
mean = 0 , standard deviation = .
3n
Thus for each n we can normalize x and fn (x) :
x x0 fn (x)
x = = = 3n x , n (x) =
f .
1 3n
3n

The next Figure shows :

The normalized fn (x) approach a limit as n get large.

This limit is the standard normal density function.

Thus our computations agree with the Central Limit Theorem !

233
 0.40

0.35

0.30

0.25
fm(x)

0.20

0.15

0.10

0.05

0.00
4 3 2 1 0 1 2 3 4
x
The normalized density functions fn (x) , for n = 1, 2, 5, 10, 25 .
( N = 5, 000, 000 values of X , M = 200 intervals )

234
EXERCISE : Suppose
X1 , X2 , , X12 , (n = 12) ,
are identical, independent, uniform random variables on [0, 1] .

We already know that each Xi has

1 1
mean = 2
, standard deviation
2 3
.
Let
1
X (X1 + X2 + + X12 ) .
12

Use the CLT to compute approximate values of

1
P (X 3
)

2
P (X 3
)

1 1
P (| X 2
| 3
)

235
EXERCISE : Suppose
X1 , X2 , , X9 , (n = 9) ,
are identical, independent, exponential random variables, with
f (x) = ex , where = 1 .

We already know that each Xi has

1 1
mean =
=1, and standard deviation
=1.
Let 1
X (X1 + X2 + + X9 ) .
9
Use the CLT to compute approximate values of

P (X 0.4)

P (X 1.6)

P (| X 1 | 0.6)

236
EXERCISE : Suppose

X1 , X2 , , Xn ,

are identical, independent, normal random variables, with

mean = 7 , standard deviation 4 .

Let
1
X (X1 + X2 + + Xn ) .
n

Use the CLT to determine at least how big n must be so that

P (| X | 1) 90 % .

237
EXAMPLE : The CLT also applies to discrete random variables .
The Binomial random variable , with
 
n
P (X = k) = pk (1 p)nk , (0 k n) ,
k
is already a sum (namely, of Bernoulli random variables).
Thus its binomial probability mass function already looks normal :
0.09

0.08
0.25
0.07

0.20 0.06

0.05
p(x)

p(x)
0.15
0.04

0.10 0.03

0.02
0.05
0.01

0.00 0.00
0 2 4 6 8 10 0 10 20 30 40 50 60 70 80 90 100
x x
Binomial : n = 10 , p = 0.3 Binomial : n = 100 , p = 0.3

238
EXAMPLE : ( continued )

We already know that if X is binomial then

p
(X) = np and (X) = np(1 p) .
Thus, for n = 100 , p = 0.3 , we have

(X) = 30 and (X) = 21
= 4.58 .

Using the CLT we can approximate

26 30
P (X 26) = ( ) = (0.87)
= 19.2 % .
4.58
The exact binomial value is
26  
X n
P (X 26) = pk (1 p)nk = 22.4 % ,
k
k=0

QUESTION : What do you say ?

239
EXAMPLE : ( continued )

We found the exact binomial value

P (X 26) = 22.4 % ,
and the CLT approximation
26 30
P (X 26) = ( ) = (0.87) = 19.2 % .
4.58
It is better to

spread P (X = 26) over the interval [25.5 , 26.5] . ( Why ? )

Thus it is better to adjust the approximation to P (X 26) by

26.5 30
P (X 26) = ( ) = (0.764) = 22.2 % .
4.58

QUESTION : What do you say now ?

240
EXERCISE :
1
Consider the Binomial distribution with n = 676 and p = 26
:

0.08

0.07

0.06

0.05
p(x)

0.04

0.03

0.02

0.01

0.00
0 10 20 30 40 50
x
1
The Binomial (n = 676, p = 26
), shown in [0, 50] .

241
 1
EXERCISE : (continued ) (Binomial : n = 676 , p = 26
)

Write down the Binomial formula for P (X = 24) .

Evaluate P (X = 24) using the Binomial recurrence formula .
p
Compute E[X] = np and (X) = np(1 p) .

The Poisson probability mass function

k
P (X = k) = e , (with = np ) ,
k!
approximates the Binomial when p is small and n large.
Evaluate P (X = 24) using the Poisson recurrence formula .

Compute the standard normal approximation to P (X = 24).

ANSWERS : 7.61 % , 7.50 % , 7.36 % .

242
EXPERIMENT :

Compare the accuracy of the Poisson and the adjusted Normal

approximations to the Binomial, for different values of n .

k n Binomial Poisson Normal

2 4 0.6875 0.6767 0.6915
4 8 0.6367 0.6288 0.6382
8 16 0.5982 0.5925 0.5987
16 32 0.5700 0.5660 0.5702
32 64 0.5497 0.5468 0.5497
64 128 0.5352 0.5332 0.5352
P (X k) , where k = np, with p = 0.5 .

Any conclusions ?

243
EXPERIMENT : ( continued )

Compare the accuracy of the Poisson and the adjusted Normal

approximations to the Binomial, for different values of n .

k n Binomial Poisson Normal

0 4 0.6561 0.6703 0.5662
0 8 0.4305 0.4493 0.3618
1 16 0.5147 0.5249 0.4668
3 32 0.6003 0.6025 0.5702
6 64 0.5390 0.5423 0.5166
12 128 0.4805 0.4853 0.4648
25 256 0.5028 0.5053 0.4917
51 512 0.5254 0.5260 0.5176
P (X k) , where k = np, with p = 0.1 .

Any conclusions ?

244
EXPERIMENT : ( continued )
Compare the accuracy of the Poisson and the adjusted Normal
approximations to the Binomial, for different values of n .
k n Binomial Poisson Normal
0 4 0.9606 0.9608 0.9896
0 8 0.9227 0.9231 0.9322
0 16 0.8515 0.8521 0.8035
0 32 0.7250 0.7261 0.6254
0 64 0.5256 0.5273 0.4302
1 128 0.6334 0.6339 0.5775
2 256 0.5278 0.5285 0.4850
5 512 0.5948 0.5949 0.5670
10 1024 0.5529 0.5530 0.5325
20 2048 0.5163 0.5165 0.5018
40 4096 0.4814 0.4817 0.4712
P (X k) , where k = np, with p = 0.01 .

Any conclusions ?

245
 SAMPLE STATISTICS

Sampling can consist of

Gathering random data from a large population, for example,
measuring the height of randomly selected adults
measuring the starting salary of random CS graduates

Recording the results of experiments , for example,

measuring the breaking strength of randomly selected bolts
measuring the lifetime of randomly selected light bulbs

We shall generally assume the population is infinite (or large) .

We shall also generally assume the observations are independent .
The outcome of any experiment does not affect other experiments.

246
DEFINITIONS :

A random sample from a population consists of

independent , identically distributed random variables,

X1 , X2 , , Xn .

The values of the Xi are called the outcomes of the experiment.

A statistic is a function of X1 , X2 , , Xn .

Thus a statistic itself is a random variable .

247
EXAMPLES :

The most important statistics are

The sample mean

1
X (X1 + X2 + + Xn ) .
n

The sample variance

n
1 X
S2 (Xk X)2 .
n k=1

( to be discussed in detail )

The sample standard deviation S = S2 .

248
For a random sample

X1 , X2 , , Xn ,

one can think of many other statistics such as :

The order statistic in which the observation are ordered in size .

The sample median , which is

the midvalue of the order statistic (if n is odd),

the average of the two middle values (if n is even).

The sample range : the difference between the largest and the
smallest observation.

249
EXAMPLE : For the 8 observations

0.737 , 0.511 , 0.083 , 0.066 , 0.562 , 0.906 , 0.358 , 0.359 ,

from the first row of the Table given earlier, we have

Sample mean :
1
X = ( 0.737 + 0.511 0.083 + 0.066
8
0.562 0.906 + 0.358 + 0.359 ) = 0.124 .

Sample variance :
1
{ (0.737 X)2 + (0.511 X)2 + (0.083 X)2
8
+ (0.066 X)2 + (0.562 X)2 + (0.906 X)2
+ (0.358 X)2 + (0.359 X)2 } = 0.26 .

Sample standard deviation : 0.26 = 0.51 .

250
EXAMPLE : ( continued )

For the 8 observations

0.737 , 0.511 , 0.083 , 0.066 , 0.562 , 0.906 , 0.358 , 0.359 ,

we also have

The order statistic :

0.906 , 0.737 , 0.562 , 0.083 , 0.066 , 0.358 , 0.359 , 0.511 .

The sample median : (0.083 + 0.066)/2 = 0.0085 .

The sample range : 0.511 (0.906) = 1.417 .

251
The Sample Mean
Suppose the population mean and standard deviation are and .

As before, the sample mean

1
X (X1 + X2 + + Xn ) ,
n
is also a random variable , with expected value
1
X E[X] = E[ (X1 + X2 + + Xn ) ] = ,
n
and variance
2 2
X V ar(X) = ,
n

Standard deviation of X : X = .
n

NOTE : The sample mean approximates the population mean .

252
How well does the sample mean approximate the population mean ?

From the Corollary to the CLT we know

X
,
/ n

is approximately standard normal when n is large.

Thus, for given n and z , (z > 0) , we can, for example, estimate

X
P( | | z)
= 1 2 (z) .
/ n

(A problem is that we often dont know the value of )

253
It follows that

 X   z 
P | | z = P | X |
/ n n

 z z 
= P [ X , X + ]
n n

= 1 2 (z) ,

which gives us a confidence interval estimate of .

254
 z z
We found : P ( [X , X + ] )
= 1 2 (z) .
n n

EXAMPLE : We take samples from a given population :

The population mean is unknown .
The population standard deviation is = 3
The sample size is n = 25 .
The sample mean is X = 4.5 .

Taking z=2 , we have

32 32
P ( [ 4.5 , 4.5 + ] ) = P ( [ 3.3 , 5.7 ] )
25 25
= 1 2 (2)
= 95 % .

We call [ 3.3 , 5.7 ] the 95 % confidence interval estimate of .

255
EXERCISE :
As in the preceding example, is unknown, = 3 , X = 4.5 .
Use the formula
z z
P ( [X , X + ])
= 1 2 (z) ,
n n
to determine
The 50 % confidence interval estimate of when n = 25 .
The 50 % confidence interval estimate of when n = 100 .
The 95 % confidence interval estimate of when n = 100 .

NOTE : In the Standard Normal Table , check that

The 50 % confidence interval corresponds to z = 0.68
= 0.7 .
The 95 % confidence interval corresponds to z = 1.96
= 2.0 .

256
The Sample Variance We defined the sample variance as
n n
2 1 X X 1
S (Xk X)2 = 2
[ (Xk X) ].
n k=1 k=1
n

Earlier, for discrete random variables X, we defined the variance as

X
2 2
E[ (X ) ] [ (Xk )2 p(Xk ) ] .
k

These two formulas look deceptively similar !

In fact, they are quite different !
The 1st sum for S 2 is only over the sampled X-values.
The 2nd sum for 2 is over all X-values.
The 1st sum for S 2 has constant weights .
The 2nd sum for 2 uses the probabilities as weights .

257
We have just argued that the sample variance
n
1 X
S2 (Xk X)2 ,
n k=1

and the population variance (for discrete random variables)

X
2 2
E[ (X ) ] [ (Xk )2 p(Xk ) ] ,
k

are quite different.

Nevertheless, we will show that for large n their values are close !

Thus for large n we have the approximation

S2
= 2 .

258
FACT 1 : We (obviously) have that
n n
1 X X
X = Xk implies Xk = nX .
n k=1 k=1

FACT 2 : From
2 V ar(X) E[(X )2 ] = E[X 2 ] 2 ,
we (obviously) have
E[X 2 ] = 2 + 2 .

FACT 3 : Recall that for independent, identically distributed Xk ,

where each Xk has mean and variance 2 , we have

2
2
X E[X] = , X E[(X )2 ] = .
n

259
FACT 4 : ( Useful for computing S 2 efficiently ) :
n n
1 X 1 X 2
S2 (Xk X)2 = [ Xk ] X 2 .
n k=1 n k=1
PROOF :
n
1 X
S2 = (Xk X)2
n k=1

n
1 X 2
= (Xk 2Xk X + X 2 )
n k=1

n n
1 X 2 X
= [ Xk 2X Xk + nX 2 ] ( now use Fact 1 )
n k=1 k=1

n n
1 X 2 1 X 2
= [ Xk 2nX 2 + nX 2 ] = [ Xk ] X 2 QED !
n k=1 n k=1

260
THEOREM : The sample variance
n
1 X
S2 (Xk X)2
n k=1
has expected value
1
E[S ] = (1 ) 2 .
2

PROOF : n
n
2 1X
E[S ] = E[ (Xk X)2 ]
n k=1
h 1X n i
= E [Xk2 ] X 2 ( using Fact 4 )
n k=1
n
1X
= E[Xk2 ] E[X 2 ]
n k=1
= 2 + 2 (X
2
+ 2X ) ( using Fact 2 n + 1 times ! )
2
2 2 2 1 2
= + ( + ) = (1 ) . ( Fact 3 ) QED !
n n
REMARK : Thus limn E[S 2 ] = 2 .

261
Most authors instead define the sample variance as
n
1 X
S 2 (Xk X)2 .
n 1 k=1

In this case the Theorem becomes :

THEOREM : The sample variance

n
1 X
S 2 (Xk X)2
n 1 k=1

has expected value

E[S 2 ] = 2 .

EXERCISE : Check this !

262
EXAMPLE : The random sample of 120 values of a uniform
random variable on [1, 1] in an earlier Table has
120
1 X
X = Xk = 0.030 ,
120 k=1
120
1 X
S2 = (Xk X)2 = 0.335 ,
120 k=1

S = S 2 = 0.579 ,

while
= 0,
Z 1
2 2 1 1
= (x ) dx = ,
1 2 3
1
= 2
= = 0.577 .
3
What do you say ?

263
EXAMPLE :

Generate 50 uniform random numbers in [1, 1] .

Compute their average.

Do the above 500 times.

Call the results Xk , k = 1, 2, , 500 .

Thus each Xk is the average of 50 random numbers.

Compute the sample statistics X and S of these 500 values.

Can you predict the values of X and S ?

264
EXAMPLE : ( continued )
500
1 X
Results : X = Xk = 0.00136 ,
500 k=1

500
1 X
S2 = (Xk X)2 = 0.00664 ,
500 k=1

S = S 2 = 0.08152 .

EXERCISE :
What is the value of E[X] ?
Compare X to E[X] .
What is the value of V ar(X) ?
Compare S 2 to V ar(X) .

265
Estimating the variance of a normal distribution
We have shown that
n
1 X
S2 (Xk X)2
= 2 .
n k=1
How good is this approximation for normal random variables Xk ?
To answer this we need :
FACT 5 :
X n n
X
(Xk )2 (Xk X)2 = n(X )2 .
k=1 k=1

PROOF :
Pn 2 2 2 2
LHS = k=1 { X k 2X k + X k + 2X k X X }

= 2nX + n2 + 2nX 2 nX 2

= nX 2 2nX + n2 = RHS . QED !

266
Rewrite Fact 5
X n n
X
(Xk )2 (Xk X)2 = n(X )2 ,
k=1 k=1
as n  n
X Xk 2 n 1 X 2
 X 2
2
(X k X) = ,
k=1
n k=1 / n

and then as n
X n 2
Zk2 2
S = Z 2
,
k=1

where
S 2 is the sample variance ,
and
Z and Zk are standard normal because the Xk are normal .

Finally, we can write the above as

n 2 2 2
2
S = n 1 . ( Why ? )

267
We have found that
n 2 2 2
2
S = n 1 .

THEOREM : For samples from a normal distribution :

n 2 2
2
S has the n1 distribution !

PROOF : Omitted (and not as obvious as it might appear !) .

REMARK : If we use the alternate definition

n
1 X
S 2 (Xk X)2 ,
n 1 k=1
then the Theorem becomes
n1 2 2
2
S has the n1 distribution .

268
 n1 2 2
For normal random variables : 2
S has the n1 distribution

EXAMPLE : For a large shipment of light bulbs we know that :

The lifetime of the bulbs has a normal distribution .

The standard deviation is claimed to be = 100 hours.
( The mean lifetime is not given. )

Suppose we test the lifetime of 16 bulbs. What is the probability

that the sample standard deviation S satisfies S 129 hours ?

SOLUTION : n 1 15 
P (S 129) = P (S 2 1292 ) = P S 2
129 2
2 1002

= P (215 24.96)
= 5 % ( from the 2 Table ) .

QUESTION : If S = 129 then would you believe that = 100 ?

269
 0.16

0.14

0.12

0.10
f (x)

0.08

0.06

0.04

0.02

0.00
0 5 10 15 20 25
x
The Chi-Square density functions for n = 5, 6, , 15 .
(For large n they look like normal density functions .)

270
EXERCISE :
In the preceding example, also compute
P ( 215 24.96 )
using the standard normal approximation .

EXERCISE :
Consider the same shipment of light bulbs :

The lifetime of the bulbs has a normal distribution .

The mean lifetime is not given.

The standard deviation is claimed to be = 100 hours.

Suppose we test the lifetime of only 6 bulbs .

For what value of s is P (S s) = 5 % ?

271
EXAMPLE : For the data below from a normal population :

Estimate the population standard deviation.

Determine a 95 percent confidence interval for .

-0.047 0.126 -0.037 0.148

0.198 0.073 -0.025 -0.070
-0.197 -0.026 -0.062 -0.004
-0.164 0.265 -0.274 0.188

SOLUTION : We find ( with n = 16 ) that

n
1 X
X = Xi = 0.00575 ,
n 1
and n
1 X
S 2 = (Xk X)2 = 0.02278 .
n 1 k=1

272
SOLUTION : We have n = 16 , X = 0.00575 , S 2 = 0.02278 .

Estimate the population standard deviation :

ANSWER : = S = 0.02278 = 0.15095 .

Compute a 95 percent confidence interval for :

ANSWER : From the Chi-Square Table :
P (215 6.26) = 0.025 , P (215 > 27.49) = 0.025 .

(n 1) S 2 2 (n 1)S 2 15 0.02278
2
= 6.26 = = = 0.05458
6.26 6.26
(n 1) S 2 2 (n 1)S 2 15 0.02278
= 27.49 = = = 0.01223
2 27.49 27.49

Thus the 95 % confidence interval for is

[ 0.01223 , 0.05458 ] = [ 0.106 , 0.234 ] .

273
Samples from Finite Populations

Samples from a finite population can be taken

(1) with replacement

(2) without replacement

In Case 1 the sample

X1 , X2 , , Xn ,
may contain the same outcome more than once.

In Case 2 the outcomes are distinct .

Case 2 arises, e.g., when the experiment destroys the sample.

274
EXAMPLE :

Suppose a bag contains three balls, numbered 1, 2, and 3.

A sample of two balls is drawn at random from the bag.

Recall that ( here with n = 2 ) :

1
X (X1 + X2 + + Xn ) .
n
n
1 X
S2 (Xk X)2 .
n k=1

For both, sampling with and without replacement , compute

E[X] and E[S 2 ] .

275
 With replacement : The possible samples are
(1, 1) , (1, 2) , (1, 3) , (2, 1) , (2, 2) , (2, 3) , (3, 1) , (3, 2) , (3, 3) ,
1
each with equal probability 9
.

The sample means X are

3 3 5 5
1 , , 2 , , 2 , , 2 , , 3,
2 2 2 2
with
1 3 3 5 5
E[X] = (1 + + 2 + + 2 + + 2 + + 3) = 2 .
9 2 2 2 2

The sample variances S 2 are

1 1 1 1
0 , , 1 , , 0 , , 1 , , 0. ( Check ! )
4 4 4 4
with
2 1 1 1 1 1 1
E[S ] = (0 + + 1 + + 0 + + 1 + + 0) = .
9 4 4 4 4 3

276
 Without replacement : The possible samples are
(1, 2) , (1, 3) , (2, 1) , (2, 3) , (3, 1) , (3, 2) ,
1
each with equal probability 6
.
The sample means X are
3 3 5 5
, 2 , , , 2 , ,
2 2 2 2
with expected value
1 3 3 5 5
E[X] = ( + 2 + + + 2 + ) = 2.
6 2 2 2 2
The sample variances S 2 are
1 1 1 1
, 1 , , , 1 , . ( Check ! )
4 4 4 4
with expected value
2 1 1 1 1 1 1
E[S ] = ( + 1 + + + 1 + ) = .
6 4 4 4 4 2

277
EXAMPLE : ( continued )

A bag contains three balls, numbered 1, 2, and 3.

A sample of two balls is drawn at random from the bag.

We have computed E[X] and E[S 2 ] :

1
With replacement : E[X] = 2 , E[S 2 ] = 3
,
1
Without replacement : E[X] = 2 , E[S 2 ] = 2
.

We also know the population mean and variance :

1 1 1
= 1 + 2 + 3 = 2,
3 3 3
2 2 1 2 1 2 1 2
= (1 2) + (2 2) + (3 2) = .
3 3 3 3

278
EXAMPLE : ( continued )

We have computed :
2
Population statistics : = 2 , 2 = 3
,

Sampling with replacement : E[X] = 2 , E[S 2 ] = 1

3
,
1
Sampling without replacement : E[X] = 2 , E[S 2 ] = 2
.

According to the earlier Theorem

1 2
E[S 2 ] = (1 ) .
n
In this example the sample size is n = 2 , thus

2 1 2 1
E[S ] = (1 ) = .
2 3

NOTE : E[S 2 ] is wrong for sampling without replacement !

279
QUESTION :
Why is E[S 2 ] wrong for sampling without replacement ?

ANSWER : Without replacement the outcomes Xk of a sample

X1 , X2 , , Xn ,
are not independent !

In our example , where n = 2 , and where the possible samples are

(1, 2) , (1, 3) , (2, 1) , (2, 3) , (3, 1) , (3, 2) ,
we have, e.g.,
1
P (X2 = 1 | X1 = 1) = 0 , P (X2 = 1 | X1 = 2) = .
2

Thus X1 and X2 are not independent . ( Why not ? )

280
NOTE :

Let N be the population size and n the sample size .

Suppose N is very large compared to n .

For example, n = 2 , and the population is

{ 1 , 2 , 3 , , N } .

Then we still have

P (X2 = 1 | X1 = 1) = 0 ,

but for k 6= 1 we have

1
P (X2 = k | X1 = 1) = .
N 1

One could say that X1 and X2 are almost independent . ( Why ? )

281
The Sample Correlation Coefficient

Recall the covariance of random variables X and Y :

X,Y Cov(X, Y ) E[ (XX ) (Y Y ) ] = E[XY ] E[X] E[Y ] .

It is often better to use a scaled version, the correlation coefficient

X,Y
X,Y ,
X Y
where X and Y are the standard deviation of X and Y .

We have
| X,Y | X Y , (the Cauchy-Schwartz inequality )
Thus | X,Y | 1 , ( Why ? )
If X and Y are independent then X,Y = 0 . ( Why ? )

282
Similarly, the sample correlation coefficient of a data set
{ (Xi , Yi ) }N
i=1 ,
is defined as
PN
i=1 (Xi X)(Yi Y )
RX,Y qP qP ;
N 2 N 2
i=1 (X i X) i=1 (Yi Y )

for which we have another version of the Cauchy-Schwartz inequality :

| RX,Y | 1.

Like the covariance, RX,Y measures concordance of X and Y :

If Xi > X when Yi > Y and Xi < X when Yi < Y then

RX,Y > 0 .

If Xi > X when Yi < Y and Xi < X when Yi > Y then

RX,Y < 0 .

283
The sample correlation coefficient
PN
i=1 (Xi X)(Yi Y )
RX,Y qP qP .
N 2 N 2
i=1 (X i X) i=1 (Yi Y )

can also be used to test for linearity of the data.

In fact,
If | RX,Y | = 1 then X and Y are related linearly .

Specifically,
If RX,Y = 1 then Yi = cXi + d, for constants c, d, with c > 0 .
If RX,Y = 1 then Yi = cXi + d, for constants c, d, with c < 0 .

Also,
If | RX,Y |
= 1 then X and Y are almost linear .

284
EXAMPLE :

Consider the average daily high temperature in Montreal in March.

The Table shows these averages, taken over a number of years :

1 -1.52 8 -0.52 15 2.08 22 3.39 29 6.95

2 -1.55 9 -0.67 16 1.22 23 3.69 30 6.83
3 -1.72 10 0.01 17 1.73 24 4.45 31 6.93
4 -0.94 11 0.96 18 1.93 25 4.74
5 -0.51 12 0.49 19 3.10 26 5.01
6 -0.29 13 1.26 20 3.05 27 4.66
7 0.02 14 1.99 21 3.32 28 6.45

Average daily high temperature in Montreal in March : 1943-2014 .

( Source : http://climate.weather.gc.ca/ )

These data have sample correlation coefficient RX,Y = 0.98 .

285
A scatter diagram showing the average daily high temperature.
The sample correlation coefficient is RX,Y = 0.98

286
EXERCISE :
The Table below shows class attendance and course grade/100.

The attendance was sampled in 18 sessions.

11 47 13 43 15 70 17 72 18 96 14 61 5 25 17 74
16 85 13 82 16 67 17 91 16 71 16 50 14 77 12 68
8 62 13 71 12 56 15 81 16 69 18 93 18 77 17 48
14 82 17 66 16 91 17 67 7 43 15 86 18 85 17 84
11 43 17 66 18 57 18 74 13 73 15 74 18 73 17 71
14 69 15 85 17 79 18 84 17 70 15 55 14 75 15 61
16 61 4 46 18 70 0 29 17 82 18 82 16 82 14 68
9 84 15 91 15 77 16 75

Class attendance - Course grade

Draw a scatter diagram showing the data.

Determine the sample correlation coefficient .

Any conclusions ?

287
Maximum Likelihood Estimators

EXAMPLE :

Suppose a random variable has a normal distribution with mean 0 .

Thus the density function is

1 12 x2 / 2
f (x) = e .
2

Suppose we dont know (the population standard deviation).

How can we estimate from observed data ?

( We want a formula for estimating . )

Dont we already have such a formula ?

288
EXAMPLE : ( continued )

We know we can estimate 2 by the sample variance

n
1 X
S2 (Xk X)2 .
n k=1

In fact, we have proved that

2 1 2
E[S ] = (1 ) .
n

Thus, we can call S 2 an estimator of 2 .

The maximum likelihood procedure derives such estimators.

289
The maximum likelihood procedure is the following :
Let
X1 , X2 , , Xn ,
be
independent, identically distributed ,
each having
density function f (x ; ) ,
with unknown parameter .

By independence , the joint density function is

f (x1 , x2 , , xn ; ) = f (x1 ; ) f (x2 ; ) f (xn ; ) ,

DEFINITION : The maximum likelihood estimate is

the value of that maximizes f (x1 , x2 , , xn ; ) .

NOTE : will be a function of x1 , x2 , , xn .

290
EXAMPLE : For our normal distribution with mean 0 we have
1 Pn
x2k
e 2 2 k=1
f (x1 , x2 , , xn ; ) = . ( Why ? )
( 2 )n

To find the maximum (with respect to ) we set

d
f (x1 , x2 , , xn ; ) = 0 , ( by Calculus ! )
d

or, equivalently , we set

Pn
12 x2k 
d  e 2 k=1
log n
= 0. ( Why equivalent ? )
d

Taking the (natural) logarithm gives

n
d  1 X 
2 x2k n log = 0.
d 2 k=1

291
EXAMPLE : ( continued )
We had n
d  1 X 2 
2 xk n log = 0.
d 2 k=1

Taking the derivative gives

Pn 2
x
k=1 k n
= 0,
3
from which n
1 X 2
2 = xk .
n k=1

Thus we have derived the maximum likelihood estimate

n
1 X 2  12
= Xk . ( Surprise ? )
n k=1

292
EXERCISE :
Suppose a random variable has the general normal density function
1 21 (x)2 / 2
f (x ; , ) = e ,
2
with unknown mean and unknown standard deviation .

Derive maximum likelihood estimators for both and as follows :

For the joint density function

f (x1 , x2 , , xn ; , ) = f (x1 ; , ) f (x2 ; , ) f (xn ; , ) ,

Take the log of f (x1 , x2 , , xn ; , ) .

Set the partial derivative w.r.t. equal to zero.
Set the partial derivative w.r.t. equal to zero.
Solve these two equations for and .

293
EXERCISE : ( continued )

The maximum likelihood estimators turn out to be

n
1 X
= Xk ,
n k=1
n  12
1 X
= (Xk X)2 ,
n k=1
that is,

= X , ( the sample mean ) ,

= S ( the sample standard deviation ) .

294
NOTE :

Earlier we defined the sample variance as

n
1X
S2 = (Xk X)2 .
n k=1

Then we proved that, in general,

2 1 2
E[S ] = (1 ) = 2.
n

In the preceding exercise we derived the estimator for !

( But we did so specifically for the general normal distribution. )

295
EXERCISE :

A random variable has the standard exponential distribution

with density function
x
e , x>0
f (x ; ) =
0, x0

Suppose we dont know .

Derive the maximum likelihood estimator of .

( Can you guess what the formula will be ? )

296
 EXAMPLE : Consider the special exponential density function
2 x
xe , x>0
f (x ; ) =
0, x0

1.50 1.0

0.9

1.25
0.8

0.7
1.00

0.6

F (x)
f (x)

0.75 0.5

0.4

0.50
0.3

0.2
0.25

0.1

0.00 0.0
0 1 2 3 4 5 0 1 2 3 4 5
x x

Density and distribution functions for = 1 , 2 , 3 , 4 .

297
EXAMPLE : ( continued )
For the maximum likelihood estimator of , we have
f (x ; ) = 2 x ex , for x > 0 ,
so, assuming independence, the joint density function is
f (x1 , x2 , , xn ; ) = 2n x1 x2 xn e(x1 +x2 + +xn )
.

To find the maximum (with respect to ) we set

d 
2n (x1 +x2 + +xn )

log x1 x2 xn e = 0.
d

Taking the logarithm gives

n n
d  X X 
2n log + log xk xk = 0 .
d k=1 k=1

298
EXAMPLE : ( continued )
We had
n n
d  X X 
2n log + log xk xk = 0.
d k=1 k=1

Differentiating gives
n
2n X
xk = 0,
k=1
from which
2n
= Pn .
k=1 xk

Thus we have derived the maximum likelihood estimate

2n 2
= P n = .
k=1 Xk X

NOTE : This result suggests that perhaps E[X] = 2/ . ( Why ? )

299
EXERCISE :

For the special exponential density function in the preceding example,

2 x
xe , x>0
f (x ; ) =
0, x0

Verify that Z
f (x ; ) dx = 1 .
0

Also compute
Z
E[X] = x f (x ; ) dx
0

Is it indeed true that

2
E[X] = ?

300
NOTE :
Maximum likelihood estimates also work in the discrete case .
In such case we maximize the probability mass function .

EXAMPLE :
Find the maximum likelihood estimator of p in the Bernoulli trial
P (X = 1) = p,
P (X = 0) = 1p .
SOLUTION : We can write
P (x ; p) P (X = x) = px (1 p)1x , ( x = 0, 1 ) (!)

so, assuming independence , the joint probability mass function is

P (x1 , x2 , , xn ; p) = px1 (1p)1x1 px2 (1p)1x2 pxn (1p)1xn

Pn Pn
= p k=1 xk n
(1 p)
(1 p) k=1 xk
.

301
EXAMPLE : ( continued )
We found
Pn Pn
P (x1 , x2 , , xn ; p) = p k=1 xk n
(1 p)
(1 p) k=1 xk
.

To find the maximum (with respect to p ) we set

d  Pn Pn 
log p k=1 xk (1 p)n (1 p) k=1 xk = 0.
dp

Taking the logarithm gives

n n
d  X X 
log p xk + n log(1 p) log (1 p) xk = 0.
dp k=1 k=1

Differentiating gives
n n
1 X n 1 X
xk + xk = 0.
p k=1 1p 1 p k=1

302
EXAMPLE : ( continued )
We found
n n
1 X n 1 X
xk + xk = 0,
p k=1 1p 1 p k=1
from which
 1 n
1 X n
+ xk = .
p 1 p k=1 1p

Multiplying by 1 p gives
1 p n n
 X 1 X
+1 xk = xk = n,
p k=1
p k=1

from which we obtain the maximum likelihood estimator

Pn
k=1 Xk
p = X . ( Surprise ? )
n

303
EXERCISE :

Consider the Binomial probability mass function

 
N
P (x ; p) P (X = x) = px (1 p)N x ,
x

where x is an integer, (0 x N ) .

What is the joint probability mass function P (x1 , x2 , , xn ; p) ?

( Be sure to distinguish between N and n ! )

Determine the maximum likelihood estimator p of p .

( Can you guess what p will be ? )

304
Hypothesis Testing

Often we want to decide whether a hypothesis is True or False.

To do so we gather data, i.e., a sample .

A typical hypothesis is that a random variable has a given mean .

Based on the data we want to accept or reject the hypothesis.

To illustrate concepts we consider an example in detail.

305
EXAMPLE :

We consider ordering a large shipment of 50 watt light bulbs.

The manufacturer claims that :

The lifetime of the bulbs has a normal distribution .
The mean lifetime is = 1000 hours.
The standard deviation is = 100 hours.

We want to test the hypothesis that = 1000 .

We assume that :
The lifetime of the bulbs has indeed a normal distribution .
The standard deviation is indeed = 100 hours.
We test the lifetime of a sample of 25 bulbs .

306
Density function of X , Density function of X (n = 25) ,
also indicating X , also indicating X X ,
(X = 1000 , X = 100) . (X = 1000 , X = 20 ).

307
EXAMPLE : ( continued )

We test a sample of 25 light bulbs :

We find the sample average lifetime is X = 960 hours.

Do we accept the hypothesis that = 1000 hours ?

Using the standard normal Table we have the one-sided probability

 960 1000 
P (X 960) = = (2.0) = 2.28 % ,
100/ 25
(assuming that the average lifetime is indeed 1000 hours).

Would you accept the hypothesis that = 1000 ?

Would you accept (and pay for !) the shipment ?

308
EXAMPLE : ( continued )

We test a sample of 25 light bulbs :

Suppose instead the sample average lifetime is X = 1040 hours.

Do we accept that = 1000 hours ?

Using the standard normal Table we have one-sided probability

 1040 1000 
P (X 1040) = 1 = 1 (2) = (2) = 2.28 % ,
100/ 25
(assuming again that the average lifetime is indeed 1000 hours).

Would you accept the hypothesis that that the mean is 1000 hours ?

Would you accept the shipment ? (!)

309
EXAMPLE : ( continued )

Suppose that we accept the hypothesis that = 1000 if

960 X 1040 .

Thus, if indeed = 1000 , we accept the hypothesis with probability

 960 1000 
P ( | X1000 | 40 ) = 1 2 = 12(2)
= 95 % ,
100/ 25

and we reject the hypothesis with probability

P ( | X 1000 | 40 ) = 100 % 95 % = 5 % .

310
Density function of X (n = 25) , with = X = 1000 , X = 20 ,

P (960 X 1040)
= 95%

311
EXAMPLE : ( continued )

What is the probability of

acceptance of the hypothesis if is different from 1000 ?

If the actual mean is = 980 , then acceptance has probability

1040 980 960 980
P (960 X 1040) = ( ) ( )
100/ 25 100/ 25
= (3) (1) = 1 (3) (1)

(1 0.0013) 0.1587 = 84 % .

If the actual mean is = 1040 , then acceptance has probability

1040 1040 960 1040
P (960 X 1040) = ( ) ( )
100/ 25 100/ 25
= (0) (4) = 50 % .

312
 = X = 980 = X = 1000 = X = 1040
P (accept) = 84% P (accept) = 95% P (accept) = 50%

Density functions of X : n = 25 , X = 20

QUESTION 1 : How does P (accept) change when we slide the

density function of X along the X-axis , i.e., when changes ?

QUESTION 2 : What is the effect of increasing the sample size n ?

313
EXAMPLE :

Now suppose there are two lots of light bulbs :

Lot 1 : Light bulbs with mean life time 1 = 1000 hours,

Lot 2 : Light bulbs with mean life time 2 = 1100 hours.

We want to decide which lot our sample of 25 bulbs is from.

Consider the decision criterion x , where 1000 x 1100 :

If X x then the sample is from Lot 1 .

If X > x then the sample is from Lot 2 .

314
There are two hypotheses :

H1 : The sample is from Lot 1 (1 = 1000) .

H2 : The sample is from Lot 2 (2 = 1100) .

We can make two types of errors :

Type 1 error : Accept H2 when H1 is True ,

Type 2 error : Accept H1 when H2 is True ,

which happen when, for given decision criterion x ,

Type 1 error : If X > x and the sample is from Lot 1 .

Type 2 error : If X x and the sample is from Lot 2 .

315
 The density functions of X (n = 25) , also indicating x .
blue : (1 , 1 ) = (1000, 100) , red : (2 , 2 ) = (1100, 200) .

Type 1 error : area under the blue curve, to the right of x .

Type 2 error : area under the red curve, to the left of x .

QUESTION : What is the effect of moving x on these errors ?

316
RECALL :

Type 1 error : If X > x and the sample is from Lot 1 .

Type 2 error : If X x and the sample is from Lot 2 .

These errors occur with probability

Type 1 error : P (X x | = 1 1000) .

Type 2 error : P (X x | = 2 1100) .

We should have, for the (rather bad) choice x = 1000 ,

Type 1 error : P (X 1000 | = 1 1000) = 0.5 .

and for the (equally bad) choice x = 1100 ,

Type 2 error : P (X 1100 | = 2 1100) = 0.5 .

317
Probability of Type 1 error vs. x Probability of Type 2 error vs. x
(1 , 1 ) = (1000, 100) (2 , 2 ) = (1100, 100) .

Sample sizes : 2 (red) , 8 (blue) , 32 (black) .

318
The probability of Type 1 and Type 2 errors versus x .
Left : (1 , 1 ) = (1000, 100), (2 , 2 ) = (1100, 100).
Right : (1 , 1 ) = (1000, 100), (2 , 2 ) = (1100, 200).
Colors indicate sample size : 2 (red), 8 (blue), 32 (black) .
Curves of a given color intersect at the minimax x-value.

319
The probability of Type 1 and Type 2 errors versus x .

Left : (1 , 1 ) = (1000, 100), (2 , 2 ) = (1100, 300).

Right : (1 , 1 ) = (1000, 100), (2 , 2 ) = (1100, 400).

Colors indicate sample size : 2 (red), 8 (blue), 32 (black) .

Curves of a given color intersect at the minimax x-value.

320
NOTE :

There is an optimal value x of x .

At x the value of

max { P (Type 1 Error) , P (Type 2 Error) }

is minimized .

We call x the minimax value.

The value of x depends on 1 and 2 .

The value of x is independent of the sample size.

(We will prove this!)

321
 0.0040 0.020

0.0035

0.0030 0.015

0.0025

fX (x)
f (x)

0.0020 0.010

0.0015

0.0010 0.005

0.0005

0.0000 0.000
400 600 800 1000 1200 1400 1600 1800 400 600 800 1000 1200 1400 1600 1800
x x

The population density functions. The density functions of X (n = 25) .

(1 , 1 ) = (1000 , 100) (blue)

(2 , 2 ) = (1100 , 200) (red)

322
The density functions of X (n = 25) , with minimax value of x .

(1 , 1 ) = (1000, 100) (blue) , (2 , 2 ) = (1100, 200) (red) .

323
The minimax value x of x is easily computed : At x we have
P ( Type 1 Error ) = P ( Type 2 Error ) ,

P (X x | = 1 ) = P (X x | = 2 ) ,
 x   x 
1
= 2 ,
1 / n 2 / n
1 x x 2
= , ( by monotonicity of ) .
1 / n 2 / n
from which
1 2 + 2 1
x = . ( Check ! )
1 + 2

With 1 = 1000 , 1 = 100 , 2 = 1100 , 2 = 200 , we have

1000 200 + 1100 100

x = = 1033 .
100 + 200

324
Thus we have proved the following :

FACT : Suppose Lot 1 and Lot 2 are normally distributed ,

with mean and standard deviation

(1 , 1 ) and (2 , 2 ) , where (1 < 2 ) ,

and sample size n .

Then the value of decision criterion x that minimizes

max { P (Type 1 Error) , P (Type 2 Error) } ,

i.e., the value of x that minimizes

max { P (X x | = 1 , = 1 ) , P (X x | = 2 , = 2 } ,

is given by
1 2 + 2 1
x = .
1 + 2

325
EXERCISE :

Determine the optimal decision criterion x that minimizes

max { P (Type 1 Error) , P (Type 2 Error) } ,

when

(1 , 1 ) = (1000, 200) , (2 , 2 ) = (1100, 300) .

For this x find the probability of a Type 1 and a Type 2 Error ,

when

n=1 , n = 25 , n = 100 .

326
EXAMPLE ( Known standard deviation ) :

Given : A sample of size 9 from a normal population with = 0.2

has sample mean
X = 4.88 ,

Claim : The population mean is

= 5.00 , ( the null hypothesis H0 )

We see that | X | = | 4.88 5.00 | = 0.12 .

We reject H0 if P ( | X | 0.12 ) is rather small , say, if

P ( | X | 0.12 ) < 10 % ( level of significance 10 % )

Do we accept H0 ?

327
SOLUTION ( Known standard deviation ) :
Given: n = 9 , = 0.2 , X = 4.88, = 5.0, | X |= 0.12 .
Since
X
Z is standard normal ,
/ n

the p-value ( from the standard normal Table ) is

 0.12 
P ( | X | 0.12 ) = P | Z |
0.2/ 9
= P ( | Z | 1.8 ) = 2 (1.8) = 7.18 % .

Thus we reject the hypothesis that = 5.00 significance level 10 % .

NOTE : We would accept H0 if the level of significance were 5 % .

( We are more tolerant when the level of significance is smaller. )

328
EXAMPLE ( Unknown standard deviation, large sample ) :

Given : A sample of size n = 64 from a normal population has

sample mean X = 4.847 ,

and
sample standard deviation S = 0.234 .

Test the hypothesis that 4.8 ,

and reject it if P (X 4.847) is small, say if

P ( X 4.847 ) < 5 % .

NOTE : Since the sample size n = 64 is large, we can assume that

= S = 0.234 .

329
SOLUTION ( Unknown standard deviation, large sample ) :

Given X = 4.847 , = 4.8 , n = 64 , and

= S = 0.234 .

Using the standard normal approximation we have that

X 4.847 ,
if and only if
X 4.847 4.8
Z = 1.6 .
/ n 0.234/8

From the standard normal Table we have the p-value

P (Z 1.6) = 1 (1.6) = (1.6) = 5.48 % .

CONCLUSION:
We (barely) accept H0 at level of significance 5 % .
( We would reject H0 at level of significance 10 % .)

330
EXAMPLE ( Unknown standard deviation, small sample ) :

A sample of size n = 16 from a normal population has

sample mean X = 4.88 ,

and
sample standard deviation S = 0.234 .

Test the null hypothesis

H0 : 4.8 ,
and reject it if
P (X 4.88) < 5 % .

NOTE :
If n 30 then the approximation
= S is not so accurate.
In this case better use the student t-distribution Tn1 .

331
 The T - distribution Table
n = 0.1 = 0.05 =0.01 = 0.005
5 -1.476 -2.015 -3.365 -4.032
6 -1.440 -1.943 -3.143 -3.707
7 -1.415 -1.895 -2.998 -3.499
8 -1.397 -1.860 -2.896 -3.355
9 -1.383 -1.833 -2.821 -3.250
10 -1.372 -1.812 -2.764 -3.169
11 -1.363 -1.796 -2.718 -3.106
12 -1.356 -1.782 -2.681 -3.055
13 -1.350 -1.771 -2.650 -3.012
14 -1.345 -1.761 -2.624 -2.977
15 -1.341 -1.753 -2.602 -2.947

This Table shows t,n values such that P (Tn t,n ) = .

( For example, P (T10 2.764) = 1% )
SOLUTION ( Unknown standard deviation, small sample ) :

With n = 16 we have X 4.88 if and only if

X 4.88 4.8
Tn1 = T15 = = 1.37 .
S/ n 0.234/4

The t-distribution Table shows that

P ( T15 1.341 ) = P ( T15 1.341 ) = 10 % ,

P ( T15 1.753 ) = P ( T15 1.753 ) = 5%.

Thus we reject H0 at level of significance 10 % ,

but we accept H0 at level of significance 5 % .

( We are more tolerant when the level of significance is smaller. )

332
EXAMPLE ( Testing a hypothesis on the standard deviation ) :
A sample of 16 items from a normal population has sample
standard deviation S = 2.58 .
Do you believe the population standard deviation satisfies 2.0 ?

SOLUTION : We know that

n1 2 2
2
S has the n1 distribution .

For our data :
n1 2 15 2
S 2.58 if and only if 2
S 2.58 = 24.96 ,
4
and from the 2 Table
P ( 2 25.0 )
15 = 5.0 % .

Thus we (barely) accept the hypothesis at significance level 5 % .

( We would reject the hypothesis at significance level 10 % . )

333
EXERCISE :

A sample of 16 items from a normal population has sample

standard deviation

S = 0.83 .

Do you believe the hypothesis that satisfies

1.2 ?

( Probably Yes ! )

334
 LEAST SQUARES APPROXIMATION

Linear Least Squares

Recall the following data :

1 -1.52 8 -0.52 15 2.08 22 3.39 29 6.95

2 -1.55 9 -0.67 16 1.22 23 3.69 30 6.83
3 -1.72 10 0.01 17 1.73 24 4.45 31 6.93
4 -0.94 11 0.96 18 1.93 25 4.74
5 -0.51 12 0.49 19 3.10 26 5.01
6 -0.29 13 1.26 20 3.05 27 4.66
7 0.02 14 1.99 21 3.32 28 6.45
Average daily high temperature in Montreal in March : 1943-2014 .
( Source : http://climate.weather.gc.ca/ )

These data have sample correlation coefficient RX,Y = 0.98 .

335
Average daily high temperature in Montreal in March

336
Suppose that :

We believe that these temperatures basically increase linearly .

In fact we found the sample correlation coefficient Rxy = 0.98 .

Thus we believe in a relation

Tk = c1 + c2 k , k = 1, 2, , 31 .

The deviations from linearity come from random influences .

These random influences can be due to many factors .

The deviations may have a normal distribution .

We want to determine the best linear approximation.

337
 Average daily high temperatures, with a linear approximation .

QUESTION : Guess how this linear approximation was obtained !

338
 There are many ways to determine such a linear approximation.

Often used is the least squares method .

This method determines the values of c1 and c2 that minimize

the least squares error :
N
X
( Tk (c1 + c2 xk ) )2 ,
k=1

where, in our example, N = 31 and xk = k .

To do so set the partial derivatives w.r.t. c1 and c2 to zero :

N
X
w.r.t. c1 : 2 ( Tk (c1 + c2 xk ) ) = 0 ,
k=1
N
X
w.r.t. c2 : 2 xk ( Tk (c1 + c2 xk ) ) = 0 .
k=1

339
The least squares error versus c1 and c2 .

340
From setting the partial derivatives to zero, we have
N
X N
X
( Tk (c1 + c2 xk ) ) = 0 , xk ( Tk (c1 + c2 xk ) ) = 0 .
k=1 k=1

Solving these two equations for c1 and c2 gives

PN PN
k=1 xk Tk x k=1 Tk
c2 = PN 2 ,
2
k=1 xk N x
and
c1 = T c2 x ,
where
N N
1 X 1 X
x = xk , T = Tk .
N k=1 N k=1

EXERCISE : Check these formulas !

341
 EXAMPLE : For our March temperatures example, we find
c1 = 2.47 and c2 = 0.289 .

Average daily high temperatures, with linear least squares approximation .

342
General Least Squares
Given discrete data points
{ (xi , yi ) }N
i=1 ,

find the coefficients ck of the function

n
X
p(x) ck k (x) ,
k=1

that minimize the least squares error

N
X
EL (p(xi ) yi )2
i=1

EXAMPLES :
p(x) = c1 + c2 x . (Already done !)

p(x) = c1 + c2 x + c3 x2 . (Quadratic approximation)

343
For any vector x RN let
N
X
k x k2 xT x x2k . (T denotes transpose).
i=1
Then

p(x1 ) y1
N
X 2
EL [ p(xi ) yi ]2 = k k

i=1
p(xN ) yN
Pn
i=1 ci i (x1 )
y1
2
= k
k
Pn
i=1 ci i (xN ) yN

1 (x1 ) n (x1 ) y1
c1 2
k Ac y k2 .

= k
k

cn
1 (xN ) n (xN ) yN

344
THEOREM :

For the least squares error EL to be minimized we must have

AT A c = AT y .

PROOF :

EL = k Ac y k2

= (Ac y)T (Ac y)

= (Ac)T Ac (Ac)T y yT Ac + yT y

= cT AT Ac cT AT y yT Ac + yT y .

345
PROOF : ( continued )
We had
EL = cT AT Ac cT AT y yT Ac + yT y .

For a minimum we need

EL EL
= 0, i .e., = 0, i = 0, 1, , n ,
c ci
which gives
cT AT A + (AT Ac)T (AT y)T yT A = 0 , ( Check ! )
i.e.,
2cT AT A 2yT A = 0 ,
or
c T AT A = y T A .
Transposing gives
AT Ac = AT y . QED !

346
EXAMPLE : Given the data points
{ (xi , yi ) }4i=1 = { (0, 1) , (1, 3) , (2, 2) , (4, 3) } ,
find the coefficients c1 and c2 of p(x) = c1 + c2 x ,
that minimize
X4
EL [ (c1 + c2 xi ) yi ]2 .
i=1
SOLUTION : Here N = 4 , n = 2 , 1 (x) = 1 , 2 (x) = x .
Use the Theorem :
  1 0     1
1 1 1 1 1 1 c1 1 1 1 1 3
= ,
0 1 2 4 1 2 c2 0 1 2 4 2
1 4 3
or     
4 7 c1 9
= ,
7 21 c2 19

with solution c1 = 1.6 and c2 = 0.371429 .

347
EXAMPLE : Given the same data points, find the coefficients of
p(x) = c1 + c2 x + c3 x2 ,
that minimize 4
X
EL [ (c1 + c2 xi + c3 x2i ) yi ]2 .
i=1
SOLUTION : Here
N =4 , n=3 , 1 (x) = 1 , 2 (x) = x , 3 (x) = x2 .
Use the Theorem
:
1 0 0 1
1 1 1 1 c1 1 1 1 1
0 1 1 1
c2 = 0 1 2 4 3 ,
1 2 4 1 2 4 2
0 1 4 16 c3 0 1 4 16
1 4 16 3
or
4 7 21 c1 9
7 21 73 c2 = 19 ,
21 73 273 c3 59

with solution c1 = 1.32727 , c2 = 0.936364 , c3 = 0.136364 .

348
 The least squares approximations from the preceding two examples :

3.5 3.5

3.0 3.0

2.5 2.5
y

y
2.0 2.0

1.5 1.5

1.0 1.0

0.5 0.5
0 1 2 3 4 0 1 2 3 4
x x

p(x) = c1 + c2 x p(x) = c1 + c2 x + c3 x2

349
EXAMPLE : From actual data :

The average daily high temperatures in Montreal (by month) are :

January -5
February -3
March 3
April 11
May 19
June 24
July 26
August 25
September 20
October 13
November 6
December -2

Source : http : //weather.uk.msn.com

350
 30

25

20

15
Temperature

10

10
0 2 4 6 8 10 12
Month
Average daily high temperature in Montreal (by month).

351
EXAMPLE : ( continued )
The graph suggests using a 3-term least squares approximation

p(x) = c1 1 (x) + c2 2 (x) + c3 3 (x) ,

of the form
x x
p(x) = c1 + c2 sin( ) + c3 cos( ) .
6 6
QUESTIONS :
Why include 2 (x) = sin( x
6
)?
x
Why is the argument 6
?

Why include the constant term 1 (x) = c1 ?

Why include 3 (x) = cos( x

6
)?
In this example we find the least squares coefficients
c1 = 11.4 , c2 = 8.66 , c3 = 12.8 .

352
 30

25

20

15
Temperature

10

10
0 2 4 6 8 10 12
Month
Least squares fit of average daily high temperatures.

353
EXAMPLE :
Consider the following experimental data :

0.6

0.5

0.4
y

0.3

0.2

0.1

0.0
0 1 2 3 4 5 6 7 8
x

354
EXAMPLE : ( continued )

Suppose we are given that :

These data contain noise .

The underlying physical process is understood.

The functional dependence is known to have the form

y = c1 xc2 ec3 x .

The values of c1 , c2 , c3 are not known.

355
EXAMPLE : ( continued )

The functional relationship has the form

y = c1 xc2 ec3 x .

NOTE :

The unknown coefficients c1 , c2 , c3 appear nonlinearly !

This gives nonlinear equations for c1 , c2 , c3 !

Such problems are more difficult to solve !

What to do ?

356
EXAMPLE : ( continued )

Fortunately, in this example we can take the logarithm :

log y = log ( c1 xc2 ec3 x ) = log c1 + c2 log x c3 x .

This gives a linear relationship

log y = c1 1 (x) + c2 2 (x) + c3 3 (x) ,
where
c1 = log c1 .
and
1 (x) = 1 , 2 (x) = log x , 3 (x) = x .

Thus
We can now use regular least squares.

We first need to take the logarithm of the data.

357
EXAMPLE : ( continued )

0.0

0.5

1.0

1.5

2.0
log y

2.5

3.0

3.5

4.0

4.5
0 1 2 3 4 5 6 7 8
x

The logarithm of the original y-values versus x .

358
EXAMPLE : ( continued )

We had
y = c1 xc2 ec3 x ,
and
log y = c1 1 (x) + c2 2 (x) + c3 3 (x) ,
with
1 (x) = 1 , 2 (x) = log x , 3 (x) = x ,
and
c1 = log c1 .

We find the following least squares values of the coefficients :

c1 = 0.00473 , c2 = 2.04 , c3 = 1.01 ,
and
c1 = ec1 = 0.995 .

359
EXAMPLE : ( continued )

3
log y

7
0 1 2 3 4 5 6 7 8
x

The least squares approximation of the transformed data.

360
EXAMPLE : ( continued )

0.6

0.5

0.4
y

0.3

0.2

0.1

0.0
0 1 2 3 4 5 6 7 8
x

The least squares approximation shown in the original data.

361
 RANDOM NUMBER GENERATION

Measured data often have random fluctuations .

This may be due to inaccurate measurements .
It may also be due to other external influences .

Often we know or believe there is a deterministic model .

(i.e., the process can be modeled by a deterministic equation .)

However, deterministic equations can also have random behavior !

The study of such equations is sometimes called chaos theory .

We will look at a simple example, namely, the logistic equation .

362
The Logistic Equation

A simple deterministic model of population growth is

xk+1 = xk , k = 1, 2, ,

for given , ( 0) , and for given x0 , (x0 0) .

The solution is

xk = k x0 , k = 1, 2, .

Thus

If 0 < 1 then xk 0 as k (extinction) .

If >1 then xk as k (exponential growth) .

363
A somewhat more realistic population growth model is

xk+1 = xk (1 xk ) , k = 1, 2, ,

This model is known as the logistic equation .

The maximum sustainable population is 1 .

is given, (0 4) .

x0 is given, (0 x0 1) .

Then 0 xk 1 for all k . (Prove this !)

QUESTION : How does the sequence {xk }

k=1 depend on ?

364
EXERCISE :

xk+1 = xk (1 xk ) , k = 1, 2, ,

Divide the interval [0, 1] into 200 subintervals .

Compute xk for k = 1, 2, 50 .

Count the xk s in each subinterval.

Determine the percentage of xk s in each subinterval.

Present the result in a diagram .

Do this for different choices of (0 4) .

365
 1.0

0.5 0.9

0.8

0.4
0.7

0.6
p(x)

0.3

y
0.5

0.4
0.2

0.3

0.1 0.2

0.1

0.0 0.0
0.0 0.2 0.4 0.6 0.8 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
x x

Percentage per interval Graphical interpretation.

= 0.9 , x0 = 0.77 , 50 iterations , 200 subintervals .

366
 0.9 1.0

0.8 0.9

0.8
0.7

0.7
0.6

0.6
0.5
p(x)

y
0.5
0.4
0.4

0.3
0.3

0.2
0.2

0.1 0.1

0.0 0.0
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
x x

Percentage per interval Graphical interpretation.

= 1.7 , x0 = 0.77 , 50 iterations , 200 subintervals .

367
 0.06 1.0

0.9

0.05
0.8

0.7
0.04

0.6
p(x)

y
0.03 0.5

0.4

0.02
0.3

0.2
0.01

0.1

0.00 0.0
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
x x

Percentage per interval Graphical interpretation.

= 3.77 , x0 = 0.77 , 50 iterations , 200 subintervals .

368
EXERCISE :

xk+1 = xk (1 xk ) , k = 1, 2, ,

Do the same as in the preceding example, but now

Compute xk for k = 1, 2, 1, 000, 000 !

Do not record the first 200 iterations.

( This to eliminate transient effects .)

You will see that there is a fixed point (a cycle of period 1 ) .

369
 1.0
1.0
0.9

0.8
0.8

0.7

0.6
0.6
p(x)

y
0.5

0.4 0.4

0.3

0.2 0.2

0.1

0.0 0.0
0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
x x

Percentage per interval Graphical interpretation.

= 1.7 , 1, 000, 000 iterations , 200 subintervals .

There is a fixed point (a cycle of period 1 ) .

370
 1.0
0.25

0.9

0.8
0.20

0.7

0.6
0.15
p(x)

y
0.5

0.10 0.4

0.3

0.05 0.2

0.1

0.00 0.0
0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
x x

Percentage per interval Graphical interpretation.

= 3.46 , 1, 000, 000 iterations , 200 subintervals .

There is a cycle of period 4 .

371
 1.0

0.12
0.9

0.8
0.10

0.7

0.08
0.6
p(x)

y
0.5
0.06

0.4

0.04 0.3

0.2
0.02
0.1

0.00 0.0
0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
x x

Percentage per interval Graphical interpretation.

= 3.561 , 1, 000, 000 iterations , 200 subintervals .

There is a cycle of period 8 .

372
 0.08 1.0

0.9
0.07

0.8
0.06
0.7

0.05
0.6
p(x)

y
0.04 0.5

0.4
0.03

0.3
0.02
0.2

0.01
0.1

0.00 0.0
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
x x

Percentage per interval Graphical interpretation.

= 3.6 , 1, 000, 000 iterations , 200 subintervals .
(The Figure on the right only shows 100 of these iterations.)
There is apparent chaotic behavior .

373
 0.035 1.0

0.9
0.030
0.8

0.025 0.7

0.6
0.020
p(x)

y
0.5

0.015
0.4

0.010 0.3

0.2
0.005
0.1

0.000 0.0
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
x x

Percentage per interval Graphical interpretation.

= 3.77 , 1, 000, 000 iterations , 200 subintervals .
(The Figure on the right only shows 100 of these iterations.)
There is apparent chaotic behavior .

374
 0.040 1.0

0.9
0.035

0.8
0.030
0.7

0.025
0.6
p(x)

y
0.020 0.5

0.4
0.015

0.3
0.010
0.2

0.005
0.1

0.000 0.0
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
x x

Percentage per interval Graphical interpretation.

= 3.89 , 1, 000, 000 iterations , 200 subintervals .
(The Figure on the right only shows 100 of these iterations.)
There is apparent chaotic behavior .

375
 0.035 1.0

0.9
0.030
0.8

0.025 0.7

0.6
0.020
p(x)

y
0.5

0.015
0.4

0.010 0.3

0.2
0.005
0.1

0.000 0.0
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
x x

Percentage per interval Graphical interpretation.

= 3.99 , 1, 000, 000 iterations , 200 subintervals .
(The Figure on the right only shows 100 of these iterations.)
There is apparent chaotic behavior .

376
CONCLUSIONS :

The behavior of the logistic equation depends on .

For certain values of we see fixed points .

For other values of there are cycles .

For yet other values of there is seemingly random behavior .

Many other deterministic equations have complex behavior !

Nature is complex !

377
Generating Random Numbers

The logistic equation is a recurrence relation of the form

xk+1 = f (xk ) , k = 1, 2, 3, .
Most random number generators also have this form.

For the logistic equation we did not see sequences {xk }N

k=1 having

a uniform distribution,
a normal distribution,
another known distribution.

QUESTION :
How to generate uniform (and other) random numbers ?
( These are useful in computer simulations .)

378
Generating Uniformly Distributed Random Numbers

Uniformly distributed random numbers can also be generated by

a recurrence relation of the form

xk+1 = f (xk ) , k = 1, 2, 3, .

Unlike the logistic equation, the xk are most often integers .

The recurrence relation typically has the form

xk+1 = (n xk ) mod p .

where p is a large prime number , and n a large integer , with

p 6| n .

379
The following fact is useful :

THEOREM :

Let p be a prime number, and n an integer such that

p 6| n .

Then the function

f : { 0, 1, 2, , p 1 } { 0, 1, 2, , p 1 } ,

given by
f (x) = (n x) mod p ,

is one-to-one (and hence onto, a bijection, and invertible).

380
 p prime , p 6 | n (n x) mod p is 1 1

EXAMPLE :
p = 7 and n = 12 .

x 12x 12x mod 7

0 0 0
1 12 5
2 24 3
3 36 1
4 48 6
5 60 4
6 72 2

Invertible !
NOTE : The values of 12x mod 7 look somewhat random !

381
 p prime , p 6 | n (n x) mod p is 1 1

EXAMPLE :
p = 6 and n = 2.

x 2x 2x mod 6
0 0 0
1 2 2
2 4 4
3 6 0
4 8 2
5 10 4

Not Invertible .

382
 p prime , p 6 | n (n x) mod p is 1 1

EXAMPLE :
p = 6 and n = 13 .

x 13x 13x mod 6

0 0 0
1 13 1
2 26 2
3 39 3
4 52 4
5 65 5

Invertible . ( So ? )

NOTE : The numbers in the right hand column dont look random !

383
 p prime , p 6 | n (n x) mod p is 1 1

PROOF : By contradiction : Suppose the function is not 1 1 .

Then there are distinct integers x1 , x2 { 0, 1, 2, , p 1 } ,

such that

(nx1 ) mod p = k and (nx2 ) mod p = k ,

where also k { 0, 1, 2, , p 1 } . It follows that

p | n(x1 x2 ) , ( Why ? )

Since p is prime and p 6 | n it follows that

p | (x1 x2 ) .

Thus x1 x2 = 0 ( Why ? ) , i.e. , x1 = x2 . Contradiction !

384
For given x0 , an iteration of the form
xk = (n xk1 ) mod p , k = 1, 2, , p 1 .

can be used to generate random numbers .

Here p is a large prime number .

The value of n is also large .

The integer n must not be divisible by p .

Do not start with x0 = 0 (because it is a fixed point !) .

Be aware of cycles (of period less than p 1) !

REMARK : Actually, more often used is an iteration of the form

xk = (n xk1 + m) mod p , k = 1, 2, , p 1 .

385
EXAMPLE : As a simple example, take again p = 7 and n = 12 :

x 12x 12x mod 7

0 0 0
1 12 5
2 24 3
3 36 1
4 48 6
5 60 4
6 72 2

With x1 = 1 the recurrence relation

xk+1 = f (xk ) , where f (x) = 12x mod 7 ,
generates the sequence
1 5 4 6 2 3 1 5
which is a cycle of maximal period p 1 (here 6) .

386
 f (x) sequence cycle period

5x mod 7 1546231 6

6x mod 7 1616161 2

8x mod 7 1111111 1

9x mod 7 1241241 3

10x mod 7 1326451 6

11x mod 7 1421421 3

12x mod 7 1546231 6

387
EXAMPLE : With x0 = 2 , compute

xk = (137951 xk1 ) mod 101 , k = 1, 2, , 100 .

Result :

71 46 17 48 88 94 4 41 92 34
96 75 87 8 82 83 68 91 49 73
16 63 65 35 81 98 45 32 25 29
70 61 95 90 64 50 58 39 21 89
79 27 100 15 78 42 77 57 54 99
30 55 84 53 13 7 97 60 9 67
5 26 14 93 19 18 33 10 52 28
85 38 36 66 20 3 56 69 76 72
31 40 6 11 37 51 43 62 80 12
22 74 1 86 23 59 24 44 47 2

QUESTION : Are there repetitions (i.e., cycles) ?

388
EXAMPLE : As in the preceding example, use x0 = 2 , and
compute
xk = (137951 xk1 ) mod 101 , k = 1, 2, , 100 ,
and set xk
xk = .
100
0.710 0.460 0.170 0.480 0.880 0.940 0.040 0.410 0.920 0.340
0.960 0.750 0.870 0.080 0.820 0.830 0.680 0.910 0.490 0.730
0.160 0.630 0.650 0.350 0.810 0.980 0.450 0.320 0.250 0.290
0.700 0.610 0.950 0.900 0.640 0.500 0.580 0.390 0.210 0.890
0.790 0.270 1.000 0.150 0.780 0.420 0.770 0.570 0.540 0.990
0.300 0.550 0.840 0.530 0.130 0.070 0.970 0.600 0.090 0.670
0.050 0.260 0.140 0.930 0.190 0.180 0.330 0.100 0.520 0.280
0.850 0.380 0.360 0.660 0.200 0.030 0.560 0.690 0.760 0.720
0.310 0.400 0.060 0.110 0.370 0.510 0.430 0.620 0.800 0.120
0.220 0.740 0.010 0.860 0.230 0.590 0.240 0.440 0.470 0.020

QUESTION : Do these numbers look uniformly distributed ?

389
EXAMPLE : With x0 = 2 , compute

xk = (137953 xk1 ) mod 101 , k = 1, 2, , 100 .

Result :

75 35 50 57 67 38 11 59 41 73
61 15 7 10 72 74 48 83 32 89
55 93 3 62 2 75 35 50 57 67
38 11 59 41 73 61 15 7 10 72
74 48 83 32 89 55 93 3 62 2
75 35 50 57 67 38 11 59 41 73
61 15 7 10 72 74 48 83 32 89
55 93 3 62 2 75 35 50 57 67
38 11 59 41 73 61 15 7 10 72
74 48 83 32 89 55 93 3 62 2

QUESTION : Are there cycles ?

390
EXAMPLE : With x0 = 4 , compute

xk = (137953 xk1 ) mod 101 , k = 1, 2, , 100 .

Result :
49 70 100 13 33 76 22 17 82 45
21 30 14 20 43 47 96 65 64 77
9 85 6 23 4 49 70 100 13 33
76 22 17 82 45 21 30 14 20 43
47 96 65 64 77 9 85 6 23 4
49 70 100 13 33 76 22 17 82 45
21 30 14 20 43 47 96 65 64 77
9 85 6 23 4 49 70 100 13 33
76 22 17 82 45 21 30 14 20 43
47 96 65 64 77 9 85 6 23 4

QUESTIONS :
Are there cycles ?
Is this the same cycle that we already found ?

391
Generating Random Numbers using the Inverse Method

There are algorithms that generate uniform random numbers.

These can be used to generate other random numbers.

A simple method to do this is the Inverse Transform Method .

392
RECALL :

Let f (x) be a density function on an interval [a, b] .

The distribution function is

Z x
F (x) f (x) dx .
a

Since f (x) 0 we know that F (x) is increasing .

If F (x) is strictly increasing then F (x) is invertible .

393
 X x1 x2
We want random numbers X with distribution F (x) (blue) .
Let the random variable Y be uniform on the interval [0, 1] .
Let X = F 1 (Y ) .
Then P (x1 X x2 ) = y2 y1 = F (x2 ) F (x1 ) .
Thus F (x) is indeed the distribution function of X !

394
 If Y is uniformly distributed on [0, 1] then

P ( y1 Y y2 ) = y2 y1 .

Let
X = F 1 (Y ) ,
with
x1 = F 1 (y1 ) and x2 = F 1 (y2 ) .

Then

P (x1 X x2 ) = y2 y1 = F (x2 )F (x1 ) . ( Why ? )

Thus F (X) is also the distribution function of X F 1 (Y ) !

NOTE : In the illustration X is on [0, 1] , but this is not necessary.

395
EXAMPLE :

Recall that the exponential density function

x
e , x>0
f (x) =
0, x0

has distribution function

1 ex , x > 0
F (x) =
0, x0

The inverse distribution function is

1 1
F (y) = log(1 y) , 0y<1. ( Check ! )

396
 0.8

0.6
F (x)

0.4

0.2

0.0
0 1 2 3 4 5 6
x
The inverse method for generating 20 exponential random numbers ,

for the exponential distribution function with = 1 .

397
EXAMPLE : ( continued )

(Using the inverse method to get exponential random numbers.)

Divide [0, 12] into 100 subintervals of equal size x = 0.12 .

Let Ik denote the kth interval, with midpoint xk .

Use the inverse method to get N exponential random numbers.

Let mk be the frequency count (# of random values in Ik ) .

Let mk
f(xk ) = .
N x
100
Then
Z
f(x) dx f(xk ) x = 1 .
X
=
0 k=1

Then f(x) approximates the actual density function f (x) .

398
 1.0 1.0

0.8 0.8

0.6 0.6
f (x)

f (x)
0.4 0.4

0.2 0.2

0.0 0.0
0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7
x x

N = 500 . N = 50, 000 .

Simulating the exponential random variable with = 1 .

(The actual density function is shown in blue.)

399
EXERCISE : Consider the Tent density function

x + 1 , 1 < x 0
f (x) = 1x , 0<x1
0, otherwise

Verify that that the distribution function is given by

1 2
2 x + x + 21 , 1 x 0
F (x) =
1 2
2 x + x + 21 , 0<x1

Verify that that the inverse distribution function is

1 + 2y , 0 y 12

F 1 (y) =
1 2 2y , 12 < y 1

Use the inverse method to generate Tent random numbers .

400
 1.0

0.8
0.8

0.6
0.6
f (x)

F (x)
0.4
0.4

0.2 0.2

0.0 0.0
1.0 0.5 0.0 0.5 1.0 1.0 0.5 0.0 0.5 1.0
x x

The Tent density Function. The Tent distribution function.

The inverse method for generating 20 Tent random numbers .

401
 1.50 1.25

1.25
1.00

1.00

0.75
f (x)

f (x)
0.75

0.50

0.50

0.25
0.25

0.00 0.00
1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00 1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.00
x x

N = 500 . N = 50, 000 .

Simulating the Tent random variable .

(The actual density function is shown in blue.)

402
SUMMARY TABLES

and

FORMULAS

403
 Discrete Continuous

p(xi ) = P (X = xi ) f (x)x
= P (x
2 < X < x + 2 )

P R
i p(xi ) = 1
f (x) dx = 1

P Rx
F (xk ) = ik p(xi ) F (x) =
f (x) dx

p(xk ) = F (xk ) F (xk1 ) f (x) = F (x)

P R
E[X] = i xi p(xi ) E[X] =
x f (x) dx

P R
E[g(X)] = i g(xi )p(xi ) E[g(X)] =
g(x) f (x) dx

P R R
E[XY ] = i,j xi yj p(xi , yj ) E[XY ] =
xyf (x, y) dy dx

404
 Name General Formula

Mean = E[X]

Variance V ar(X) = E[(X )2 ] = E[X 2 ] 2

Covariance Cov(X, Y ) = E[(X X )(Y Y )] = E[XY ] X Y

Markov P (X c) E[X]/c

Chebyshev P (| X | k) 1/k2

Moments (t) = E[etX ] , (0) = E[X] , (0) = E[X 2 ]

405
 Name Probability mass function Domain

Bernoulli P (X = 1) = p , P (X = 0) = 1 p 0, 1

 
n
Binomial P (X = k) = pk (1 p)nk 0kn
k

k
Poisson P (X = k) = e k!
k = 0, 1, 2,

406
 Name Mean Standard deviation
p
Bernoulli p p(1 p)

p
Binomial np np(1 p)

Poisson

407
Name Density function Distribution Domain

1 xa
Uniform ba ba
x (a, b]

Exponential ex 1 ex x (0, )

1 2
Std. Normal 1 e 2 x x (, )
2

1 2 2
Normal 1 e 2 (x) / x (, )
2

408
Name Mean Standard Deviation

a+b ba
Uniform 2

2 3

1 1
Exponential

Standard Normal 0 1

General Normal

Chi-Square n 2n

409
 The Standard Normal Distribution (z)

z (z) z (z)
0.0 .5000 -1.2 .1151
-0.1 .4602 -1.4 .0808
-0.2 .4207 -1.6 .0548
-0.3 .3821 -1.8 .0359
-0.4 .3446 -2.0 .0228
-0.5 .3085 -2.2 .0139
-0.6 .2743 -2.4 .0082
-0.7 .2420 -2.6 .0047
-0.8 .2119 -2.8 .0026
-0.9 .1841 -3.0 .0013
-1.0 .1587 -3.2 .0007

( For example, P (Z 2.0) = (2.0) = 2.28% )

410
 The 2n - Table
n = 0.975 = 0.95 = 0.05 = 0.025
5 0.83 1.15 11.07 12.83
6 1.24 1.64 12.59 14.45
7 1.69 2.17 14.07 16.01
8 2.18 2.73 15.51 17.54
9 2.70 3.33 16.92 19.02
10 3.25 3.94 18.31 20.48
11 3.82 4.58 19.68 21.92
12 4.40 5.23 21.03 23.34
13 5.01 5.89 22.36 24.74
14 5.63 6.57 23.69 26.12
15 6.26 7.26 25.00 27.49

This Table shows z,n values such that P (2n z,n ) = .

( For example, P (210 3.94) = 95% )

411
 The T - distribution Table
n = 0.1 = 0.05 =0.01 = 0.005
5 -1.476 -2.015 -3.365 -4.032
6 -1.440 -1.943 -3.143 -3.707
7 -1.415 -1.895 -2.998 -3.499
8 -1.397 -1.860 -2.896 -3.355
9 -1.383 -1.833 -2.821 -3.250
10 -1.372 -1.812 -2.764 -3.169
11 -1.363 -1.796 -2.718 -3.106
12 -1.356 -1.782 -2.681 -3.055
13 -1.350 -1.771 -2.650 -3.012
14 -1.345 -1.761 -2.624 -2.977
15 -1.341 -1.753 -2.602 -2.947

This Table shows t,n values such that P (Tn t,n ) = .

( For example, P (T10 2.764) = 1% )

412