Two Interpretations of Probability

The Frequentist Interpretation:

Probability (outcome k) = Relative Frequency of k
i.e. if S = set of all outcomes
n = # of trials = # of times k occurs in n trials
then

N k (n )
P k  lim
n  n

The Probabilistic/Axiomatic Approach:

Probabilities are numerical values assigned to outcomes of set S
such that axioms of probability are satisfied. The axioms are
a) All probabilities are between 0 and 1.
b) The probabilities sum to 1 over the set S.
c) Probabilities of mutually exclusive events are additive.
 Definitions
• Random Experiment: An experiment with random results.

• Outcome: An elementary result from a random experiment.

• Sample Space: The space S of all possible outcomes in a random experiment.

sample spaces can be:
__ Finite or countable infinite
__ Uncountable

• Event: A subset of S.
Certain event = S
Null Event = Ф
e.g. Random Experiment : Toss of a coin twice
Outcomes = HH,HT,TH,TT
Sample Space = {HH,HT,TH,TT}
Event : At least one head = {HH,HT,TH}
 The Axioms of Probability

1) 0  P ( A )  A S

2) P(S) = 1

3) If, for A , B  S , A  B   Probabilities of disjoint events

add together
then P ( A  B )  P ( A )  P ( B )

For Infinite Sample Spaces (3) must be replaced by:

3a) If A 1 , A 2 ,   S such that


Ai  A j     
 i  j then P U Ak  
 k 1 

k 1
P ( Ak )

 n 
Since Axiom 3 only lets us evaluate Lim P   A k  which may not be equal to
n 
 k 1 
  
P U Ak 
 k 1 
 Some Elementary Consequences
1) P(A)  1  A S

2) P ( A ')  1  P ( A) where A'  S  A

3) P ( )  0
n n
4) P (U A k ) 
k 1

j 1
P(Aj)  
j k
P ( A j  Ak ) Subtract even-numbered combinations,
add odd-numbered ones.

 
j k l
P ( A j  Ak  Al )

 .......... .... A1 A2

 (  1 ) n  1 P ( A 1  A 2  ....  A n )

 P ( A  B )  P ( A)  P (B )  P ( A  B ) A3

5) If A  B P ( A)  P (B ) S
Conditional Probability:
P(A | B) = Probability that A occurs given B has occurred.

P(A  B)
P(A | B)  ; P (B )  0
P (B )

Independent Events:

A,B  S are independent if,

P (A | B )  P ( A)
P (B | A)  P (B )

Which leads to
P ( A  B )  P ( A)P (B )
Note that Independence  Mutual exclusion

In fact, if P(A), P(B) > 0, independent  A,B are not mutually exclusive.
Bayes’ Rule:
n
If S U Bk and A  S
k 1

B k form a partition of S.

Bi  B j  if i  j

P(A  B j) P ( A | B j) P (B j)
P (B j | A)   n
P(A)

k 1
P ( A | Bk )P (Bk )

The form
P (A | B j) P (B j)
P (B j | A) 
P(A)

is widely used in estimation.

P ( B j ) and P ( A ) are called prior probabilities.

Bernoulli Trial: A single trial of an experiment whose only outcomes are
“success” and “failure” or {0,1} etc.

If p = P(success) then
P(k successes in n independent Bernoulli Trials)
n
 Pn ( k )    p k
( 1  p )nk k  0 ,1 , 2 ,..., n
k 
 
This is called binomial probability law.

Example: Coin toss with “success” = heads

Fair coin  p = P(heads) = 0.5
P (3 heads in 5 tosses)  P5 ( 3 )

5 
   0 .5 3 ( 1  0 .5 ) 2
3 
 
5!
 ( 0 . 125 )( 0 . 25 )  0 . 3125
3! 2 !
Logic of the Binomial Law
n
Pn ( k )    p k
( 1  p )nk
k 
 
Number of ways Prob of Prob of
to get k successes k successes n-k failures
in n attempts

Multinomial Probability Law:

If  B k  1m is a partition of S
and rk = Number of times Bk occurs in n trials.
then, n!
P ( r1 , r 2 , ....., r m )  P1 r1 P 2r 2 .... P mr m
r1 ! r 2 ! ..... r m !

where Pk  P ( B k ) .

The binomial law is the m=2 case.

 Random Variables
A Random Variable X is a function that assigns a real number, X(), to each
Outcome, , of a random experiment: X : S  

Example: Random Experiment = 3 coin tosses

X  Substitute H = 1 and T = 0 and read as a binary number.

So X(HHH) = 7 (111)
X(TTT) = 0 (000)
X(HTH) = 5 (101) etc

Example: Random Experiment = Examine patient

Outcomes = {Healthy, Sick}

X(Healthy) = 0 X(Sick)=1

A random variable is a deterministic function, not a random one.

Why bother?
 Cumulative Distribution Function (CDF)
The CDF describes how probability accumulates over the range of a random variable
FX (x)  P (X  x)    x 

Thus F X ( x )  P ({  : X (  )  x })

e.g. in the coin toss example:

F X (3)  P ( X  3)

 P ({TTT , TTH , THT , THH })

 P ({ 000 , 001 , 010 , 011 })
 0 . 5 for a fair coin
Properties of CDF:

1) 0  F X ( x )  1

2) lim FX (x) 1
x 

3) lim FX (x)  0
x 

4) Nondecreasing in x :
a  b  F X ( a )  F X (b )

5) Right – continuous : F X ( a )  lim F X ( a  h )  F X ( a  )

h 0

for h  0

6) P ( a  X  b )  F X ( b )  F X ( a )
a b

7) P ( X  a )  F X ( a )  F X ( a  )
The RHS is obtained from P ( X  a )  lim
h 0
F X (a )  F X (a  h )
  If FX (x) is also left-continuous at a  P(X=a) = 0

For continuous FX (x) at a,b

P( a  X  b )  P( a  X  b )
 P( a  X  b )
 P( a  X  b )

If FX (x) is not continuous at a,b, the equalities do not hold.

 Three Types of Random Variables

1) Discrete :

2) Continuous:

3) Mixed:
Probability Mass Function (PMF) :
For a discrete random variable X with range
S X  { x 1 , x 2 ,....}

PMF  { p X (xk )  P ( X  xk ): xk  S X }

Probability Density Function(PDF):

d FX (x)
fX (x) 
d x
for differentiable F X ( x. )

P ( x  X  x  h )  hf X (x)
as h  0

i.e.
P ( x  X  x  dx )  f X ( x ) dx
Properties of PDF:

1) fX (x)  0
b
2) P (a  X  b)   f X ( x ) dx
a
x
3) FX (x)  

f X ( s ) ds

4)


fX (x) 1


Any function g(x) such that 


g ( x ) dx  c and g (x)  0 x
can form a valid pdf
1
fX (x)  g (x)
c
For Discrete Random Variables pdf is defined using the delta
function
f X ( x)   p X ( xk ) ( x  xk )
k
 x
0 x  0
u(x)  

 ( t ) dt  
1 x 0

 (x  xk )
 (x  x k ) dx 1

xk
xk

FX ( x)   p X ( xk )u ( x  xk )
k

pmf cdf
 Conditional CDF’s and PDF’s

P ({ X  x }  A )
FX (x | A)  ; A  S
P(A)

d
fX (x | A)  FX (x | A)
dx

Example 1:
FX (x | X  t)  P ( X  x | X  t)
0 if x  t
P ( X  x, X  t)  
 P (t  X  x ) else

0 x  t

FX ( x | X  t )   F X ( x )  F X (t )
 x  t
 1  F X (t )
Example 2: Rolling a fair 6-sided die
1
Fx (x)

Fx (x | X even)

1/6
0
0 1 2 3 4 5 6 7 x

P ({ X  2 }  X even)
F X ( x | X even) 
P ( X even)
P ({ X  2 })

P ( X even)
1/ 6
  1/ 3
1/ 2