1.

Probability Basics
Dave Goldsman
Georgia Institute of Technology, Atlanta, GA, USA
10/19/10
Goldsman 10/19/10 1 / 98
Outline
1
Intro / Examples
2
Set Theory
3
Experiments
4
Probability
5
Finite Sample Spaces
6
Counting Techniques
7
Counting Applications
8
Conditional Probability and Independence
9
Bayes Theorem
Goldsman 10/19/10 2 / 98
Intro / Examples
Mathematical Models for describing observable phenomena:
Deterministic
Probabilistic
Deterministic Models
Ohms Law (I = E/R)
Drop an object from height h
0
. After t sec, height is
h(t) = h
0
16t
2
.
Deposit $1000 in a continuously compounding checking 3%
account. At time t, its worth $1000e
.03t
.
Goldsman 10/19/10 3 / 98
Intro / Examples
Probabilistic Models Involve uncertainty
How much snow will fall tomorrow?
Will IBM make a prot this year?
Should I buy a call or put option?
Can I win in blackjack if I use a certain strategy?
What is the cost-effectiveness of a new drug?
Which horse will win the Kentucky Derby?
Goldsman 10/19/10 4 / 98
Intro / Examples
Some Cool Examples
1. Birthday Problem Assume all 365 days have equal probability of
being a persons birthday (ignore Feb 29). Then. . .
If there are 23 people in the room, the odds are better than 5050 that
there will be a match.
If there are 50 people, the probability is about 97%!
2. Monopoly In the long run, the property having the highest
probability of being landed on is Illinois Ave.
Goldsman 10/19/10 5 / 98
Intro / Examples
3. Poker Pick 5 cards from a standard deck. Then
P(exactly 2 pairs) 0.0475
P(full house) 0.00144
P(ush) 0.00198
4. Stock Market Monkeys randomly selecting stocks could have
outperformed most market analysts during the past year.
Goldsman 10/19/10 6 / 98
Intro / Examples
5. A couple has two kids and at least one is a boy. Whats the
probability that BOTH are boys?
Possibilities: GG, BG, GB, BB. Eliminate GG since we know that
theres at least one boy. Then P(BB) = 1/3.
6. Vietnam Lottery
7. Ask Marilyn. You are a contestant at a game show. Behind one of
three doors is a car; behind the other two are goats. You pick door A.
Monty Hall opens door B and reveals a goat. Monty offers you a
chance to switch to door C. What should you do?
Goldsman 10/19/10 7 / 98
Intro / Examples
Working Denitions
Probability Methodology that describes the random variation in
systems. (Well spend about 40% of our time on this.)
Statistics Uses sample data to draw general conclusions about the
population from which the sample was taken. (60% of our time.)
Goldsman 10/19/10 8 / 98
Set Theory
Outline
1
Intro / Examples
2
Set Theory
3
Experiments
4
Probability
5
Finite Sample Spaces
6
Counting Techniques
7
Counting Applications
8
Conditional Probability and Independence
9
Bayes Theorem
Goldsman 10/19/10 9 / 98
Set Theory
The Joy of Sets
Denition: A set is a collection of objects. Members of a set are called
elements.
Notation:
A, B, C, . . . for sets; a, b, c, . . . for elements
for membership, e.g., x A
/ for non-membership, e.g., x / A
U is the universal set (i.e., everything)
is the empty set.
Goldsman 10/19/10 10 / 98
Set Theory
Examples:
A = 1, 2, . . . , 10. 2 A, 49 / A.
B = basketball, baseball
C = x[0 x 1 ([ means such that)
D = x[x
2
= 9 = 3 (either is ne)
E = x[x ', x
2
= 1 = (' is the real line)
Goldsman 10/19/10 11 / 98
Set Theory
Denition: If every element of set A is an element of set B then A is a
subset of B, i.e., A B.
Denition: A = B iff (if and only if) A B and B A.
Properties:
A; A U; A A
A B and B C (implies) A C
Remark: Order of element listing is immaterial, e.g, a, b, c = b, c, a.
Goldsman 10/19/10 12 / 98
Set Theory
Denitions: Complement of A with respect to U is

A x[x U and x / A.
Intersection of A and B is A B x[x A and x B.
Union of A and B is A B x[x A or x B (or both).
(Remember Venn diagrams?)
Goldsman 10/19/10 13 / 98
Set Theory
Example:
Suppose U = letters of the alphabet, A = vowels, and
B = a, b, c. Then

A = consonants
A B = a
A B = a, b, c, e, i, o, u
If A B = , then A and B are disjoint (or mutually exclusive).
Goldsman 10/19/10 14 / 98
Set Theory
Denitions:
Minus: AB A

B
Symmetric difference or XOR:
AB (AB) (B A) = (A B) (A B)
The cardinality of A, [A[, is the number of elements in A. A is nite if
[A[ < .
Examples:
A = 3, 4 is nite since [A[ = 2.
B = 1, 2, 3, . . . is countably innite.
C = x[x [0, 1] is uncountably innite.
Goldsman 10/19/10 15 / 98
Set Theory
Laws of Operation:
1. Complement Law: A

A = U, A

A = ,

A = A
2. Commutative: A B = B A, A B = B A
3. DeMorgans: A B =

A

B, A B =

A

B
4. Associative: A (B C) = (A B) C,
A (B C) = (A B) C
5. Distributive: A (B C) = (A B) (A C),
A (B C) = (A B) (A C)
Proofs: Easy. Could use Venn diagrams or many other ways.
Goldsman 10/19/10 16 / 98
Experiments
Outline
1
Intro / Examples
2
Set Theory
3
Experiments
4
Probability
5
Finite Sample Spaces
6
Counting Techniques
7
Counting Applications
8
Conditional Probability and Independence
9
Bayes Theorem
Goldsman 10/19/10 17 / 98
Experiments
Experiments and Sample Spaces
Consider a random experiment:
E
1
: Toss a die and observe the outcome.
Denition: A sample space associated with an experiment E is the set
of all possible outcomes of E. Its usually denoted by S or .
Goldsman 10/19/10 18 / 98
Experiments
Examples:
E
1
has sample space S
1
= 1, 2, 3, 4, 5, 6.
Another sample space for E
1
is S

1
= odd, even.
So a sample space doesnt have to be unique!
E
2
: Toss a coin 3 times and observe the sequence of Hs and Ts.
S
2
= HHH, HHT, HTH, HTT, THH, THT, TTH, TTT.
E
3
: A new light bulb is tested to see how long it lasts.
S
3
= t[t 0.
Goldsman 10/19/10 19 / 98
Experiments
Denition: An event is a set of possible outcomes. Thus, any subset of
S is an event.
Example (for E
1
): If A
1
is the event an even number occurs, then
A
1
= 2, 4, 6, i.e., when the die is tossed, we get 2 or 4 or 6.
Remark: is an event of S (nothing happens)
S is an event of S (something happens)
Remark: If A is an event, then

A is the complementary (opposite)
event.
Example (for E
1
):
A
1
= 2, 4, 6

A
1
= 1, 3, 5 (i.e., an odd number occurs)
Goldsman 10/19/10 20 / 98
Experiments
Remark: If A and B are events, then A B and A B are events.
Example (for E
2
): Let
A
2
= exactly one T was observed
= HHT, HTH, THH
B
2
= no Ts observed = HHH
C
2
= rst coin is H
= HHH, HHT, HTH, HTT
Then
A
2
B
2
= at most one T observed
= HHT, HTH, THH, HHH
A
2
C
2
= HHT, HTH
Goldsman 10/19/10 21 / 98
Probability
Outline
1
Intro / Examples
2
Set Theory
3
Experiments
4
Probability
5
Finite Sample Spaces
6
Counting Techniques
7
Counting Applications
8
Conditional Probability and Independence
9
Bayes Theorem
Goldsman 10/19/10 22 / 98
Probability
Probability Basics (see Meyer 1970)
Suppose A is some event for a sample space S. Whats the prob that
A will occur, i.e., P(A)?
Example: Toss a fair coin. S = H, T. Whats the prob that H will
come up?
P(H) = P(H) = 1/2.
What does this mean?
Goldsman 10/19/10 23 / 98
Probability
Frequentist view: If the experiment were repeated n times, where n is
very large, wed expect about 1/2 of the tosses to be Hs.
Total # of Hs out of n tosses
n
1/2.
Example: Toss a fair die. S = 1, 2, 3, 4, 5, 6, where each individual
outcome has prob 1/6. Then P(1, 2) = 1/3.
Goldsman 10/19/10 24 / 98
Probability
Denition: With each event A S, we associate a number P(A),
called the probability of A, satisfying the following axioms:
(1) 0 P(A) 1 (probs are always betw. 0 and 1).
(2) P(S) = 1 (prob of some outcome is 1).
Example: Die. P(S) = P(1, 2, 3, 4, 5, 6) = 1.
(3) If A B = , then P(A B) = P(A) +P(B).
Example: P(1 or 2) = P(1) +P(2) = 1/6 + 1/6 = 1/3.
(4) Suppose A
1
, A
2
, . . . is a sequence of disjoint events (i.e.,
A
i
A
j
= for i ,= j). Then
P
_

_
i=1
A
i
_
=

i=1
P(A
i
).
Goldsman 10/19/10 25 / 98
Probability
Example: Toss a coin until the rst H appears.
S = H, TH, TTH, TTTH, . . ..
Dene the disjoint events
A
1
= H, A
2
= TH, A
3
= TTH, . . . .
Then
1 = P(S) = P
_

_
i=1
A
i
_
=

i=1
P(A
i
).
Goldsman 10/19/10 26 / 98
Probability
More Nifty Properties
Theorem 1: P() = 0.
Proof: Since A = , we have that A and are disjoint. So Axiom (3)
implies
P(A) = P(A ) = P(A) +P().
Note: Converse is false: P(A) = 0 does not imply A = .
Example: Pick a random number betw. 0 and 1.
Goldsman 10/19/10 27 / 98
Probability
Theorem 2: P(

A) = 1 P(A).
Proof:
1 = P(S) (by Axiom (2))
= P(A

A)
= P(A) +P(

A) (A

A = ; Axiom (3)).
Example: The probability that itll rain tomorrow is one minus the
probability that it wont rain.
Goldsman 10/19/10 28 / 98
Probability
Theorem 3: For any two events A and B,
P(A B) = P(A) +P(B) P(A B).
Proof: First observe that B = (A B) (

A B) where A B and

A B are disjoint. Thus,

P(B) = P(A B) +P(

A B) ()
and so
P(A B) = P(A) +P(

A B) (A,

A B are disjoint)
= P(A) +P(B) P(A B) (by ()).
Remark: Can also do an easy Venn diagram proof. (Subtract P(A B)
to avoid double-counting.)
Remark: Axiom (3) is a special case of this theorem with A B = .
Goldsman 10/19/10 29 / 98
Probability
Example: Suppose theres. . .
40% chance of colder weather
10% chance of rain and colder weather
80% chance of rain or colder weather.
Find the chance of rain.
P(R) = P(R C) P(C) +P(R C)
= 0.8 0.4 + 0.1 = 0.5.
Goldsman 10/19/10 30 / 98
Probability
Theorem 4: For any three events A, B, and C,
P(A B C)
= P(A) +P(B) +P(C)
P(A B) P(A C) P(B C)
+P(A B C)
Goldsman 10/19/10 31 / 98
Probability
Example: 75% of Atlantans jog (J), 20% like ice cream (I), and 40%
enjoy music (M). Also, 15% J and I, 30% J and M, 10% I and M,
and 5% do all three. Find the prob that a random resident will engage
in at least one of the three activities.
P(J I M)
= P(J) +P(I) +P(M)
P(J I) P(J M) P(I M)
+P(J I M)
= .75 +.20 +.40 .15 .30 .10 +.05 = .85.
Find the prob of precisely one activity.
P(J

I

M) +P(

J I

M) +P(

J

I M)
= .35 + 0 +.05 = .40.
Howd we get those?? Use Venn diagram, starting from the center and
building out.
Goldsman 10/19/10 32 / 98
Probability
Remark: Here is the general principle of inclusion-exclusion:
P(A
1
A
2
A
n
)
=
n

i=1
P(A
i
)

i<j
P(A
i
A
j
)
+

i<j<k
P(A
i
A
j
A
k
)
+ + (1)
n1
P(A
1
A
2
A
n
).
Goldsman 10/19/10 33 / 98
Probability
(Bonus) Theorem 5: A B P(A) P(B).
Proof:
P(B) = P(A (

A B))
= P(A) +P(

A B)
P(A).
Remark: A B means that B occurs whenever A occurs; so the
Theorem makes intuitive sense.
Goldsman 10/19/10 34 / 98
Finite Sample Spaces
Outline
1
Intro / Examples
2
Set Theory
3
Experiments
4
Probability
5
Finite Sample Spaces
6
Counting Techniques
7
Counting Applications
8
Conditional Probability and Independence
9
Bayes Theorem
Goldsman 10/19/10 35 / 98
Finite Sample Spaces
Finite Sample Spaces
Suppose S is nite, say S = a
1
, a
2
, . . . , a
n
.
Let B be an event consisting of r ( n) outcomes, i.e.,
B = a
j
1
, a
j
2
, . . . , a
j
r
, where the j
i
s represent r indices from
1, 2, . . . , n. Then P(B) =

r
i=1
P(a
j
i
).
Note: Choosing an object at random means that each object has the
same prob of being chosen.
Example: You have 2 red cards, a blue card, and a yellow. Pick one
card at random.
S = red, blue, yellow = a
1
, a
2
, a
3

P(a
1
) = 1/2, P(a
2
) = 1/4, P(a
3
) = 1/4.
P(red or yellow) = P(a
1
) +P(a
3
) = 3/4.
Goldsman 10/19/10 36 / 98
Finite Sample Spaces
Denition: A simple sample space (SSS) is a nite sample space in
which all outcomes are equally likely.
Remark: In the above example, S is not simple since P(a
1
) ,= P(a
2
).
Example: Toss 2 fair coins.
S = HH, HT, TH, TT is a SSS (all probs are 1/4).
S

= 0, 1, 2 (number of Hs) is not a SSS. Why?

Theorem: For any event A in a SSS S,
P(A) =
[A[
[S[
=
# elements in A
# elements in S
.
Example: Die. A = 1, 2, 4, 6 (each with prob 1/6). P(A) = 4/6.
Goldsman 10/19/10 37 / 98
Finite Sample Spaces
Example: Roll a pair of dice. Possible results (each w.p. 1/36):
1,1 1,2 1,6
2,1 2,2 2,6
.
.
.
6,1 6,2 6,6
Sum 2 3 4 5 6 7 8 9 10 11 12
Prob
1
36
2
36
3
36
4
36
5
36
6
36
5
36
4
36
3
36
2
36
1
36
Goldsman 10/19/10 38 / 98
Counting Techniques
Outline
1
Intro / Examples
2
Set Theory
3
Experiments
4
Probability
5
Finite Sample Spaces
6
Counting Techniques
7
Counting Applications
8
Conditional Probability and Independence
9
Bayes Theorem
Goldsman 10/19/10 39 / 98
Counting Techniques
Counting Techniques count the elements in events from a SSS.
Multiplication Rule
Addition Rule
Permutations
Combinations
Goldsman 10/19/10 40 / 98
Counting Techniques
Multiplication Rule
Two operations are performed one after the other:
(a) The rst operation can be done in n
1
ways.
(b) Regardless of the way in which the rst operation was performed,
the second can be performed in n
2
ways.
The # ways to perform the two operations together is n
1
n
2
.
Example: 3 ways to go from City A to B, and 4 ways to go from B to C.
Then the you can go from A to C (via B) in 12 ways.
Example: Roll 2 dice. How many outcomes? (Assume (3, 2) ,= (2, 3).)
Answer is 36.
Goldsman 10/19/10 41 / 98
Counting Techniques
Example: Select 2 cards from a deck without replacement and care
about order (i.e., (Q, 7) ,= (7, Q). How many ways can you do
this? Answer: 52 51 = 2652.
Example: Box of 10 sox 2 red and 8 black. Pick 2 w/o repl.
(a) Let A be the event that both are red.
P(A) =
# ways to pick 2 reds
# ways to pick 2 sox
=
2 1
10 9
=
1
45
.
(b) Let B be the event that both are black. Similarly, P(B) =
87
109
=
28
45
.
(c) Let C be one of each color.
P(C) = 1 P(

C)
= 1 P(A B)
= 1 P(A) P(B) (A and B disjoint)
= 16/45.
Goldsman 10/19/10 42 / 98
Counting Techniques
Remark: The multiplication rule can be extended to more than 2
operations.
Example: Flip 3 coins. 2 2 2 = 8 possible outcomes.
Example: Toss n dice. 6
n
possible outcomes.
Goldsman 10/19/10 43 / 98
Counting Techniques
Addition Rule
Can use method A in n
A
ways.
Can use method B in n
B
ways.
If only one method can be used, you have n
A
+n
B
ways of doing so.
Example: Go to Starbucks and have a mufn (blueberry or oatmeal) or
a bagel (sesame, plain, salt), but not both. 2 + 3 = 5 choices.
Goldsman 10/19/10 44 / 98
Counting Techniques
Permutations
Denition: An arrangement of n symbols in a denite order is a
permutation of the n symbols.
Example: How many ways to arrange the numbers 1,2,3?
Answer: 6 ways 123, 132, 213, 231, 312, 321.
Example: How many ways to arrange 1, 2, . . . , n?
(choose rst)(choose second) (choose nth)
n(n 1)(n 2) 2 1 = n!.
Example: Baseball manager has 9 players on his team. Find the # of
possible batting orders. Answer: 9! = 362880.
Goldsman 10/19/10 45 / 98
Counting Techniques
Denition: The # of r-tuples we can make from n different symbols
(each used at most once) is called the # of permutations of n things
taken r-at-a-time,
P
n,r

n!
(n r)!
().
Note that 0! = 1 and P
n,n
= n!.
Example: How many ways can you take two symbols from a, b, c, d?
Ans: P
4,2
= 4!/2! = 12 ab, ac, ad, ba, bc, bd, ca, cb, cd, da, db, dc.
Goldsman 10/19/10 46 / 98
Counting Techniques
Proof (of ()):
P
n,r
= (choose rst)(second) (rth)
= n(n 1)(n 2) (n r + 1)
=
n(n 1) (n r + 1)(n r) 2 1
(n r) 2 1
=
n!
(n r)!
.
Goldsman 10/19/10 47 / 98
Counting Techniques
Example: How many ways to ll the rst 4 positions of a batting order?
n = 9 players, r = 4 positions.
P
9,4
= 9!/(9 4)! = 3024 ways.
Example: How many of these 3024 ways has Smith batting rst?
Method 1: First 4 positions: (Smith,?,?,?). This is equiv to choosing 3
players from the remaining 8.
P
8,3
= 8!/(8 3)! = 336 ways.
Method 2: Its clear that each of the 9 players is equally likely to bat
rst. Thus, 3024/9 = 336.
Goldsman 10/19/10 48 / 98
Counting Techniques
Example: How many license plates of 6 digits can be made from the
numbers 1,2,. . . ,9. . .
(a) with no repetitions? (e.g., 123465) P
9,6
= 9!/3! = 60480.
(b) allowing repetitions? (e.g., 123345 or 123465)
9 9 9 = 9
6
= 531441.
(c) containing repetitions? 531441 60480 = 470961.
Goldsman 10/19/10 49 / 98
Counting Techniques
Combinations
Suppose we only want to count the number of ways to choose r out of
n objects without regard to order, i.e., count the number of different
subsets of these n objects that contain exactly r objects.
Example: How many subsets of 1, 2, 3 contain exactly 2 elements?
(Order isnt important.)
3 subsets 1, 2, 1, 3, 2, 3
Denition: The # of subsets with r elements of a set with n elements is
called the number of combinations of n things taken r-at-a-time.
Notation:
_
n
r
_
or C
n,r
(read as n choose r). These are also called
binomial coefcients.
Goldsman 10/19/10 50 / 98
Counting Techniques
Difference between permutations and combinations:
Combinations not concerned w/order: (a, b, c) = (b, a, c).
Permutations concerned w/order: (a, b, c) ,= (b, a, c).
The number of permutations of n things taken r-at-a-time is always as
least as large as the number of combinations. In fact,. . .
Goldsman 10/19/10 51 / 98
Counting Techniques
Remark: Choosing a permutation is the same as rst choosing a
combination and then putting the elements in order, i.e.,
n!
(n r)!
=
_
n
r
_
r!
So
_
n
r
_
=
n!
(n r)!r!
.
In particular,
_
n
r
_
=
_
n
n r
_
,
_
n
0
_
=
_
n
n
_
= 1,
_
n
1
_
=
_
n
n 1
_
= n.
Goldsman 10/19/10 52 / 98
Counting Techniques
Binomial Theorem:
(x +y)
n
=
n

i=0
_
n
i
_
x
i
y
ni
This is where Pascals comes from!
Corollary: Surprising fact:
n

i=0
_
n
i
_
= 2
n
.
Proof:
2
n
= (1 + 1)
n
=
n

i=0
_
n
i
_
1
i
1
ni
.
Goldsman 10/19/10 53 / 98
Counting Techniques
Example: An NBA team has 12 players. How many ways can the
coach choose the starting 5?
_
12
5
_
=
12!
5!7!
= 792.
Example: Smith is one of the players on the team. How many of the
792 starting line-ups include him?
_
11
4
_
=
11!
4!7!
= 330.
(Smith gets one of the ve positions for free; there are now 4 left to be
lled by the remaining 11 players.)
Goldsman 10/19/10 54 / 98
Counting Techniques
Example: 7 red shoes, 5 blues. Find the number of arrangements.
R B R R B B R R R B R B
I.e., how many ways to put 7 reds in 12 slots?
Answer:
_
12
7
_
.
Goldsman 10/19/10 55 / 98
Counting Applications
Outline
1
Intro / Examples
2
Set Theory
3
Experiments
4
Probability
5
Finite Sample Spaces
6
Counting Techniques
7
Counting Applications
8
Conditional Probability and Independence
9
Bayes Theorem
Goldsman 10/19/10 56 / 98
Counting Applications
Some applications of counting techniques.
Hypergeometric problems
Binomial problems
Permutations vs. Combinations
Birthday problem
Envelope problem
Poker probabilities
Multinomial coefcients
Goldsman 10/19/10 57 / 98
Counting Applications
Hypergeometric Distribution
You have a objects of type 1 and b objects of type 2.
Select n objects without replacement from the a +b.
P(k type 1s were picked)
=
(# ways to choose k 1s)(choose n k 2s)
# ways to choose n out of a +b
=
_
a
k
__
b
nk
_
_
a+b
n
_ (the hypergeometric distrn).
Example: 25 sox in a box. 15 red, 10 blue. Pick 7 w/o replacement.
P(exactly 3 reds are picked) =
_
15
3
__
10
4
_
_
25
7
_
Goldsman 10/19/10 58 / 98
Counting Applications
Binomial Distribution
You again have a objects of type 1 and b objects of type 2.
Now select n objects with replacement from the a +b.
P(k type 1s were picked)
= (# ways to choose k 1s and n k 2s)
P(choose k 1s in a row, then n k 2s in a row)
=
_
n
k
_
_
a
a +b
_
k
_
b
a +b
_
nk
(binomial distrn).
Example: 25 sox in a box. 15 red, 10 blue. Pick 7 with replacement.
P(exactly 3 reds are picked) =
_
7
3
_
_
15
25
_
3
_
10
25
_
73
Well have more to say about the Binomial distribution later.
Goldsman 10/19/10 59 / 98
Counting Applications
Permutations vs. Combinations Its all how you approach the
problem!
Example: 4 red marbles, 2 whites. Put them in a row in random order.
Find. . .
(a) P(2 end marbles are W)
(b) P(2 end marbles arent both W)
(c) P(2 Ws are side by side)
Goldsman 10/19/10 60 / 98
Counting Applications
Method 1 (using permutations): Let the sample space
S = every random ordering of the 6 marbles.
(a) A: 2 end marbles are W WRRRRW.
[A[ = 2!4! = 48 P(A) =
[A[
[S[
=
48
720
=
1
15
.
(b) P(

A) = 1 P(A) = 14/15.
Goldsman 10/19/10 61 / 98
Counting Applications
(c) B: 2 Ws side by side WWRRRR or RWWRRR or . . . or
RRRRWW
[B[ = (# ways to select pair of slots for 2 Ws)
(# ways to insert Ws into pair of slots)
(# ways to insert Rs into remaining slots)
= 5 2! 4! = 240.
P(B) =
[B[
[S[
=
240
720
=
1
3
.
But The above method took too much time! Heres an easier way. . .
Goldsman 10/19/10 62 / 98
Counting Applications
Method 2 (using combinations): Which 2 positions do the Ws occupy?
Now let
S = possible pairs of slots that the Ws occupy.
Clearly, [S[ =
_
6
2
_
= 15.
(a) Since the Ws must occupy the end slots in order for A to occur,
[A[ = 1 P(A) = [A[/[S[ = 1/15.
(b) P(

A) = 14/15.
(c) [B[ = 5 P(B) = 5/15 = 1/3.
Goldsman 10/19/10 63 / 98
Counting Applications
Birthday Problem
n people in a room. Find the prob that at least two have the same
birthday. (Ignore Feb. 29, and assume that all 365 days have equal
prob.)
A: All birthdays are different.
S = (x
1
, . . . , x
n
) : x
i
= 1, 2, . . . , 365 (x
i
is person is birthday), and
note that [S[ = (365)
n
.
Goldsman 10/19/10 64 / 98
Counting Applications
[A[ = P
365,n
= (365)(364) (365 n + 1)
P(A) =
(365)(364) (365 n + 1)
(365)
n
= 1
364
365

363
365

365 n + 1
365
We want
P(

A) = 1
_
1
364
365

363
365

365 n + 1
365
_
Notes: When n = 366, P(

A) = 1.
For P(

A) to be > 1/2, n must be 23. (surprising)
When n = 50, P(

A) = 0.97.
Goldsman 10/19/10 65 / 98
Counting Applications
Envelope Problem
A group of n people receives n envelopes with their names on them
but someone has completely mixed up the envelopes! Find the prob
that at least one person will receive the proper envelope.
A
i
: Person i receives his correct envelope.
We obviously want P(A
1
A
2
A
n
).
Goldsman 10/19/10 66 / 98
Counting Applications
By the general principle of inclusion-exclusion, we have. . . .
P(A
1
A
2
A
n
)
=
n

i=1
P(A
i
)

i<j
P(A
i
A
j
)
+

i<j<k
P(A
i
A
j
A
k
)
+ + (1)
n1
P(A
1
A
2
A
n
)
= nP(A
1
)
_
n
2
_
P(A
1
A
2
) +
_
n
3
_
P(A
1
A
2
A
3
)
+ + (1)
n1
P(A
1
A
2
A
n
)
=
n
n

_
n
2
_
1
n

1
n 1
+
_
n
3
_
1
n

1
n 1

1
n 2
+ + (1)
n1
1
n!
= 1
1
2!
+
1
3!
+ + (1)
n1
1
n!
1
1
e
.
Goldsman 10/19/10 67 / 98
Counting Applications
Poker Problems
Draw 5 cards at random from a standard deck.
# of possible hands is [S[ =
_
52
5
_
= 2, 598, 960.
Terminology:
rank = 2, 3, . . . , Q, K, A,
suit = , , ,
Goldsman 10/19/10 68 / 98
Counting Applications
(a) 2 pairs e.g., A, A, 3, 3, 10
Select 2 ranks (e.g., A, 3). Can do this
_
13
2
_
ways.
Select 2 suits for rst pair (e.g., , ).
_
4
2
_
ways.
Select 2 suits for second pair (e.g., , ).
_
4
2
_
ways.
Select remaining card to complete the hand. 44 ways.
[2 pairs[ =
_
13
2
__
4
2
__
4
2
_
44 = 123, 552
P(2 pairs) =
123, 552
2, 598, 960
0.0475.
Goldsman 10/19/10 69 / 98
Counting Applications
(b) Full house (1 pair, 3-of-a-kind) e.g., A, A, 3, 3, 3
Select 2 ordered ranks (e.g., A, 3). P
13,2
ways.
Select 2 suits for pair (e.g., , ).
_
4
2
_
ways.
Select 3 suits for 3-of-a-kind (e.g., , , ).
_
4
3
_
ways.
[full house[ = 13 12
_
4
2
__
4
3
_
= 3744
P(full house) =
3744
2, 598, 960
0.00144.
Goldsman 10/19/10 70 / 98
Counting Applications
(c) Flush (all 5 cards from same suit)
Select a suit.
_
4
1
_
ways.
Select 5 cards from that suit.
_
13
5
_
ways.
P(ush) =
5148
2, 598, 960
0.00198.
(d) Straight (5 ranks in a row)
Select a starting point for the straight (A, 2, 3, . . . , 10).
_
10
1
_
ways.
Select a suit for each card in the straight. 4
5
ways.
P(straight) =
10 4
5
2, 598, 960
0.00394.
Goldsman 10/19/10 71 / 98
Counting Applications
(e) Straight ush
Select a starting point for the straight. 10 ways.
Select a suit. 4 ways.
P(straight ush) =
40
2, 598, 960
0.0000154.
Remark: Can you do bridge problems? Yahtzee?
Goldsman 10/19/10 72 / 98
Counting Applications
Multinomial Coefcients
Example: n
1
blue sox, n
2
reds. # of assortments is
_
n
1
+n
2
n
1
_
(binomial
coefcients).
Generalization (for k types of objects): n =

k
i=1
n
i
# of arrangements is n!/(n
1
!n
2
! n
k
!).
Example: How many ways can Mississippi be arranged?
# perms of 11 letters
(# ms)!(# ps)!(# is)!(# ss)!
=
11!
1!2!4!4!
= 34, 650.
Goldsman 10/19/10 73 / 98
Conditional Probability and Independence
Outline
1
Intro / Examples
2
Set Theory
3
Experiments
4
Probability
5
Finite Sample Spaces
6
Counting Techniques
7
Counting Applications
8
Conditional Probability and Independence
9
Bayes Theorem
Goldsman 10/19/10 74 / 98
Conditional Probability and Independence
Conditional Probability
Example: Die. A = 2, 4, 6, B = 1, 2, 3, 4, 5. So P(A) = 1/2,
P(B) = 5/6.
Suppose we know that B occurs. Then the prob of A given B is
P(A[B) =
2
5
=
[A B[
[B[
So the prob of A depends on the info that you have! The info that B
occurs allows us to regard B as a new, restricted sample space.
And. . .
P(A[B) =
[A B[
[B[
=
[A B[/[S[
[B[/[S[
=
P(A B)
P(B)
.
Goldsman 10/19/10 75 / 98
Conditional Probability and Independence
Denition: If P(B) > 0, the conditional prob of A given B is
P(A[B) P(A B)/P(B).
Remarks: If A and B are disjoint, then P(A[B) = 0. (If B occurs,
theres no chance that A can also occur.)
What happens if P(B) = 0? Dont worry! In this case, makes no sense
to consider P(A[B).
Goldsman 10/19/10 76 / 98
Conditional Probability and Independence
Example: Toss 2 dice and take the sum.
A: odd toss = 3, 5, 7, 9, 11
B: 2, 3
P(A) = P(3) + +P(11) =
2
36
+
4
36
+ +
2
36
=
1
2
.
P(B) =
1
36
+
2
36
=
1
12
.
P(A[B) =
P(A B)
P(B)
=
P(3)
P(B)
=
2/36
1/12
= 2/3.
Goldsman 10/19/10 77 / 98
Conditional Probability and Independence
Example: 4 white sox, 8 red. Select 2 w/o repl.
A: 1st sock W; B: 2nd W; C: Both W (= A B).
P(C) = P(A B) = P(A)P(B[A) =
4
12

3
11
=
1
11
.
P(B) = P(A B) +P(

A B)
= P(A)P(B[A) +P(

A)P(B[

A)
=
4
12

3
11
+
8
12

4
11
=
1
3
.
Could you have gotten this result w/o thinking?
Goldsman 10/19/10 78 / 98
Conditional Probability and Independence
Example: A couple has two kids and at least one is a boy. Whats the
prob that BOTH are boys?
S = GG, GB, BG, BB, (BG means boy then girl)
C: Both are boys = BB.
D: At least 1 is a boy. = GB, BG, BB.
P(C[D) =
P(C D)
P(D)
=
P(C)
P(D)
= 1/3.
(My intuition was 1/2 the wrong answer! The problem was that we
didnt know whether D meant the rst or second kid.)
Goldsman 10/19/10 79 / 98
Conditional Probability and Independence
Extremely Tough Example: A couple has two kids and at least one is a
boy born on a Tuesday. Whats the prob that BOTH are boys?
B
x
[G
x
] = Boy [Girl] born on day x, x = 1, 2, . . . , 7 (x = 3 is Tuesday).
S = (G
x
, G
y
), (G
x
, B
y
), (B
x
, G
y
), (B
x
, B
y
), x, y = 1, 2, . . . , 7
(so [S[ = 4 49 = 196)
C: Both are boys (with at least one born on a Tuesday)
= (B
x
, B
3
), x = 1, 2, . . . , 7 (B
3
, B
y
), y = 1, 2, . . . , 7.
Note that [C[ = 13 (to avoid double counting (B
3
, B
3
)).
D: Both boys (with at least one born on Tuesday)
= (G
x
, B
3
), (B
3
, G
y
), x, y = 1, 2, . . . , 7 C.
So [D[ = 27 (list em out if you dont believe me). Then
P(C[D) =
P(C D)
P(D)
=
P(C)
P(D)
=
13/196
27/196
= 13/27.
Goldsman 10/19/10 80 / 98
Conditional Probability and Independence
Properties analogous to Axioms of probability.
(1) 0 P(A[B) 1.
(2) P(S[B) = 1.
(3) A
1
A
2
= P(A
1
A
2
[B) = P(A
1
[B) +P(A
2
[B).
(4) If A
1
, A
2
, . . . are all disjoint, then P
_

i=1
A
i

B
_
=

i=1
P(A
i
[B).
Goldsman 10/19/10 81 / 98
Conditional Probability and Independence
Independence Day Any unrelated events are independent.
A: It rains on Mars tomorrow.
B: Coin lands on H.
Denition: A and B are independent iff P(A B) = P(A)P(B).
Example: If P(rains on Mars) = 0.2 and P(H) = 0.5, then
P(rains and H) = 0.1.
Remark: If P(A) = 0, then A is indep of any other event.
Remark: Events dont have to be physically unrelated to be indep.
Example: Die. A = 2, 4, 6, B = 1, 2, 3, 4, A B = 2, 4, so
P(A) = 1/2, P(B) = 2/3, P(A B) = 1/3.
P(A)P(B) = 1/3 = P(A B) A, B indep.
Goldsman 10/19/10 82 / 98
Conditional Probability and Independence
More natural interpretation of independence. . .
Theorem: Suppose P(B) > 0. Then A and B are indep iff
P(A[B) = P(A).
Proof: A, B indep P(A B) = P(A)P(B)
P(A B)/P(B) = P(A).
Remark: So if A and B are indep, the prob of A doesnt depend on
whether or not B occurs.
Goldsman 10/19/10 83 / 98
Conditional Probability and Independence
(Bonus) Theorem: A, B indep A,

B indep.
Proof: P(A) = P(A

B) +P(A B), so that
P(A

B) = P(A) P(A B)
= P(A) P(A)P(B) (A, B indep)
= P(A)[1 P(B)] = P(A)P(

B).
Goldsman 10/19/10 84 / 98
Conditional Probability and Independence
Dont confuse independence with disjointness!
Theorem: If P(A) > 0 and P(B) > 0, then A and B cant be indep and
disjt at the same time.
Proof: A, B disjt (A B = ) P(A B) = 0 < P(A)P(B). Thus A, B
not indep. Similarly, indep implies not disjt.
Goldsman 10/19/10 85 / 98
Conditional Probability and Independence
Extension to more than two events.
Denition: A, B, C are indep iff
(a) P(A B C) = P(A)P(B)P(C) and
(b) All pairs must be indep:
P(A B) = P(A)P(B)
P(A C) = P(A)P(C)
P(B C) = P(B)P(C)
Goldsman 10/19/10 86 / 98
Conditional Probability and Independence
Note that condition (a) by itself isnt enough.
Example: S = 1, 2, . . . , 8 (each element w.p. 1/8).
A = 1, 2, 3, 4, B = 1, 5, 6, 7, C = 1, 2, 3, 8.
(a) A B C = 1. P(A B C) = P(A)P(B)P(C) = 1/8, so (a) is
satised. However, (b) is not . . .
(b) A B = 1. P(A B) = 1/8 ,= 1/4 = P(A)P(B).
On the other hand, (b) by itself isnt enough.
Example: S = 1, 2, 3, 4 (each element w.p. 1/4).
A = 1, 2, B = 1, 3, C = 1, 4.
(b) P(A B) = 1/4 = P(A)P(B). Same deal with A, C and B, C. So
(b) is OK. But (a) isnt. . .
(a) P(A B C) = 1/4 ,= 1/8 = P(A)P(B)P(C).
Goldsman 10/19/10 87 / 98
Conditional Probability and Independence
General Denition: A
1
, . . . , A
k
are indep iff
P(A
1
A
k
) = P(A
1
) P(A
k
) and
all subsets of A
1
, . . . , A
k
are indep.
Independent Trials: Perform n trials of an experiment such that the
outcome of one trial is indep of the outcomes of the other trials.
Example: Flip 3 coins indeply.
(a) P(1st coin is H) = 1/2. Dont worry about the other two coins since
theyre indep of the 1st.
(b) P(1st coin H, 3rd T) = P(1st coin H)P(3rd T) = 1/4.
Remark: For indep trials, you just multiply the individual probs.
Goldsman 10/19/10 88 / 98
Conditional Probability and Independence
Example: Flip a coin innitely many times (each ip is indep of the
others).
p
n
P(1st H on nth trial)
= P(TT T
. .
n1
H)
= P(T)P(T) P(T)
. .
n1
P(H) = 1/2
n
.
P(H eventually) =

n=1
p
n
= 1.
Goldsman 10/19/10 89 / 98
Bayes Theorem
Outline
1
Intro / Examples
2
Set Theory
3
Experiments
4
Probability
5
Finite Sample Spaces
6
Counting Techniques
7
Counting Applications
8
Conditional Probability and Independence
9
Bayes Theorem
Goldsman 10/19/10 90 / 98
Bayes Theorem
Partition of a sample space split the sample space into disjoint,
yet all-encompassing subsets.
Denition: The events A
1
, A
2
, . . . , A
n
form a partition of the sample
space S if
(1) A
1
, A
2
, . . . , A
n
are disjoint.
(2)

n
i=1
A
i
= S.
(3) P(A
i
) > 0 for all i.
Goldsman 10/19/10 91 / 98
Bayes Theorem
Remark: When an experiment is performed, exactly one of the A
i
s
occurs.
Example: A and

A form a partition.
Example: vowels and consonants form a partition of the letters.
Example: Suppose A
1
, A
2
, . . . , A
n
form a partition of S, and B is some
arbitrary event. Then
B =
n
_
i=1
(A
i
B).
Goldsman 10/19/10 92 / 98
Bayes Theorem
So if A
1
, A
2
, . . . , A
n
is a partition,
P(B) = P
_
n
_
i=1
(A
i
B)
_
=
n

i=1
P(A
i
B) (A
1
, A
2
, . . . , A
n
are disjoint)
=
n

i=1
P(A
i
)P(B[A
i
) (by defn of condl prob).
This is the Law of Total Probability.
Example: P(B) = P(A)P(B[A) +P(

A)P(B[

A), which we saw in the

last module.
Goldsman 10/19/10 93 / 98
Bayes Theorem
Bayes Theorem: If A
1
, A
2
, . . . , A
n
form a partition of S and B is any
event, then
P(A
i
[B) =
P(A
i
B)
P(B)
=
P(A
i
)P(B[A
i
)

n
j=1
P(A
j
)P(B[A
j
)
.
The P(A
i
)s are prior probabilities (before B).
The P(A
i
[B)s are posterior probabilities (after B).
The P(A
i
[B) add up to 1. Thats why the funny-looking denominator.
Goldsman 10/19/10 94 / 98
Bayes Theorem
Example: In a certain city with good police,
P(Any defendent brought to trial is guilty) = 0.99.
In any trial,
P(Jury sets defendent free if he is innocent) = 0.95.
P(Jury convicts if defendent is guilty) = 0.95.
Find P(Defendent is innocent[Jury sets free).
Goldsman 10/19/10 95 / 98
Bayes Theorem
Events: I = innocent, G = guilty =

I, F = sets him free. Since
the partition is I, G, Bayes
P(I[F) =
P(I)P(F[I)
P(I)P(F[I) +P(G)P(F[G)
=
(0.01)(0.95)
(0.01)(0.95) + (0.99)(0.05)
= 0.161.
Notice how the posterior probs depend strongly on the prior probs.
Goldsman 10/19/10 96 / 98
Bayes Theorem
Example: A store gets 1/2 of its items from Factory 1, 1/4 from Factory
2, and 1/4 from Factory 3.
2% of Factory 1s items are bad.
2% of Factory 2s items are bad.
4% of Factory 3s items are bad.
An item from the store is found to be bad. Find the prob it comes from
Factory 1. [Answer should be < 1/2 since bad items favor Factory 3.]
Goldsman 10/19/10 97 / 98
Bayes Theorem
Events: F
i
= Factory i, B = bad item. Partition is F
1
, F
2
, F
3
.
P(F
1
[B) =
P(F
1
)P(B[F
1
)

3
j=1
P(F
j
)P(B[F
j
)
=
(0.5)(0.02)
(0.5)(0.02) + (0.25)(0.02) + (0.25)(0.04)
= 0.4.
It turns out that P(F
2
[B) = 0.2 and P(F
3
[B) = 0.4.
Goldsman 10/19/10 98 / 98