Lecture 5: Basic Probability Theory

Donglei Du
([email protected])

Faculty of Business Administration, University of New Brunswick, NB Canada Fredericton

E3B 9Y2

Donglei Du (UNB) ADM 2623: Business Statistics 1 / 55

Table of contents

1 Probability Theory
What is probability?
Random Experiment (RE)
Set/events operations
How to Interpret Probabilities?
Rules of Counting

Donglei Du (UNB) ADM 2623: Business Statistics 2 / 55

Layout

1 Probability Theory
What is probability?
Random Experiment (RE)
Set/events operations
How to Interpret Probabilities?
Rules of Counting

Donglei Du (UNB) ADM 2623: Business Statistics 3 / 55

What is probability?

Probability is a measure of the likelihood that an event in the future

will happen.
It can only assume a value between 0 and 1;
A value near zero means the event is not likely to happen;
A value near one means it is likely to happen.

Donglei Du (UNB) ADM 2623: Business Statistics 4 / 55

Random Experiment (RE)

RE is a process that satisfies the following propositions:

the process can be repeated as many trials as you want
the outcome of any trial is uncertain
well-defined set of possible outcomes
each outcome has a probability associated with it

Donglei Du (UNB) ADM 2623: Business Statistics 5 / 55

An example

Toss a coin once:

This process can be repeated as many times as you want (repetitive
nature)
Nobody knows whether Head or Tail will appear in any particular
tossing (uncertain outcome)
However, we know either Head or Tail must appear (Well-defined
sample space)
Now what is the chance you will get a Head if you toss a coin now?

Donglei Du (UNB) ADM 2623: Business Statistics 6 / 55

Some Terminologies

Outcome: is a particular result of a random experiment

Sample Space: is the collection or set of all the possible outcomes of
a random experiment
Event: is the collection of one or more outcomes of an experiment

Donglei Du (UNB) ADM 2623: Business Statistics 7 / 55

Example

Toss one coin in very trial

The sample space is S = {H, T}.
An event is the occurrence of head: E = {h}.
Flip two coins in very trial
The sample space has two outcomes
S = {(H,H), (H,T), (T,H), (T,T)}.
An event is the occurrence of a head in the first coin:
E = {(H, H), (H, T )}
Roll one die in very trial
The sample space contains 6 outcomes S = {1, . . . , 6}.
An event is the occurrence of an even number: E = {2, 4, 6}.
Roll two dice in very trial
The sample space contains the 36 outcomes S = {(1, 1), . . . , (6, 6)}.
An event is the occurrence of two dice with sum equal to 4:
E = {(1, 3), (2, 2), (3, 1)}

Donglei Du (UNB) ADM 2623: Business Statistics 8 / 55

Venn Diagram

A Venn diagram or set diagram is a diagram that shows all possible

logical relations between a finite collection of sets.
The Venn diagram for the above die example.

1 3 5

2 4 6

Donglei Du (UNB) ADM 2623: Business Statistics 9 / 55

An example
Toss two coins in very trial:
This process can be repeated as many times as you want (repetitive
nature)
Every trial contains the outcome of two coins

Origin First Second

Flip
p Flip
p

H HH
H
T HT
H TH

T T TT
Donglei Du (UNB) ADM 2623: Business Statistics 10 / 55
Set operations

Union
Intersection:
Complement:

Donglei Du (UNB) ADM 2623: Business Statistics 11 / 55

Union

Sample Space
A B

A B

Union: The union event of two events A and B is denoted as A ∪ B,

which consists of all outcomes in either A or B or in both. Namely
event A ∪ B occurs if either A or B occurs.
Example: Roll a die: E1 = {1, 3, 5} and E2 = {1, 2, 3}. Then
E1 ∪ E2 = {1, 2, 3, 5}
Donglei Du (UNB) ADM 2623: Business Statistics 12 / 55
Intersection

Sample Space

A A B B

Intersection: The intersection event of two events A and B is denoted

as A ∩ B, which consists of all outcomes in both A and B or in both.
Namely event A ∩ B occurs if both A and B occur.
Example: Roll a die: E1 = {1, 3, 5} and E2 = {1, 2, 3}. Then
E1 ∩ E2 = {1, 3}
Two events A and B are mutually exclusive if A ∩ B = ∅.
Donglei Du (UNB) ADM 2623: Business Statistics 13 / 55
Complement

Sample Space

A A

Complement: The complement event of an event A is denoted as Ā,

which consists of all outcomes that are not in A. Namely event Ā
occurs if A does not occur.
Example: Roll a die: E1 = {1, 3, 5}. Then Ē1 = {2, 4, 6}
Donglei Du (UNB) ADM 2623: Business Statistics 14 / 55
Probabilities defined on events

For any random experiment with sample space S, the probability of

any event is P (E) satisfying
(i) 0 ≤ P (E) ≤ 1.
(ii) P (S) = 1.
(iii) (special addition rule) For any pair-wise mutually exclusive events
E1 , E2 , . . .
∞ ∞
!
[ X
P En = P (En ).
n=1 i=1

Donglei Du (UNB) ADM 2623: Business Statistics 15 / 55

Examples
Toss one fair coin in very trial: If we assume a head is equally likely to
appear as a tail, then
1
P (H) = P (T ) =
2
Toss one biased coin in very trial: If we assume a head is twice as
likely to appear as a tail, then
2 1
P (H) = , P (T ) = .
3 3
Toss one fair die in very trial: If we assume all sides are equally likely
to appear, then
1
P (i) = , i = 1, . . . , 6.
6
From (iii) in the previous slide, we must have
1
P ({1, 3, 5}) = P (1) + P (2) + P (3) = .
2
Donglei Du (UNB) ADM 2623: Business Statistics 16 / 55
Probability Distributions defined on sample space

A (discrete) probability distribution on a given sample space is a table

of all disjoint outcomes and their associated probabilities.
Example: Toss two fair coins in very trial: The table below is the
probability distribution for the sample space
S = {HH, HT, T H, T T }:

Outcome Prob.
HH 0.25
HT 0.25
TH 0.25
TT 0.25

Donglei Du (UNB) ADM 2623: Business Statistics 17 / 55

Rule of Probability

Complement Rule
Addition Rule
Multiplication Rule

Donglei Du (UNB) ADM 2623: Business Statistics 18 / 55

Rule of Complement

For any event A:

P (Ā) = 1 − P (A)

Donglei Du (UNB) ADM 2623: Business Statistics 19 / 55

Rule of Addition

For any two events A and B:

P (A ∪ B) = P (A) + P (B) − P (A ∩ B).

If events A and B are mutually exclusive:

P (A ∪ B) = P (A) + P (B).

The above can be extended to any number of mutually exclusive

events A1 , . . . , An :

P (A1 ∪ . . . ∪ An ) = P (A1 ) + . . . + P (An ).

Donglei Du (UNB) ADM 2623: Business Statistics 20 / 55

Rule of Addition: Example

Toss two fair coins in very trial and assume that the four possible
outcomes S = {HH, HT, T H, T T } are equally likely to happen.
Let A = {HH, HT } be the event that the first coin falls head, and
B = {HH, T H} be the event that the second coin falls head.
Problem: Find the probability that either the first coin falls head or
the second coin falls head?
Solution:
1 1 1 3
P (A ∪ B) = P (A) + P (B) − P (A ∩ B) = + − = .
2 2 4 4

Donglei Du (UNB) ADM 2623: Business Statistics 21 / 55

Rule of Multiplication

Joint probability
Conditional probability

Donglei Du (UNB) ADM 2623: Business Statistics 22 / 55

Joint probability

Joint probability: For two events A and B defined on the same

sample space, the joint probability of events A and B is P (A ∩ B).
Example: Roll one die in very trial. Let A = {1, 2, 3} and
B = {1, 3, 5} be two events. Then the joint probability of A and B is
2 1
P (A ∩ B) = P ({1, 3}) = = .
6 3

Donglei Du (UNB) ADM 2623: Business Statistics 23 / 55

Represent Joint probability as contingency table

Joint probability distribution is given in a tabular form, called

contingency table. The probability distribution for each variable is
called the marginal distribution.
Example: A survey of undergraduate students in the Faculty of
Business Management at UNB revealed the following regarding the
gender and majors of the students:

Accoun. IB HR Marginal
Male 150/750 150/750 50/750 350/750
Female 175/750 160/750 65/750 400/750
Marginal 325/750 310/750 115/750 1

Example: What is the Probability of selecting a Female Accounting

student?
Solution:
P (F ∩ A) = 175/750 = 23.33%
Donglei Du (UNB) ADM 2623: Business Statistics 24 / 55
Conditional probability

Conditional probability: For any event A and B, the probability of

event A, given the occurrence of event B:

P (A ∩ B)
P (A|B) = .
P (B)

Example: What is the probability of selecting a Female, given that

the person selected is an International Business major?
Solution:
P (F |I) = 160/310 = 51.6%
Compare to the unconditional probability:

P (F ) = 400/750 = 50%

Donglei Du (UNB) ADM 2623: Business Statistics 25 / 55

Independent events: two events

Two events A and B are independent if and only if

P (A|B) = P (A)

or equivalently
P (A ∩ B) = P (A) × P (B).
Example: the events F = {(F, A), (F, IB), (F, HR)} and
IB = {(F, IB), (M, IB)} are not independent, since

P (F |IB) = 160/310 = 51.6% 6= 50% = 400/750 = P (F )

Donglei Du (UNB) ADM 2623: Business Statistics 26 / 55

Independent events: more than two events

n events A1 , . . . , An are independent if and only if for any r ≤ n:

P (A1 ∩ . . . ∩ Ar ) = P (A1 ) × . . . × P (Ar ).

Example: (Pairwise independent events may not be independent

overall): Roll a four-faced die once. Let A1 = {1, 2}, A2 = {1, 3} and
A3 = {1, 4} be three events. Then
1
P (A1 ∩ A2 ) = P (A1 ) × P (A2 ) =
4
1
P (A1 ∩ A3 ) = P (A1 ) × P (A3 ) =
4
1
P (A2 ∩ A3 ) = P (A2 ) × P (A3 ) =
4
1 1
P (A1 ∩ A2 ∩ A3 ) = 6= = P (A1 ) × P (A2 ) × P (A3 )
4 8

Donglei Du (UNB) ADM 2623: Business Statistics 27 / 55

Rule of Multiplication

For any two events A and B:

P (A ∩ B) = P (A|B)P (B).

The above can be extended to any number of events A1 , . . . , An :

P (A1 ∩ . . . ∩ An ) = P (A1 )P (A2 |A1 ) . . . P (An |A1 ∩ . . . ∩ An−1 ).

If events A and B are independent:

P (A ∩ B) = P (A)P (B).

The above can be extended to any number of independent events

A1 , . . . , A n :

P (A1 ∩ . . . ∩ An ) = P (A1 ) . . . P (An ).

Donglei Du (UNB) ADM 2623: Business Statistics 28 / 55

Rule of Multiplication: Example

Example: Draw three cards with replacement i.e., draw one card,
look at it, put it back, and repeat twice more.
Problem: Find the probability of drawing 3 Queens in a row:
Solution: Let Qi (i = 1, 2, 3) be the event that ith draw gives you a
Queen. Then the three events are independent of each other.
4 4 4
P (Q1 ∩ Q2 ∩ Q3 ) = P (Q1 )P (Q2 )P (Q3 ) = × × ≈ 0.00046.
52 52 52

Donglei Du (UNB) ADM 2623: Business Statistics 29 / 55

Rule of Multiplication: Example

Example: Draw three cards without replacement i.e., draw one card,
look at it, keep it, and repeat twice more.
Problem: Find the probability of drawing 3 Queens in a row:
Solution: Let Qi (i = 1, 2, 3) be the event that ith draw gives you a
Queen. Then the three events are dependent of each other.

P (Q1 ∩ Q2 ∩ Q3 ) = P (Q1 )P (Q2 |Q1 )P (Q3 |Q1 ∩ Q2 )

4 3 2
= × × ≈ 0.00018.
52 51 50

Donglei Du (UNB) ADM 2623: Business Statistics 30 / 55

More examples on rules of probability

Example: Draw one card from a deck of 52 cards

Problem: What is the probability of getting a red card or a Queen?
Solution:
26 + 4 − 2 7
P (R ∪ Q) = P (R) + P (Q) − P (R ∩ Q) = = ≈ 0.538
52 13

Donglei Du (UNB) ADM 2623: Business Statistics 31 / 55

More examples on rules of probability

Example: Two men throw their identical hats into the center of the
room at a party. Then the hats are mixed up and each man randomly
selects a hat.
Problem: What is the probability that none of them selects his own
hat.
Solution: Let Ai (i = 1, 2) be the event that the ith man selects his
own hat. Then the desired probability can be calculated via the
complement rule. The complement event is that at least one man
selects his own hat:

P (A1 ∪ A2 ) = P (A1 ) + P (A2 ) − P (A1 ∩ A2 )

= 0.5 + 0.5 − P (A1 )P (A2 |A1 ) = 1 − 0.5(1) = 0.5

Therefore none of them selects his own hat is

P A1 ∪ A2 = 1 − P (A1 ∪ A2 ) = 1 − 0.5 = 0.5

Donglei Du (UNB) ADM 2623: Business Statistics 32 / 55

Another method for solving the above problem

Solution: Let Ai (i = 1, 2) be the event that the ith man selects his
own hat. Therefore none of them selects his own hat is


P Ā1 ∩ Ā2 = P (Ā1 )P (Ā2 |Ā1 ) = 0.5(1) = 0.5.

Donglei Du (UNB) ADM 2623: Business Statistics 33 / 55

Bayes’ Theorem

Sample Space
Sample Space

A1  B A2  B An  B

A1 A2 …… An

Given an event B, and a set of mutually exclusive and exhaustive

events A1 , . . . , An , and we know the prior probabilities
P (A1 ), . . . , P (An ). Then the posterior is:

P (B|A1 )P (A1 )
P (A1 |B) =
P (B|A1 )P (A1 ) + . . . + P (B|An )P (An )

Donglei Du (UNB) ADM 2623: Business Statistics 34 / 55

Bayes’ Theorem: Example

Example: 25 percent of residents in an area leaves their garage doors

open when they left their home. The chances of being raided are 5
percent and 1 percent for those who leave their doors open, and who
do not leave their doors open, respectively.
Problem: Suppose one resident was robbed, what is the probability
that he or she originally left his or her door open?
Solution: Let B = robbed residents, A1 = open doors residents, A2
= close door residents. We know that P (A1 ) = 0.25, P (A2 ) = 0.75,
P (B|A1 ) = 0.05, P (B|A2 ) = 0.01.

Donglei Du (UNB) ADM 2623: Business Statistics 35 / 55

Bayes’ Theorem: Tree diagram
B  A1
0.05
B | A1

B | A1
0.95 B  A1
0.25 A
1

A2 0.01 B  A2
0 75
0.75
B | A2

B | A2
0.99
B  A2
Donglei Du (UNB) ADM 2623: Business Statistics 36 / 55
Bayes’ Theorem: Example

Therefore
P (B|A1 )P (A1 )
P (A1 |B) =
P (B|A1 )P (A1 ) + P (B|A2 )P (A2 )
0.05(0.25) 0.0125
= = = 0.625.
0.05(0.25) + 0.01(0.75) 0.02

Similarly we can calculate

0.01(0.75) 0.0075
P (A2 |B) = = = 0.375.
0.05(0.25) + 0.01(0.75) 0.02

Donglei Du (UNB) ADM 2623: Business Statistics 37 / 55

Bayes’ Theorem: Example

Example: An insurance company classifies drivers as good, medium,

or poor risks. Drivers who apply to for insurance fall into these three
groups in the proportions: 30%, 50%, and 20%, respectively. The
probabilities of a good-risk, medium-risk, and poor-risk drivers will
have an accident are 0.01, 0.03, and 0.10, respectively.
Problem: Suppose the company sells Mr. Brophy an insurance policy
and he has an accident. What is the probability that Mr. Brophy was
a good-risk driver?
Solution: Let B = accident drivers, A1 = good drivers, A2 =
medium drivers, and A3 = poor drivers. We know that P (A1 ) = 0.3,
P (A2 ) = 0.5, P (A3 ) = 0.2; P (B|A1) = 0.01, P (B|A2 ) = 0.03,
P (B|A3 ) = 0.10.

Donglei Du (UNB) ADM 2623: Business Statistics 38 / 55

Bayes’ Theorem: Tree diagram

B  A1
0.01
B | A1

B | A1
0.99 B  A1
0.3
A1
0 03
0.03 B  A2
A2 B | A2
0.5
B | A2
0.97
B  A2
02
0.2
A3 B  A3
0.10
B | A3

B | A3
0.90
B  A3

2013/9/17 Donglei Du: Lecture 1
Donglei Du (UNB) ADM 2623: Business Statistics 39 / 55
Bayes’ Theorem: Example

Therefore
P (B|A1 )P (A1 )
P (A1 |B) =
P (B|A1 )P (A1 ) + P (B|A2 )P (A2 ) + P (B|A3 )P (A3 )
0.01(0.3) 0.03
= = ≈ 0.079
0.01(0.3) + 0.03(0.5) + 01.0(0.2) 0.038

Similarly we can calculate

0.03(0.5) 0.015
P (A2 |B) = = ≈ 0.395
0.038 0.038
0.10(0.2) 0.02
P (A3 |B) = = ≈ 0.526
0.038 0.038
Now we know that Mr. Brophy is mostly likely a poor-risk driver.

Donglei Du (UNB) ADM 2623: Business Statistics 40 / 55

How to Interpret Probabilities

There are two ways to Interpret probability:

Objectivists
Subjectivists

Donglei Du (UNB) ADM 2623: Business Statistics 41 / 55

How to Interpret Probabilities: Objectivists

Objectivists: assign numbers to describe some objective or physical

state of affairs.
The most popular version of objective probability is frequentist
probability, which claims that the probability of a random event
denotes the relative frequency of occurrence of an experiment’s
outcome, when repeating the experiment. This interpretation
considers probability to be the relative frequency ”in the long run” of
outcomes.

Donglei Du (UNB) ADM 2623: Business Statistics 42 / 55

How to Interpret Probabilities: Subjectivists

Subjectivists: assign numbers per subjective probability, i.e., as a

degree of belief.
The most popular version of subjective probability is Bayesian
probability, which includes expert knowledge as well as experimental
data to produce probabilities. The expert knowledge is represented by
some (subjective) prior probability distribution. The data is
incorporated in a likelihood function. The product of the prior and
the likelihood, normalized, results in a posterior probability
distribution that incorporates all the information known to date.

Donglei Du (UNB) ADM 2623: Business Statistics 43 / 55

Rules of Counting

Multiplication Rule
Permutation Rule
Combination Rule

Donglei Du (UNB) ADM 2623: Business Statistics 44 / 55

Multiplication Rule

If one thing can be done in M ways, and if after this is done,

something else can be done in N ways, then both things can be done
in a total of M*N different ways in that stated order!

Donglei Du (UNB) ADM 2623: Business Statistics 45 / 55

Permutation Rule

A counting technique that is used when order is important

A permutation of r objects chosen from n objects is a group of any r
objects, when order is important
n!
Pnr =
(n − r)!

Donglei Du (UNB) ADM 2623: Business Statistics 46 / 55

Example

Example: You are assigned the task of choosing 2 of your 6

classmates to serve on a task force. One will act as the Chair of the
task force, and the other will be the Secretary.
Problem: In how many ways can you make this assignment?
Solution:
6!
P62 = = 30
(6 − 2)!

Donglei Du (UNB) ADM 2623: Business Statistics 47 / 55

Combination Rule

A counting technique that is used when order is NOT important

A combination of r objects chosen from n objects is a group of any r
objects, when order is not important
n!
Cnr =
r!(n − r)!

Donglei Du (UNB) ADM 2623: Business Statistics 48 / 55

Example

Example: You are assigned the task of choosing 2 of your 6

classmates to serve on a task force. Responsibilities are evenly shared.
Problem: In how many ways can you make this assignment?
Solution:
6!
C62 = = 15
2!(6 − 2)!

Donglei Du (UNB) ADM 2623: Business Statistics 49 / 55

The Twin Paradox (Probabilistic Pigeonhole Principle)

Example: The Statistics Professor wants to play a little game with

his students. “I bet that there are two of you who have the same
birthday! What do you think”.
Several students reply immediately: “There are 366 possible
birthdays, so you could only conclude this if there were at least 367 of
us in the class! But there are only 50 of us, and so you would lose the
bet for sure!”
Problem: Let us calculate the probability of winning by the professor.

Donglei Du (UNB) ADM 2623: Business Statistics 50 / 55

The Twin Paradox (Probabilistic Pigeonhole Principle)

Solution: Let A be the event of winning by the professor. Then its

complement is the event of wining by students, which is equivalent to
the event that all n = 50 students having different birthdays.
number of different birthdays for 50 students
P (Ā) =
number of birthdays for 50 students
50
C366 366! 366 × . . . × 316
= 50
= 50
= ≈ 0.03.
366 /50! (366 − 50)!366 36650

Therefore the teacher has a much higher chance of wining:

P (A) = 1 − 0.03 = 97%.!

Donglei Du (UNB) ADM 2623: Business Statistics 51 / 55

The Twin Paradox (Probabilistic Pigeonhole Principle)

Here are the probability of wining by the professor for different n, the
number of students.
n Prob.
10 12
20 41
30 71
40 81
50 97
60 99

Donglei Du (UNB) ADM 2623: Business Statistics 52 / 55

Case: conditional probability and market direction
prediction

Probability of consecutive up or down price moves

G*#2)354"?+67"@#/3)"J;3*I6"

Data: S&P 500 from

Jan 1950 to May 2015

K"#
K"#$"%"
#$"!
!B"
K"#$"!B"
@1A"BCD"BE!F"

Figure: Probability of consecutive up and down price movements

Donglei Du (UNB) ADM 2623: Business Statistics 53 / 55

Case: conditional probability and market direction
prediction

Probability of 4th up day (A), given 3 consecutive up days (B) is

P (A ∩ B) P (A) 0.086
P (A|B) = = = = 53.1%
P (B) P (B) 0.162

Donglei Du (UNB) ADM 2623: Business Statistics 54 / 55

Case: conditional probability and market direction
prediction

G*#212+)+034"#$"@1*:36"@#;34"
G*#2)354"?+67"@#/3)"J;3*I6"
The probabilities
are all roughly
53% – which is
the same as the
probability of
having a
‘normal’ up day.

Q"#
Q"#$"%"
#$"!
!B"
Q"#$"!B"
@1A"BCD"BE!F"

Figure: Probability of consecutive up and down price movements

Donglei Du (UNB) ADM 2623: Business Statistics 55 / 55