DATA1001 Introduction to Data Science and Decisions

Lecture 1: Probability and Randomness

Jeffrey Kwan
Term 2, 2023

1/40
Introduction to Probability

2/40
Experiment, Sample Space, Event

An experiment is any process leading to recorded observations.

Example
A non-exhaustive list of experiments include: coin tossing, birth month, blood pressure
measurements, progress of an economic indicator, etc.

Probability theory is interested in the outcomes of random experiments.

3/40
Experiment, Sample Space, Event

Together all the possible outcomes form the sample space Ω.

Example
When tossing a coin there are two possible outcomes: Head or Tail. The sample space
is then Ω = {H, T }.
When tossing a coin twice there are four possible outcomes. The sample space is then
Ω = {HH, HT , TH, TT }.

4/40
Experiment, Sample Space, Event

An event A is a set of outcomes, a subset of Ω (denoted A ⊂ Ω).

An event occurs if the result of the experiment is one of the outcomes in that event.
Example
When tossing a coin twice, the event A:"At least one tail" corresponds to the set
{HT , TH, TT }.

The events are assigned a probability: a number between 0 and 1.

5/40
Event relations

Consider an event A (left).

The event Ā (right) defines the complement of A: the elements in Ω that are not in A.

A A

6/40
Event relations

Consider two events A and B.

The set A ∩ B (left) is the intersection of A and B: it occcurs if both A and B occur.

The set A ∪ B (right) is the union of A and B: it occcurs if at least A or B occur.

A B A B

7/40
Event relations

Example
We roll a dice once.
Define the events A = {1, 3} and B = {2, 3, 5}. Then
• A ∩ B = {3}
• A ∪ B = {1, 2, 3, 5}

8/40
Mutual exclusivity

Events are mutually exclusive (disjoint) if they have no outcomes in common; that is, if
they cannot both occur.

If A and B are mutually exclusive, we can say that A ∩ B = ∅, where ∅ denotes the
empty set.
Example
We toss a coin 3 times and let H denote "head", T denote "tail".
The sample space is S = {HHH, HHT , HTH, THH, TTH, THT , HTT , TTT }.
Let A be the event ‘at least one head’:
A = {HHH, HHT , HTH, THH, TTH, THT , HTT }.
Let B be the event ‘exactly two heads’: B = {HHT , HTH, THH}.
Let C be the event ‘at least two tails’: C = {TTH, THT , HTT , TTT }.
9/40
Mutual exclusivity
Example
Which of A, B, and C are mutually exclusive?
• We have that B ⊂ A thus A ∩ B = B. So if B occurs then A must occur.
• Next we have A ∩ C = {TTH, THT , HTT } =
6 ∅.
• Last B and C cannot both occur, so they are mutually exclusive.

(Video explanation)
Example
Let’s say {HTT } is the result of the experiment. Which of A, B and C have occurred?
A and C have occurred, but not B.

(Video explanation)
10/40
Axioms of Probability

To each event A ⊂ Ω, we assign a number P(A), the probability that A occurs, defined
by
#(basic outcomes in A)
P(A) =
#(basic outcomes in Ω)

The function P must satisfy the axioms:

1. P(A) ≥ 0,

2. P(Ω) = 1,

3. If A1 , A2 , . . . are mutually exclusive

(Ai ∩ Aj = ∅ for all i, j with i 6= j),

S∞ P∞
then P ( i=1 Ai ) = i=1 P(Ai ).
11/40
Axioms of Probability

It follows that

1. If A1 , A2 , . . . , Ak are mutually exclusive,

k k
!
= P(Ai ),
[ X
P Ai
i=1 i=1

2. P(∅) = 0,

3. For any A ⊂ Ω, 0 ≤ P(A) ≤ 1 and P(A) = 1 − P(A).

4. If B ⊂ A, then P(B) ≤ P(A). Thus if B occurs =⇒ A occurs then P(B) ≤ P(A).

12/40
Axioms of Probability

Rule for calculating probabilities: if Ω = {ω1 , ω2 , . . .} is discrete, P(A) = sum of

probabilities for outcomes in A.
Example
We toss a coin 3 times and let H denote "head", T denote "tail".
The sample space is S = {HHH, HHT , HTH, THH, TTH, THT , HTT , TTT }.
Let B be the event ‘exactly two heads’: B = {HHT , HTH, THH}.
Since B = {HHT } ∪ {HTH} ∪ {THH} and {HHT }, {HTH}, {THH} are mutually
exclusive, the probability of the event B occurring is
1 1 1 3
P(B) = P({HHT }) + P({HTH}) + P({THH}) = + + = .
8 8 8 8

13/40
Counting Rules

Counting rules are necessary to evaluate the number of elements in Ω.

VARIATIONS

• Every group of i elements chosen from a set of n elements in which the ordering of
elements matters is called a variation of the i-th order of n elements.

• A variation with repetition is a variation in which every element can be present

more than once, in that case the possible number of variations is ni .

• If there are k experiments with ni possible outcomes in the i-th (i = 1, . . . , k), then
the total number of possible outcomes for the k experiments is
i=1 ni = n1 × · · · × nk .
Qk

14/40
Counting Rules
Example
We toss a 6-sided die three times. What is the number of outcomes in the sample
space (Find k and nj )?
Here there are k = 3 experiments each having six possible outcomes, hence
n1 = n2 = n3 = 6. The sample space contains 216 possible outcomes.
(Video explanation)
Example
An urn contains a black (B), a red (R) and a green (G) ball. We draw two balls with
repetition. What is the number of outcomes in the sample space?
This is a variation of the 2nd order of 3 elements with repetition. There are 32 = 9
possible outcomes: {B, B}, {B, R}, {B, G}, {R, B}, {R, R, {R, G}, {G, B}, {G, R}
and {G, G}.
15/40
Counting Rules

PERMUTATIONS

• An ordered arrangement of a set of distinct objects (no repetition) is a permutation.

• The number of possible permutations of r objects selected from n distinct objects is

nP =
(n−r )! , where n! = n × (n − 1) × . . . 2 × 1 for integers n ≥ 1 and 0! ≡ 1.
n!
r

In R we could use the following code to calculate permutations

permutations <- function(n, r) {
factorial(n) / factorial(n-r)
}
permutations(n=4, r=2)

## [1] 12
16/40
Counting Rules
Example
A particular committee has four members labeled as a, b, c and d. One member must
chair the committee, and a different committee member must take minutes from
meetings.
How many different ways are there of choosing a Chair and a Minute-taker for this
committee?
There are n = 4 members and we need to choose r = 2 of them. Here the order
matters. Hence the number of possible ways to select a Chair and a Minute-taker is
4 P = 12. These are {a, b}, {a, c}, {a, d}, {b, a}, {b, c}, {b, d}, {c, a}, {c, b}, {c, d},
2
{d, a}, {d, b} and {d, c}
(Video explanation)

The number of possible permutations of r objects (n = r ) is r Pr = r !.

17/40
Counting Rules

COMBINATIONS

• A combination is a permutation where the order does not matter.

• The number of ways of choosing r objects from n distinct objects is

r !(n−r )! ≡ C (n, r )
n!

This number is computed in R by the choose function:

choose(4,0)
## [1] 1
choose(4,1)
## [1] 4
choose(4,2)
## [1] 6
18/40
Counting Rules

NOTE: The number of permutations (ordered) is n Pr . When ignoring order, a particular

set of r objects, which contributes r ! permutations to the total, now counts as one
n
selection and so the number of ways of choosing r from n is rP!r = r !(n−r
n!
)! .

Example
A particular committee has four members labeled as a, b, c and d. Two committee
members will need to present the committee’s recommendations to the board of
directors.
How many ways are there of choosing two committee members to report to the board
of directors?
We choose r = 2 members and ignore order. The number of possibilities is C (4, 2) = 6.
Indeed the possibilities are: {ab}, {ac}, {ad}, {bc}, {bd} and {cd}.
(Video explanation)
19/40
Conditional Probability

The conditional probability that an event A occurs, given that an event B has occurred is

P(A ∩ B)
P(A|B) = for P(B) > 0.
P(B)

Given that B has occurred, the total probability for possible results of the experiment
equals P(B), so that the probability that A occurs equals the total probability for
outcomes in A (only those in A ∩ B) divided by the total probability, P(B).

In other words, by assuming that B has occurred, we have defined a new sample space
Ω = B and a new probability P(A|B).

20/40
Conditional Probability

Lemma

P(A|B) = P(A) ⇐⇒ P(B|A) = P(B).

Proof:

“=⇒”: if P(A|B) = P(A) then

P(B ∩ A) P(A ∩ B) P(A|B)P(B)

P(B|A) = = = = P(B).
P(A) P(A) P(A)

“⇐=”: interchange A and B in the above.

If the probability of A remains the same, whether or not B has occurred, we say that the
two events are statistically independent.
21/40
Independent Events

Events A and B are independent if P(A ∩ B) = P(A)P(B).

For any two events A and B, P(A ∩ B) = P(A|B)P(B), so A and B are independent
implies that P(A|B) = P(A). By the previous Lemma it also implies P(B|A) = P(B).
Example
We toss a two fair dice and let:
• A be the event "4 on first die"
• B be the event "sum of numbers is 7"
• C be the event "sum of numbers is 8".
Show that A and B are independent.
Show that A and C are dependent.
22/40
1.5. INDEPENDENT
Independent Events EVENTS 7

Example
We can write S as
We can write the sample space as
S = {(1, 1), (1, 2), . . . , (1, 6), (2, 1), . . . , (2, 6), . . . , (6, 6)}
Ω = {(1, 1), (1, 2), . . . , (1, 6), (2, 1), . . . , (2, 6), . . . , (6, 6)}.
or display it as follows;
It contains 36 possible outcomes, each with probability 1/36.
2nd die
1 2 3 4 5 6
1 · · · · ·
2 · · · · ·
1st die 3 · · · · · A ⇥
4 ⇥ ⇥ ⌦ ⇥ ⇥ ⇥ B
5 · · · · · C ⇤
6 · · · · ·
23/40
Independent Events

Example
• P(A) = 36
6
= 16 , P(B) = 36
6
= 16 , P(A ∩ B) = 1
36 and 1
36 = P(A ∩ B) = P(A)P(B).
Thus A and B are independent.
• P(C ) = 36
5
,P(A ∩ C ) = 36 ,
1
so 1
36 = P(A ∩ C ) 6= P(A)P(C ) = 1
6 × 36 .
5
Thus A and
C are not independent.
P(A ∩ C ) 1/36 1
• Also, P(A|C ) = = = 6= P(A), again confirming A and C are
P(C ) 5/36 5
not independent.

(Video explanation)

24/40
Independent Events

For a countable sequence of events {Ai }, the events are pairwise independent if

P(Ai ∩ Aj ) = P(Ai )P(Aj ) for all i 6= j

and the events are (mutually) independent if for any collection Ai1 , Ai2 , . . . , Ain ,

P(Ai1 ∩ · · · ∩ Ain ) = P(Ai1 ) . . . P(Ain ).

Clearly, independence implies pairwise independence, but not vice versa, as in the
following example.

25/40
Independent Events

Example
A coin is tossed twice and let
• A be the event "head on the first toss"
• B the event "head on the second toss"
• C the event "exactly one head turned up".
The events A, B and C are pairwise independent (can you show this?), but

1
 3
P(A ∩ B ∩ C ) = 0 6= P(A)P(B)P(C ) = ,
2
so A, B and C are not independent.
(Video explanation)
26/40
Independent Events
Example
A ball is drawn at random from an urn containing 4 balls numbered 1,2,3 and 4. Let
• A be the event "ball 1 or ball 2 is drawn" =⇒ A = {1, 2}
• B the event "ball 1 or ball 3 is drawn" =⇒ B = {1, 3}
• C the event "ball 1 or ball 4 is drawn" =⇒ C = {1, 4}.
Show that A, B, and C are pairwise independent but not independent.
We have P(A ∩ B) = P({1}) = 14 = P(A)P(B) so A and B are (pairwise) independent.
Similarly, A and C , and B and C are independent.
On the other hand, P(A ∩ B ∩ C ) = P({1}) = 1
4 6= P(A)P(B)P(C ) = 18 , so A, B and
C are not independent.
(Video explanation)
27/40
Some Probability Laws

MULTIPLICATIVE LAW

For events A1 , A2

P(A1 ∩ A2 ) = P(A2 ∩ A1 )
= P(A2 |A1 )P(A1 ).
For events A1 , A2 , A3

P(A1 ∩ A2 ∩ A3 ) = P(A3 ∩ A2 ∩ A1 )
= P(A3 |A2 ∩ A1 )P(A2 ∩ A1 )
= P(A3 |A1 ∩ A2 )P(A2 |A1 )P(A1 ).

The same pattern applies to higher numbers of events.

28/40
Some Probability Laws

Example
To gain entry to a selective high school students must pass 3 tests.
• 20% fail the first test and are excluded.
• Of the 80% who pass the first, 30% fail the second and are excluded.
• Of those who pass the second, 60% pass the third.
Let Ai be the event "pass i-th test", i = 1, 2, 3.
Question 1: What proportion of students pass the first two tests?

P(A1 ∩ A2 ) = P(A2 |A1 )P(A1 ) = 0.7 × 0.8 = 0.56

29/40
Some Probability Laws
Example
Question 2: What proportion of students gain entry to the selective high school?

P(Student gains entry) = P(A1 ∩ A2 ∩ A3 )

= P(A3 |A1 ∩ A2 )P(A2 |A1 )P(A1 )
= 0.6 × 0.7 × 0.8
= 0.336.

Question 3: What proportion pass the first two tests, but fail the third?

P(A1 ∩ A2 ∩ A3 ) = P(A3 |A1 ∩ A2 )P(A2 |A1 )P(A1 )

= 0.4 × 0.7 × 0.8
= 0.224.

(Video explanation) 30/40

Some Probability Laws

ADDITIVE LAW

For events A and B, P(A ∪ B) = P(A) + P(B) − P(A ∩ B).

A B A B A B A B

Proof: From the Venn diagrams (next slide) A ∪ B = A ∪ (Ā ∩ B) and

B = (A ∩ B) ∪ (Ā ∩ B). Note that A and Ā ∩ B are mutually exclusive and so are A ∩ B
and Ā ∩ B. So from the axioms
P(A ∪ B) = P(A) + P(Ā ∩ B), P(B) = P(A ∩ B) + P(Ā ∩ B).

Substituting P(Ā ∩ B) in the left equation by P(Ā ∩ B) = P(B) − P(A ∩ B)

31/40
Some Probability Laws

A B A B A B

B A B A B

32/40
Some Probability Laws

Example
Three letters are placed at random into three addressed envelopes.
What is the probability that none is in the correct envelope?
Let A, B, C be the events that envelopes 1, 2 and 3 contain the correct letters.
Then P(A) = P(B) = P(C ) = 31 and P(A ∩ B) = P(A ∩ C ) = P(B ∩ C ) = 1
6 since,
for example, P(A ∩ B) = P(B|A)P(A) = 12 × 13 .
Also, P(A ∩ B ∩ C ) = 16 since all 3 envelopes must contain the correct letters if any 2
envelopes contain the correct letters.
This means that P(A ∩ B ∩ C ) = P(A ∩ B) = P(A ∩ C ) = P(B ∩ C ).

33/40
Some Probability Laws

Example
Thus,

P(none is in the correct envelope)

= 1 − P(at least one is in the correct envelope)
= 1 − P(A ∪ B ∪ C )
= 1 − {P(A) + P(B) + P(C ) − P(A ∩ B) − P(A ∩ C ) − P(B ∩ C ) + P(A ∩ B ∩ C )}
1 1 1 1 1 1 1
=1−{ + + − − − + }
3 3 3 6 6 6 6
1
= .
3

34/40
Some Probability Laws

LAW OF TOTAL PROBABILITY

Suppose A1 , A2 , . . . , Ak are mutually exclusive (Ai ∩ Aj = ∅ for all i 6= j) and exhaustive

( ki=1 Ai = Ω) events; that is, A1 , . . . , Ak form a partition of Ω. Then, for any event B,
S

k
P(B) = P(B|Ai )P(Ai ).
X

i=1

Proof: B = ki=1 (B ∩ Ai ) (disjoint union since the Ai ’s are disjoint). By axiom 3 (in
S

the finite case):

k k k
!
P(B) = P (B ∩ Ai ) = P(B ∩ Ai ) = P(B|Ai )P(Ai ).
[ X X

i=1 i=1 i=1

35/40
Some Probability Laws
Example
Urn I contains 3 red and 4 white balls. Urn II contains 2 red balls and 4 white. A ball
is drawn from Urn I and placed unseen into Urn II. A ball is now drawn at random
from Urn II.
What is the probability that this second ball is red?
Let A1 be the event "1st ball drawn red", A2 be the event "1st ball drawn white" and
B be the event "2nd ball drawn red". A1 and A2 are mutually exclusive (they cannot
both occur) and exhaustive (one of them must occur) and so

P(B) = P(B|A1 )P(A1 ) + P(B|A2 )P(A2 )

3 3 2 4 17
= × + × = .
7 7 7 7 49

36/40
Bayes’ formula

Bayes’ formula calculates conditional probabilities when the ordering of conditioning is

reversed. In the simple two event situation it corresponds to:

P(A ∩ B) P(B|A)P(A)
P(A|B) = = .
P(B) P(B)

The most general version is:

For a partition A1 , A2 , . . . , Ak and an event B,

P(B|Aj )P(Aj ) P(B|Aj )P(Aj )

P(Aj |B) = Pk = .
i=1 P(B|Ai )P(Ai )
P(B)

37/40
Bayes’ formula

Example
Recall the previous example.
Given that the second ball is red, what is the probability that the first ball was white?

P(B|A2 )P(A2 ) 2
× 4
8
P(A2 |B) = = 7 7
= .
P(B) 17
49
17

(Video explanation)

38/40
Bayes’ formula

Example
A diagnostic test for a certain disease is claimed to be 90% accurate because, if a
person has the disease, the test will show a positive result with probability 0.9 while if
a person does not have the disease the test will show a negative result with probability
0.9. Only 1% of the population has the disease.
If a person is chosen at random from the population and tests positive for the disease,
what is the probability that the person does in fact have the disease?

39/40
Bayes’ formula
Example
Let A be the event "person has disease" and B be the event "person tests positive".

P(A ∩ B) P(B|A)P(A)
P(A|B) = =
P(B) P(B|A)P(A) + P(B|Ā)P(Ā)

since A and Ā form a partition (they are mutually exclusive and exhaustive).
Now P(B|A) = 0.9, P(A) = 0.01, P(B|Ā) = 0.1 andP(Ā) = 0.99 therefore

0.9 × 0.01 1
P(A|B) = = .
0.9 × 0.01 + 0.1 × 0.99 12
This means that, given that the person’s test result is positive the probability that a
person has the disease is 12
1
.

(Video explanation)
40/40