28/01/2024

Probability theory

Statistics (ECB1STAT)
Utrecht University School of Economics

Topics covered

 Random variables;

 Properties of probabilities;

 Sets and Events, including, e,g., the intersection and union of events;

 Bivariate and conditional probabilities;

 Statistical independence;

 Bayes’ Theorem.

1
 28/01/2024

Random variables and outcomes

 A variable is a characteristic of a subject (e.g., an individual’s level of education or a country’s GDP).

 The outcome of a variable is the realization of a variable (an observation).

 Before the outcome is known, there is often uncertainty about what the outcome will be.

 A random variable: a variable which outcome is a priori unknown.

 The outcome is determined by a random draw from the distribution

of possible outcomes in the population.

 A random variable has more than one possible outcome.

 No two outcomes of the random variable can occur simultaneously.

 A random variable is often denoted by X, Y or Z, to keep it general.

Random variables and outcomes

Example: A dice is rolled once.

 A single random draw.

 The distribution of possible outcomes: 1, 2, 3, 4, 5 or 6, each with a probability of .

 Before rolling the dice, it is unknown what the outcome will be.
 No two outcomes can occur simultaneously. E.g., if the outcome is 2, it cannot also be any
other value.
 Define the random variable X as the outcome of rolling a dice.

In short: X = the outcome of rolling a dice.

2
 28/01/2024

Properties of probabilities

 The probability of a single outcome 𝑂 is greater than or equal to 0 and less than or equal to 1:

0≤𝑃 𝑂 ≤1

 The set of all possible outcomes of a random variable is the sample space 𝑆, hence 𝑂 ∈ 𝑆 (∈:‘element of’)

 The sum of the probabilities of each of the possible outcomes equals 1:

𝑃 𝑆 =∑ ∈ 𝑃(𝑂) = 1

Properties of probabilities

Example continued: A dice is rolled once.

Previously: X = the outcome of rolling a dice.
 The outcome is either 1, 2, 3, 4, 5 or 6: the sample space S = {1, 2, 3, 4, 5, 6}.

 The probabilities are, for this example, the same for each outcome: 𝑃 𝑂 = for 𝑂 ∈ 𝑆
 0 ≤ 𝑃 𝑂 ≤ 1 for 𝑂 ∈ 𝑆 (check!)
 The sum of the probabilities of each of the possible outcomes equals 1:
𝑃 𝑆 =∑ ∈ 𝑃(𝑂)
=𝑃 1 +𝑃 2 +𝑃 3 +𝑃 4 +𝑃 5 +𝑃 6

= + + + + + =1 (check!)

3
 28/01/2024

Events

 An event is a collection of possible outcomes of a random variable.

 More formally: An event is a subset of the sample space S: 𝐴 ∈ 𝑆
 The probability of event 𝐴 is denoted by 𝑃 𝐴 .
 The probability of event 𝐴 is greater than or equal to 0 and less than or equal to 1:
0 ≤ 𝑃 𝐴 ≤ 1.
 The probability of event 𝐴 is the sum of the probabilities of each of the outcomes of event 𝐴:
𝑃 𝐴 =∑ ∈ 𝑃(𝑂)
 The set of outcomes belonging to 𝑆 but not to event 𝐴
is called the complement of 𝐴, denoted by 𝐴̅.
 In terms of probabilities: 𝑃 𝐴̅ = 𝑃 𝑆 − 𝑃 𝐴 = 1 − 𝑃 𝐴 .

Events

Example continued: A dice is rolled once.

Previously: X = the outcome of rolling a dice, S = {1, 2, 3, 4, 5, 6} and 𝑃 𝑂 = for 𝑂 ∈ 𝑆.

Define event A: an even outcome: A = {2, 4, 6}.
A is a subset of the sample space S : 𝐴 ∈ 𝑆 (check!).

𝑃 𝐴 =∑ ∈ 𝑃(𝑂) = 𝑃 2 + 𝑃 4 + 𝑃 6 = + + = => 0 ≤ 𝑃 𝐴 ≤ 1 (check!).

The complement of event A is event 𝐴̅ = {1, 3, 5}

with 𝑃 𝐴̅ = ∑ ∈ ̅ 𝑃(𝑂 ) =𝑃 1 +𝑃 3 +𝑃 5 =

𝑃 𝐴̅ = 1 − 𝑃 𝐴 = 1 − = (check!)

To practice: Exercise 3.17 a-c

4
 28/01/2024

Events

 Two events A and B: 𝐴 ∈ 𝑆 and 𝐵 ∈ 𝑆.

 The intersection of A and B: 𝐴 ∩ 𝐵 (common outcomes).
 No common outcomes: 𝐴 ∩ 𝐵 = ∅ (empty set).
 If 𝐴 ∩ 𝐵 = ∅, A and B are mutually exclusive events.
 The union of A and B: 𝐴 ∪ 𝐵 (all outcomes of A and B).
 If 𝐴 ∪ 𝐵 = 𝑆, A and B are collectively exhaustive events (they
cover the entire sample space 𝑆).
 Corresponding probabilities:
• 𝑃 𝐴∪𝐵 =∑ ∈ ∪ 𝑃(𝑂 );

• 𝑃 𝐴∩𝐵 =∑ ∈ ∩ 𝑃(𝑂 );
• General rule: 𝑃 𝐴 ∪ 𝐵 = 𝑃 𝐴 + 𝑃 𝐵 − 𝑃 𝐴 ∩ 𝐵 .

Events

Example continued I: A dice is rolled once.

Previously: X = the outcome of rolling a dice, S = {1, 2, 3, 4, 5, 6} and 𝑃 𝑂 = for 𝑂 ∈ 𝑆.

Define: Event A = {2, 4, 6} (even outcomes).
Define: Event B = {1, 3, 5} (uneven outcomes).
There are no common outcomes, i.e., the intersection of A and B contains no outcomes:
𝐴 ∩ 𝐵 = ∅ => A and B are mutually exclusive events.
The outcomes of A and B cover the entire sample space 𝑆.
That is, the union of A and B contains all possible outcomes:
𝐴 ∪ 𝐵 = 𝑆 => A and B are collectively exhaustive events.

10

5
 28/01/2024

Sets and events

Example continued II: A dice is rolled once.
Previously: X = the outcome of rolling a dice, S = {1, 2, 3, 4, 5, 6} and 𝑃 𝑂 = for 𝑂 ∈ 𝑆
Define: Event C = {1, 3} and Event D = {2, 3}
• Probabilities: 𝑃 𝐶 = ∑ ∈ 𝑃(𝑂) = 𝑃 1 + 𝑃 3 = + = and 𝑃 𝐷 = ∑ ∈ 𝑃(𝑂 ) =. . =
• Intersection 𝐶 ∩ 𝐷 = {3} =>𝑃 𝐶 ∩ 𝐷 = ∑ ∈ ∩ 𝑃(𝑂 ) = 𝑃 3 =
• Union 𝐶 ∪ 𝐷 = {1, 2, 3} => 𝑃 𝐶 ∪ 𝐷 = ∑ ∈ ∪ 𝑃(𝑂) = 𝑃 1 + 𝑃 2 + 𝑃 3 = + + =
Also:
• 𝐶 and 𝐷 are not mutually exclusive events: 𝐶 ∩ 𝐷 ≠ ∅
or 𝑃 𝐶∩𝐷 ≠ 0
• 𝐶 and 𝐷 are not collectively exhaustive events: 𝐶 ∪ 𝐷 ≠ 𝑆
or 𝑃 𝐶∪𝐷 ≠1
Check general rule: 𝑃 𝐶 ∪ 𝐷 = 𝑃 𝐶 + 𝑃 𝐷 − 𝑃 𝐶 ∩ 𝐷
Hence, 𝑃 𝐶 ∪ 𝐷 = + − = (same as above => check!)

To practice: Exercises 3.3 & 3.17

11

Bivariate probabilities

 Two random variables X and Y with sample spaces SX and SY, respectively.

 Event A is a subset of SX and event B is a subset of SY.

 The bivariate probability that events A and B occur is denoted by:

𝑃 𝐴 ∩ 𝐵 , where 𝐴 ∩ 𝐵 is the intersection of A and B.

 The bivariate probability that events A or B occur is denoted by:

𝑃 𝐴 ∪ 𝐵 , where 𝐴 ∪ 𝐵 is the union of A and B.

 General rule: 𝑃 𝐴∪𝐵 = 𝑃 𝐴 +𝑃 𝐵 −𝑃 𝐴∩𝐵

Also:

 𝑃 𝐴 and 𝑃 𝐵 are referred to as marginal probabilities.

 Special case X =Y and SX =SY: e.g., example of rolling a dice once.

12

6
 28/01/2024

Bivariate probabilities: Example

Of all cars in the population, the table to the right provides CD No CD Total
information how many have an airco (AC), a compact disk AC 20% 50% 70%
player (CD), or both. The table provides percentages. E.g., 70%
of the cars have an AC. No AC 20% 10% 30%

What is the probability that a car has a CD and has an AC? Total 40% 60% 100%

The answer can be seen straightaway from the table, but how
do you write down your answer?

Steps:
1. X = whether a car has a CD, Y = whether a car has an AC
2. Outcome spaces: SX = {CD, No CD} , SY = {AC, No AC}.
3. Event A= {CD} , Event B = {AC}
4. Asked for: 𝑃 𝐴 ∩ 𝐵 ?
5. Answer: 𝑃 𝐴∩𝐵 =𝑃 𝐶𝐷, 𝐴𝐶 = = 0.2

13

Conditional probability

 Two random variables X and Y with sample spaces SX and SY, respectively.
 Event A is a subset of SX and event B is a subset of SY.
 The conditional probability 𝑃 | 𝐴|𝐵 is the probability that event 𝐴 occurs given (or
conditional on) that event 𝐵 occurs.
 In short: 𝑃 | 𝐴|𝐵 is the probability of A given B.
 The conditional probability:
∩
(General formula) 𝑃 | 𝐴|𝐵 = ,
( )

with bivariate probability 𝑃 𝐴 ∩ 𝐵 and

marginal probability 𝑃 (𝐵).

14

7
 28/01/2024

Conditional probability: Example

Of all cars in the population, the table to the right provides
CD No CD Total
information how many have an airco (AC), a compact disk player
(CD), or both. The table provides percentages. E.g., 70% of the AC 20% 50% 70%
cars have an AC. No AC 20% 10% 30%
What is the probability that a car has a CD player conditional Total 40% 60% 100%
on that it has AC?

Random variables: X = whether a car has a CD, Y = whether a car has an AC

Using a more concise notation than previous example:
Asked for is 𝑃 | 𝐶𝐷 𝐴𝐶 :
∩ /
𝑃 | 𝐶𝐷 𝐴𝐶 = = = 0.2857
( ) /

Interpretation:
the probability that a car with an airco has a CD player is 28.6%.

To practice: Exercises 3.23 and 3.77

15

Statistical independence

 Two random variables X and Y, sample spaces 𝑆 and 𝑆 , event 𝐴 ∈ 𝑆 and event 𝐵 ∈ 𝑆 .
 The two events A and B are statistically independent if the conditional probability of A given B
equals the probability of A:
𝑃 | 𝐴|𝐵 = 𝑃 𝐴
Intuition: the outcome of Y provides no information about the probability of the outcome of X.
 Also, under the independence assumption:
– 𝑃 𝐴 ∩ 𝐵 =𝑃 𝐴 ⋅ 𝑃 𝐵 (intersection)
– 𝑃 𝐴∪𝐵 =𝑃 𝐴 +𝑃 𝐵 −𝑃 𝐴 ⋅𝑃 𝐵 (union)
 A general rule: X and Y are independent if:
𝑃 | 𝑋|𝑌 = 𝑃 𝑋
Or, equivalently, 𝑃 𝑋, 𝑌 = 𝑃 𝑋 ⋅ 𝑃 𝑌

16

8
 28/01/2024

Statistical independence: Example

Of all cars in the population, the table to the right provides CD No CD Total
information how many have an airco (AC), a compact disk AC 20% 50% 70%
player (CD), or both. The table provides percentages. E.g.,
70% of the cars have an AC. No AC 20% 10% 30%

Are AC and CD independent events? Total 40% 60% 100%

Random variables: X = whether a car has a CD, Y = whether a car has an AC

In terms of probabilities the question is, e.g.: 𝑃 𝐶𝐷 ∩ 𝐴𝐶 = 𝑃 𝐶𝐷 ⋅ 𝑃 𝐴𝐶 ?

From the table: 𝑃 𝐶𝐷 ∩ 𝐴𝐶 =0.2, 𝑃 𝐶𝐷 = 0.4, and 𝑃 𝐴𝐶 = 0.7.

=> 𝑃 𝐶𝐷 ⋅ 𝑃 𝐴𝐶 =0.28
Because 𝑃 𝐶𝐷 ∩ 𝐴𝐶 ≠ 𝑃 𝐶𝐷 ⋅ 𝑃 𝐴𝐶 (0.2 ≠ 0.28)
=> conclude that AC and CD are not independent events.

To practice: Exercises 3.69

17

Bayes’ Theorem

 Consider the events A and B. Bayes’ theorem states:

𝑃 𝐴|𝐵 ⋅ 𝑃 𝐵
𝑃 𝐵|𝐴 =
𝑃 𝐴

∩
Check it is a ‘simple’ rewrite of conditional probabilities: 𝑃 𝐴|𝐵 ⋅ 𝑃 𝐵 = 𝑃 𝐴 ∩ 𝐵 => 𝑃 𝐵|𝐴 =

 The theorem is especially useful for sequential decision making in economics.

E.g., Investment strategies chosen by managers when

new information becomes available.

18

9
 28/01/2024

Bayes’ Theorem: Example, a quiz

Consider the following quiz:

 Three boxes and one of the boxes contains a prize.
 The prize is equally likely to be in one of the boxes.
 The quizmaster knows which box contains the prize.
First stage:
 The candidate chooses box 1.
Second stage:
 The quizmaster opens box 3 with the purpose of showing
it does not contain the prize.
 The quizmaster offers the candidate to switch from box 1 to box 2.

Question: Should the candidate switch from box 1 to box 2?

19

Bayes’ Theorem: Example, a quiz

Write down the information in statistical terms:

 Random variable X = box contains the prize

 𝑃 (𝑥) = , 𝑥 ∈ {1, 2, 3}. Hence, 𝑃 1 =

 Random variable 𝑌 = the box opened by the quizmaster to show it does not contain the prize. 𝑦 ∈ {2, 3}
 Whether the candidate should switch from box 1 to box 2 depends on the probability that the prize is
in box 1 given the prize is not in box 3.
Recognize that this is a conditional probability: 𝑃 | 1|3 .

20

10
 28/01/2024

Bayes’ Theorem: Example, a quiz

 X = box contains the prize

 𝑃 (𝑥) = , 𝑥 ∈ {1, 2, 3}. Hence, 𝑃 1 =
 𝑌 = the box opened by the quizmaster to show it does not contain the prize. 𝑦 ∈ {2, 3}
 Whether the candidate should switch from box 1 to box 2 depends on the probability that the prize is
in box 1 given the prize is not in box 3: 𝑃 | 1|3 .
 Also:
• If the prize is in box 1, the quizmaster is equally likely to open box 2 or 3:
𝑃 | 3|1 =

• If the prize is in box 2, the quizmaster can only open box 3:

𝑃 | 3|2 = 1

• If the prize is in box 3, the quizmaster would never open box 3:

𝑃 | 3|3 = 0

21

Bayes’ Theorem: Example, a quiz

 Apply Bayes’ theorem to find the probability that the prize is in box 1, given the quizmaster showed the
| | ⋅
prize is not in box 3: 𝑃 | 1|3 = , where 𝑃 1 = , 𝑃 | 3|1 = (previous slide)
 Compute (ingredients on previous slide):
𝑃 3 =∑ 𝑃 | 3|𝑥 ⋅ 𝑃 𝑥 = 𝑃 | 3|1 ⋅ 𝑃 1 + 𝑃 | 3|2 ⋅ 𝑃 2 + 𝑃 | 3|3 ⋅ 𝑃 3
1 1 1 1 3 1
= ⋅ +1⋅ +0⋅ = =
2 3 3 3 6 2
| | ⋅ ⋅ 𝟏
 Hence (prize in box 1), 𝑃 | 1|3 = = =
𝟑
𝟐
 And (prize in box 2), 𝑃 | 2|3 = 1 − 𝑃 | 1|3 =
𝟑
 Note: 𝑃 | 1|3 < 𝑃 | 2|3

Conclude: The candidate is advised to switch from box 1 to box 2.

Intuition: The quizmaster gave valuable information.

The candidate can update his or her choice accordingly.
To practice: Exercise 3.86

22

11
 28/01/2024

The end of this video

Next is Topic Video 3:

Random variables: Preliminaries

23

12