Chapter 3

Axioms of Probability

Probability, or probability theory, is a framework for dealing with uncertainty

and randomness. It provides the required tools for setting up a probabilistic
model. A probabilistic model is a mathematical description of an uncertain
situation. The main elements of a probabilistic model are illustrated in Fig.
3.1. This chapter is devoted to the introduction of these elements.

3.1 Sample Space and Events

In the context of probability, an experiment is a random occurrence that pro-
duces one of several outcomes. The set of all possible outcomes is called the
sample space of the experiment, and it is denoted by Ω. An admissible subset
of the sample space is called an event (see Fig. 3.2).

Example 3.1.
(a) Flipping a coin: The sample space is Ω = {Head, Tail}.
(b) Rolling a die: The sample space is Ω = {1, 2, 3, 4, 5, 6}.
(c) Flipping two coins: The sample space is Ω = {HH, HT, T H, T T }

Probability law

Event B
 
Experiment
Event A
 

Sample Space

  Events

Figure 3.1: The main elements of a probabilistic model

10
 Figure 3.2: An illustration of sample space, outcome, and events

Example 3.2. The rolling of a die forms a common experiment. The set
of prime numbers less than or equal to six, namely {2, 3, 5}, is one of many
possible events. The actual number observed after rolling the die is the outcome
of the experiment.

Note that the sample space should be (a) collectively exhaustive: it should
include all the possible outcomes and (b) mutually exclusive: if one outcome
happens the others may not.

3.2 Probability Laws

A probability law assigns a number Pr(A) to an event A. The probability
Pr(A) describes our knowledge or belief about the likelihood of the event A.
The probability of an event which is assigned by the probability law should
satisfy the following axioms:

ˆ (Non-negativity) Pr(A) ≥ 0
ˆ (Additivity) If A and B are two disjoint events, then the probability
of their union is
Pr(A ∪ B) = Pr(A) + Pr(B) (3.1)

ˆ (Normalization) The probability of the entire sample space is equal to

1, i.e. Pr(Ω) = 1.

Note: If events A and B are not disjoint, then the probability of their union
is given by
Pr(A ∪ B) = Pr(A) + Pr(B) − Pr(A ∩ B) (3.2)

11
 Consider an experiment involving a single coin toss with the sample space
Ω = {H, T }. If the coin is fair, then P(A) = P(B). The additivity axiom
implies that
P(Ω) = P(H ∪ T ) = P(H) + P(T ) = 1

Hence,
P(H) = P(T ) = 0.5

Similarly, for an experiment of rolling a die the probability of each outcome

is 1/6. Consider the event

A = {even number occurs} = {2, 4, 6}

Using additivity,

P(A) = P(2) + P(4) + P(6) = 1/6 + 1/6 + 1/6 = 3/6

Based on the previous examples we give the following definition.

Discrete Uniform Probability Law

For a sample space that consists of n equally likely possible outcomes, the
probability of any event A is given by

number of elements of A
P(A) = (3.3)
n

3.3 Conditional Probability

Conditional probability provides a way to compute the likelihood of an event
based on partial information. We start the description of conditional prob-
ability with an illustrative example. Suppose that two dice are tossed. The
sample space for this experiment consists of 36 possible outcomes with equal
probabilities of 1/36. Furthermore, suppose that a 3 is observed at the first
die. Given this information, what is the probability that the sum of the two
dice equals 8? Given that the initial die is a 3, there are six possible outcomes
for this experiment, i.e. the initial sample space with 36 possible outcomes
now is reduced to

Ω = {(3, 1), (3, 2), (3, 3), (3, 4), (3, 5), (3, 6)}

12
with the (conditional) probability of each outcome being 1/6. Hence, the
probability of having a sum of 8 given that the first die is 3 equals 1/6.
If we let A and B denote the event that the sum is 8 and the event that
the first die is 3, respectively, then the probability obtained above is called the
conditional probability that A occurs given that B has occurred and is defined
as
Conditional Probability
If Pr(B) > 0
Pr(A, B)
Pr(A|B) = (3.4)
Pr(B)

Example 3.3. We toss a fair coin three successive times. What is the condi-
tional probability Pr{A|B} when A and B are the events
A = {more heads than tails come up} , B = {1st toss is head}
Solution: The sample space is given by

Ω = {HHH, HHT, HT H, HT T, T HH, T HT, T T H, T T T }

The event B consists of the four elements HHH, HHT, HT H, HT T , so its

probability is
4
Pr{B} =
8
The event (A, B) consists of three elements HHH, HHT, HT H. Hence,

3
Pr{A, B} =
8

Thus, the conditional probability is given by

Pr{A, B} 3/8 3
Pr{A|B} = = =
Pr{B} 4/8 4

Example 3.4. (unconditional from conditional) If an aircraft is present in

a certain area, a radar detects it and generates an alarm signal with probability
0.99. If an aircraft is not present, the radar generates a (false) alarm, with
probability 0.10. We assume that an aircraft is present with probability 0.05.
What is the probability of no aircraft presence and a false alarm? What is the
probability of aircraft presence and no detection?

13
Solution: This experiment can be represented by a sequential representation.
Let A and B be the events

A = {an aircraft is present}

B = {the radar generates an alarm}

Then, the sequential representation of this experiment is given as in Fig. 3.3.

The probability of the possible outcomes is equal to the product of the prob-
abilities associated with the branches in the path from the root to the leaf.
Hence the desired probabilities are given by

Pr{not present, false alarm} = Pr{AC , B} = Pr{AC } Pr{B|AC }

= 0.95 × 0.01 = 0.095

Pr{present, no detection} = Pr{A, B C } = Pr{A} Pr{B C |A}

= 0.05 × 0.01 = 0.0005



 

































 







Figure 3.3: The sequential representation of example 3.4

14
3.4 Total Probability Theorem and Bayes’ Rule
Total Probability Theorem
If A1 , A2 , ..., An are disjoint events that from a partition of the sample
space, then

Pr(B) = Pr(A1 , B) + Pr(A2 , B) + ... + Pr(An , B) (3.5)

= Pr(A1 ) Pr(B|A1 ) + Pr(A2 ) Pr(B|A2 ) + ... + Pr(An ) Pr(B|An )
(3.6)

This theorem is illustrated in Fig. 3.4. Initially, the sample space is partitioned
into a number of disjoint events Ai ’s. Then the probability that B occurs is a
weighted average of its conditional probability under each event.

Example 3.5. The probability of winning for a football team is 0.3 against
half of the teams (call them type 1), 0.4 against a quarter of the teams (type
2), and 0.5 against the remaining teams (type 3). What is the probability of
winning against a random team?
Solution: Let Ai be the event of playing against a type i team. Then,
Pr(A1 ) = 0.5, Pr(A2 ) = 0.25, Pr(A3 ) = 0.25
and let B be the event of winning. Hence,
Pr(B|A1 ) = 0.3, Pr(B|A2 ) = 0.4, Pr(B|A3 ) = 0.5,
Using total probability theorem, the probability of winning is

Pr(B) = Pr(A1 ) Pr(B|A1 ) + Pr(A2 ) Pr(B|A2 ) + Pr(A3 ) Pr(B|A3 )

= 0.5 × 0.3 + 0.25 × 0.4 + 0.25 × 0.5 = 0.375

Figure 3.4: The illustration of the total probability theorem

15
 The total probability theorem is often used in conjunction with the fol-
lowing theorem, which relates conditional probabilities of the form Pr(A|B)
with conditional probabilities of the form Pr(B|A), in which the order of the
conditioning is reversed.

Bayes’ Rule
Let A1 , A2 , ..., An are disjoint events that from a partition of the sample
space. Then
Pr(Ai ) Pr(B|Ai )
Pr(Ai |B) = (3.7)
Pr(B)
Combining this with the total probability theorem results in

Pr(Ai ) Pr(B|Ai )
= (3.8)
Pr(A1 ) Pr(B|A1 ) + ... + Pr(An ) Pr(B|An )

Bayes’ rule is often used for inference. There are a number of ”causes” that
may result in a certain ”effect”. We observe the effect and wish to infer the
cause. The events A1 , A2 , ..., An are associated with the causes and the event
B represents the effect. Given that the effect B has been observed, we wish
to evaluate the probability that the cause Ai is present, i.e. Pr(Ai |B). We
refer to Pr(Ai |B) as the posterior probability of event Ai given the prior
probability Pr(Ai ). The cause-effect versus inference in illustrated in Fig.
3.5.

Example 3.6. In the radar problem of example 3.4, what is the probability
that an aircraft is present when an alarm is generated?
Solution: Solution: Let A and B be the events

A = {an aircraft is present}

B = {the radar generates an alarm}

By Bayes’ rule, the desired probability is given by

Pr(A) Pr(B|A)
Pr(A|B) = (3.9)
Pr(B)
Pr(A) Pr(B|A)
= (3.10)
Pr(A) Pr(B|A) + Pr(AC ) Pr(B|AC )
0.05 × 0.99
= = 0.342 (3.11)
0.5 × 0.99 + 0.95 × 0.1

16
 cause-effect
 
   

inference
 
   
Figure 3.5: Cause-effect and inference

Example 3.7. In the football problem of example 3.5, if the team wins what
is the probability that it played against a type 1 team?
Solution: Ai is the event of playing against a type i team,
Pr(A1 ) = 0.5, Pr(A2 ) = 0.25, Pr(A3 ) = 0.25
and B is the event of winning.
Pr(B|A1 ) = 0.3, Pr(B|A2 ) = 0.4, Pr(B|A3 ) = 0.5,
Using Bayes’ rule, the desired probability Pr(A1 |B) is given by

Pr(A1 ) Pr(B|A1 )
Pr(A1 |B) =
Pr(A1 ) Pr(B|A1 ) + Pr(A2 ) Pr(B|A2 ) + Pr(A3 ) Pr(B|A3 )
0.5 × 0.3
= = 0.4
0.5 × 0.3 + 0.25 × 0.4 + 0.25 × 0.5

3.5 Independence
The conditional probability Pr(A|B) captures the partial information that
event B provides about event A. An important special case arises when the oc-
currence of B provides no such information and does not alter the probability
that A has occurred, i.e.,
Pr(A|B) = Pr(A)

When the above equality holds, we say that A is independent of B. Given that
Pr(A|B) = Pr(A, B)/ Pr(B), this is equivalent to

Pr(A, B) = Pr(A) Pr(B)

Independence is a symmetric property; that is, if A is independent of B, then

B is independent of A, and we say that A and B are independent events.

17
 A common first thought is that two events are independent if they are
disjoint, but in fact the opposite is true: two disjoint events A and B with
Pr(A) > 0 and Pr(B) > 0 are never independent, since their intersection
A ∩ B is empty and has probability 0. For example, an event A and its
complement AC are not independent (unless Pr(A) = 0 or Pr(A) = 1) , since
knowledge that A has occurred provides precise information about whether AC
has occurred.

Example 3.8. Consider an experiment involving two successive rolls of a 4-

sided die in which all 16 possible outcomes are equally likely and have proba-
bility 1/16. Are the following events independent?
(a) Ai = {1st roll is i}, Bj = {2nd roll is j}
(b) A = {1st roll is 1}, B = {sum of the two rolls is 5}
(c) [HW] A = {max. of the two rolls is 2}, B = {min of the two rolls is 2}
Solution:
(a) We have
1
Pr(Ai , Bj ) = Pr({i, j}) =
16
and
4
Pr(Ai ) =
16
4
Pr(Bj ) =
16
Hence,
Pr(Ai , Bj ) = Pr(Ai ) Pr(Bj )

and the two events are independent.

(b) We have
1
Pr(A, B) = Pr({1, 4}) =
16
and
1
Pr(A) =
4
Pr(B) = Pr{(1, 4), (2, 3), (3, 2), (4, 1)} = 4/16

Hence,
Pr(Ai , Bj ) = Pr(Ai ) Pr(Bj )

and the two events are independent.

18