Week 3

Probability and Statistics

(Bayesian Decision Theory)
Dr. Nehad Ramaha,
Computer Engineering Department
Karabük Universities 1
The class notes are a compilation and edition from many sources. The instructor does not claim intellectual property or ownership of the lecture notes.
Probability
❖ Probability is the science of chance behavior

❖ Chance behavior is unpredictable in the

short run but has a regular and predictable
pattern in the long run
❖ Random: individual outcomes are uncertain

❖ But there is a regular distribution of outcomes in a

large number of repetitions.

❖ Example: select any number from a bag of numbers

{1,2,3,…,100}
 ❖ …a random experiment is an action or process that leads to
❖If an experiment has n possible outcomes [all equally
one of several possible outcomes. For example:
likely to occur].
Experiment Outcomes

Flip a coin Heads, Tails

Selecting a color ball Green, red, blue

Rolling a die 1,2,3,4,5,6

Picking a card from a

52 cards
deck
❖ Relative frequency (proportion of occurrences) of an
outcome settles down to one value over the long run.
That one value is then defined to be the probability of
that outcome.

❖ Can be determined (or checked) by observing a long

series of independent trials (empirical data)

➢experience with many samples

➢simulation
Coin flipping:
 The sample space S of a random phenomenon is the
set of all possible outcomes.

 An event is an outcome or a set of outcomes (subset of

the sample space).

 A probability model is a mathematical description of

long-run regularity consisting of a sample space S and a
way of assigning probabilities to events.
 Even
t3 Even
Even t4
t1
Sample Space

Even
Even
t5
t2
 Rolling an odd
number={2,4,6
}
Rolling an even
number={2,4,6}

Sample Space
={1,2,3,4,5,6}
Rolling a prime
number={2,3,5}
Random phenomenon: roll pair of fair dice.
Sample space:

Event: rolling even numbers on both dice

11
Random phenomenon: Arrange 52 card deck in a zigzag way
Sample space:

Event: pick an ace

What is a PROBABILITY?
- Probability is the chance that some event
will happen

- It is the ratio of the number of ways a

certain event can occur to the number of
possible outcomes
What is a PROBABILITY?

number of favorable outcomes

P(event) = number of possible outcomes

Examples that use Probability:

(1) Dice, (2) Spinners, (3) Coins, (4) Deck of Cards, (5)
Evens/Odds, (6) Alphabet, etc.
What is a PROBABILITY?

0 ¼ or .25 ½ or .5 ¾ or .75 1

Impossible Not Very Equally Likely Somewhat

Certain
Likely Likely
Example 2: Roll a dice.
What is the probability of rolling an even number?

# 𝑓𝑎𝑣𝑜𝑟𝑎𝑏𝑙𝑒 𝑜𝑢𝑡𝑐𝑜𝑚𝑒𝑠 3 1
𝑃 𝑒𝑣𝑒𝑛 # = = =
#𝑝𝑜𝑠𝑠𝑖𝑏𝑙𝑒 𝑜𝑢𝑡𝑐𝑜𝑚𝑒𝑠 6 2

The probability of rolling an even number

is 3 out of 6.
Example 3: Roll a dice.
Random phenomenon: roll pair of fair dice and
count the number of pips on the up-faces.
Find the probability of rolling a 5.

P(roll a 5) = P( )+P( )+P( )+P( )

= 1/36 + 1/36 + 1/36 + 1/36
= 4/36
= 0.111
18
Example 4: Spinners.
What is the probability of spinning green?
# 𝑓𝑎𝑣𝑜𝑟𝑎𝑏𝑙𝑒 𝑜𝑢𝑡𝑐𝑜𝑚𝑒𝑠 1
𝑃 𝑔𝑟𝑒𝑒𝑛 = =
#𝑝𝑜𝑠𝑠𝑖𝑏𝑙𝑒 𝑜𝑢𝑡𝑐𝑜𝑚𝑒𝑠 4

The probability of spinning green is 1 out of 4

Example 5: Flip a coin.
What is the probability of flipping a tail?

# 𝑓𝑎𝑣𝑜𝑟𝑎𝑏𝑙𝑒 𝑜𝑢𝑡𝑐𝑜𝑚𝑒𝑠 1
𝑃 𝐻𝑒𝑎𝑑 = =
#𝑝𝑜𝑠𝑠𝑖𝑏𝑙𝑒 𝑜𝑢𝑡𝑐𝑜𝑚𝑒𝑠 2

The probability of spinning green is 1 out of 2

Example 6: Deck of Cards.
 What is the probability of picking a heart?

# 𝒇𝒂𝒗𝒐𝒓𝒂𝒃𝒍𝒆 𝒐𝒖𝒕𝒄𝒐𝒎𝒆𝒔 𝟏𝟑 𝟏
𝑃 𝐻𝑒𝑎𝑟𝑡 = = =
#𝒑𝒐𝒔𝒔𝒊𝒃𝒍𝒆 𝒐𝒖𝒕𝒄𝒐𝒎𝒆𝒔 𝟓𝟐 𝟒
The probability of picking a heart is 1 out of 4

 What is the probability of picking a non heart?

# 𝒇𝒂𝒗𝒐𝒓𝒂𝒃𝒍𝒆 𝒐𝒖𝒕𝒄𝒐𝒎𝒆𝒔 𝟑𝟗 𝟑
𝑃 𝑛𝑜𝑛 − 𝐻𝑒𝑎𝑟𝑡 = = =
#𝒑𝒐𝒔𝒔𝒊𝒃𝒍𝒆 𝒐𝒖𝒕𝒄𝒐𝒎𝒆𝒔 𝟓𝟐 𝟒
The probability of picking a heart is 3 out of 4
Key Concepts:

- Probability is the chance that some event will

happen

- It is the ratio of the number of ways a certain

even can occur to the total number of possible
outcomes
Complementary Events
❖ The complement of an event E is the set of all
outcomes in a sample space that are not included in
event E.

❖ The complement of an event E is denoted by 𝐸 ′ 𝑜𝑟 𝐸ത

0  P( E )  1
P( E ) + P( E ) = 1
Properties of Probability:
P( E ) = 1 − P( E )
P( E ) = 1 − P( E )
Complementary Events
❖ Example I: A sequence of 5 bits is randomly generated. What is the
probability that at least one of these bits is zero?
❖ Solution: There are 25 = 32 possible outcomes of generating such a
sequence.
Define event E as at least one of the bits is zeros
ത “none of the bits is zero”, includes only one
Then event 𝐸,
of these outcomes, namely the sequence 11111.
ത = 1/32.
Therefore, p(𝐸)
Now p(E) can easily be computed as
ത = 1 – 1/32 = 31/32.
p(E) = 1 – p(𝐸)
❖ We talk about conditional probability when the probability of one
event depends on whether or not another event has occurred.

❖ e.g. There are 2 red and 3 blue counters in a bag and, without
looking, we take out one counter and do not replace it.

❖ The probability of a 2nd counter taken from the bag being red
depends on whether the 1st was red or blue.
❖ Conditional probability problems can be solved by considering the
individual possibilities or by using a table, a Venn diagram, a tree
diagram or a formula.
P(A B) means

“the probability that event A occurs given that B

has occurred”. This is conditional probability.
e.g. 1. The following table gives data on the type of car, grouped
by petrol consumption, owned by 100 people.

Low Medium High Total

Male 12 33 7
Female 23 21 4

100

One person is selected at random.

L is the event “the person owns a low rated car”
Example

e.g. 1. The following table gives data on the type of car, grouped
by petrol consumption, owned by 100 people.

Low Medium High Total

Male 12 33 7
Female 23 21 4
100

One person is selected at random.

L is the event “the person owns a low rated car”
F is the event “a female is chosen”.
e.g. 1. The following table gives data on the type of car, grouped
by petrol consumption, owned by 100 people.
Low Medium High Total
Male 12 33 7
Female 23 21 4

One person is selected at random. 100

L is the event “the person owns a low rated car”
F is the event “a female is chosen”.
e.g. 1. The following table gives data on the type of car, grouped by
petrol consumption, owned by 100 people.

Low Medium High Total

Male 12 33 7
Female 23 21 4

One person is selected at random. 100

L is the event “the person owns a low rated car”
F is the event “a female is chosen”.
Find (i) P(L) (ii) P(F and L) (iii) P(F L)

We need to be careful which row or column we look at.

Solution: Low Medium High Total
Male 12 33 7
Female 23 21 4
35 100
Find (i) P(L) (ii) P(F and L) (iii) P(F L)
35 7 7
(i) P(L) = =
100 20 20
Solution: Low Medium High Total
Male 12 33 7
Female 23 21 4
100
Find (i) P(L) (ii) P(F and L) (iii) P(F L)
35 7 7
(i) P(L) = =
100 20 20
23 The probability of selecting a
(ii) P(F and L) =
100 female with a low rated car.
Solution: Low Medium High Total
Male 12 33 7
Female 23 21 4
35 100
Find (i) P(L) (ii) P(F and L) (iii) P(F L)
35 7 7
(i) P(L) = =
100 20 20
23
(ii) P(F and L) =
100
23 We
(iii) P(F L) = Themust be careful
probability with the a female
of selecting
35 denominators
given the car isinlow
(ii) rated.
and (iii). Here we
are given the car is low rated. We want
the total of that column.
Solution: Low Medium High Total
Male 12 33 7
Female 23 21 4
100
Find (i) P(L) (ii) P(F and L) (iii) P(F L)
35 7 7
(i) P(L) = =
100 20 20 Notice that
23 1
7 23 23
(ii) P(F and L) = P(L)  P(F L) =  =
100 20 35 5 100
23 = P(F and L)
(iii) P(F L) =
35
So, P(F and L) = P(F L)  P(L)
Conditional Probability

P(F and L) = P(F L)  P(L)

❖ This result can be used to help solve harder conditional probability

problems.
 Bayesian Decision Theory is a fundamental
statistical approach that quantifies the tradeoffs
between various decisions using probabilities
and costs that accompany such decisions.
 First, we will assume that all probabilities are
known.
 Then, we will study the cases where the
probabilistic structure is not completely known.

36
 Design classifiers to recommend decisions
that minimize some total expected ”risk”.
 The simplest risk is the classification error
(i.e., costs are equal).
 Typically, the risk includes the cost
associated with different decisions.

37
 State of nature ω (random variable):
◦ e.g., ω1 for sea bass, ω2 for salmon
 Probabilities P(ω1) and P(ω2) (priors):
◦ e.g., prior knowledge of how likely is to get a
seabassor a salmon
 Probability density function p(x) (evidence):
◦ e.g., how frequently we will measure a pattern with
feature value x (e.g., x corresponds to lightness)

38
39
40
41
42
43
44