Bayes' Theorem

Bayes' Theorem is a mathematical formula that helps determine the conditional probability of an event based on prior knowledge and new evidence.

It adjusts probabilities when new information comes in and helps make better decisions in uncertain situations.

Bayes' Theorem helps us update probabilities based on prior knowledge and new evidence. In this case, knowing that the pet is quiet (new information), we can use Bayes' Theorem to calculate the updated probability of the pet being a cat or a dog, based on how likely each animal is to be quiet.

Bayes Theorem and Conditional Probability

Bayes' theorem (also known as the Bayes Rule or Bayes Law) is used to determine the conditional probability of event A when event B has already occurred.

The general statement of Bayes’ theorem is “The conditional probability of an event A, given the occurrence of another event B, is equal to the product of the event of B, given A, and the probability of A divided by the probability of event B.” i.e.

For example, if we want to find the probability that a white marble drawn at random came from the first bag, given that a white marble has already been drawn, and there are three bags each containing some white and black marbles, then we can use Bayes’ Theorem.

Check: Bayes’s Theorem for Conditional Probability

Bayes Theorem Formula

For any two events A and B, Bayes's formula for the Bayes theorem is given by:

Where,

• P(A) and P(B) are the probabilities of events A and B, also, P(B) is never equal to zero.
• P(A|B) is the probability of event A when event B happens,
• P(B|A) is the probability of event B when A happens.

Bayes Theorem Statement

Bayes's Theorem for n sets of events is defined as,

Let E₁, E₂,…, E_n be a set of events associated with the sample space S, in which all the events E₁, E₂,…, E_n have a non-zero probability of occurrence. All the events E₁, E₂,…, E form a partition of S. Let A be an event from space S for which we have to find the probability, then according to Bayes theorem,

P(E_i \mid A) = \frac{P(E_i) \cdot P(A \mid E_i)}{\sum_{k=1}^{n} P(E_k) \cdot P(A \mid E_k)}

for k = 1, 2, 3, …., n

Bayes Theorem Derivation

The proof of Bayes's, Theorem is given as, according to the conditional probability formula,
P(E_i \mid A) = \frac{P(E_i \cap A)}{P(A)}.....(i)

Then, by using the multiplication rule of probability, we get
P(E_i \cap A) = P(E_i) \cdot P(A \mid E_i)......(ii)

Now, by the total probability theorem,
P(A) = \sum_{k=1}^{n} P(E_k) \cdot P(A \mid E_k).....(iii)

Substituting the value of P(E_i∩A) and P(A) from eq (ii) and eq(iii) in eq(i) we get,

P(E_i \mid A) = \frac{P(E_i) \cdot P(A \mid E_i)}{\sum_{k=1}^{n} P(E_k) \cdot P(A \mid E_k)}

Bayes’ theorem is also known as the formula for the probability of “causes”. As we know, the E_i‘s are a partition of the sample space S, and at any given time, only one of the events E_i occurs. Thus, we conclude that the Bayes theorem formula gives the probability of a particular E_i, given that event A has occurred.

Terms Related to Bayes Theorem

After learning about Bayes theorem in detail, let us understand some important terms related to the concepts we covered in the formula and derivation.

Hypotheses

• Hypotheses refer to possible events or outcomes in the sample space, they are denoted as E₁, E₂, …, E_n.
• Each hypothesis represents a distinct scenario that could explain an observed event.

Priori Probability

• Priori Probability P(E_i) is the initial probability of an event occurring before any new data is taken into account.
• It reflects existing knowledge or assumptions about the event.
• Example: The probability of a person having a disease before taking a test.

Posterior Probability

• Posterior probability (P(Ei∣A) is the updated probability of an event after considering new information.
• It is derived using the Bayes Theorem.
• Example: The probability of having a disease given a positive test result.

Conditional Probability

• The probability of an event A based on the occurrence of another event B is termed conditional Probability.
• It is denoted as P(A|B) and represents the probability of A when event B has already happened.

Joint Probability

• When the probability of two or more events occurring together and at the same time is measured, it is marked as Joint Probability.
• For two events A and B, it is denoted by joint probability is denoted as P(A∩B).

Random Variables

• Real-valued variables whose possible values are determined by random experiments are called random variables.
• The probability of finding such variables is the experimental probability.

Bayes Theorem Applications

Bayesian inference is very important and has found application in various activities, including medicine, science, philosophy, engineering, sports, law, etc., and Bayesian inference is directly derived from Bayes theorem.

Some of the Key Applications are:

• Medical Testing → Finding the real probability of having a disease after a positive test.
• Spam Filters → Checking if an email is spam based on keywords.
• Weather Prediction → Updating the chance of rain based on new data.
• AI & Machine Learning → Used in Naïve Bayes classifiers to predict outcomes.

Check, Bayes' Life Applications of Bayes theorem

Difference Between Conditional Probability and Bayes Theorem

The difference between Conditional Probability and Bayes's. The theorem can be understood with the help of the table given below.

Theorem of Total Probability

Let E1, E2,…., En be mutually exclusive and exhaustive events of a sample space S, and let E be any event that occurs with some E_i​. Then, prove that :

P(E) = ⁿ∑_i=1P(E/E_i) . P(E_i)

Proof:

Let S be the sample space.
Since the events E₁, E₂,…,E_n​ are mutually exclusive and exhaustive, we have:

S = E₁ ∪ E₂ ∪ E₃  ∪ . . . ∪ En and E_i ∩ E_j = ∅ for i ≠ j.
Now, consider the event E: E = E ∩ S 
Substituting S with the union of Ei's:
⇒ E = E ∩ (E₁ ∪ E₂ ∪ E₃ ∪ . . . ∪ E_n)
Using distributive law:
⇒ E = (E ∩ E₁) ∪ (E ∩ E₂) ∪ . . . ∪ (E ∩ E_n)
Since the events Ei​ are mutually exclusive, the intersections E∩Ei are also mutually exclusive. Therefore:

P(E) = P{(E ∩ E₁) ∪ (E ∩ E₂)∪ . . . ∪(E ∩ E_n)}
⇒ P(E) = P(E ∩ E₁) + P(E ∩ E₂) + . . . + P(E ∩ E_n)
{Therefore, (E ∩ E₁), (E ∩ E₂), . . . ,(E ∩ E_n)} are pairwise disjoint}

⇒ P(E) = P(E/E₁) . P(E₁) + P(E/E₂) . P(E₂) + . . . + P(E/E_n) . P(E_n)  [by multiplication theorem]
⇒ P(E) = ⁿ∑_i=1P(E/E_i) . P(E_i)

Articles Related to Bayes' Theorem-

• Conditional Probability
• Probability Distribution

Bayes Theorem for Programmers-

• Bayes Theorem in Artificial Intelligence
• Bayes Theorem in Machine Learning
• Bayes’ Theorem in Data Mining

Solved Examples of Bayes's Theorem

Example 1: A person has undertaken a job. The probabilities of completion of the job on time with and without rain are 0.44 and 0.9, and 5, respectively. If the probability that it will rain is 0.45, then determine the probability that the job will be completed on time.

Solution:

Let E₁ be the event that the mining job will be completed on time and E₂ be the event that it rains. We have,

P(A) = 0.45,
P(no rain) = P(B) = 1 − P(A) = 1 − 0.45 = 0.55

By multiplication law of probability,
P(E₁) = 0.44, and P(E₂) = 0.95

Since, events A and B form partitions of the sample space S, by total probability theorem, we have

P(E) = P(A) P(E₁) + P(B) P(E₂)
⇒ P(E) = 0.45 × 0.44 + 0.55 × 0.95
⇒ P(E) = 0.198 + 0.5225 = 0.7205

So, the probability that the job will be completed on time is 0.7205

Example 2: There are three urns containing 3 white and 2 black balls, 2 white and 3 black balls, and 1 black and 4 white balls, respectively. There is an equal probability of each urn being chosen. One ball is equal probability chosen at random. What is the probability that a white ball will be drawn?

Solution:

Let E₁, E₂, and E₃ be the events of choosing the first, second, and third urn respectively. Then,
P(E₁) = P(E₂) = P(E₃) = 1/3

Let E be the event that a white ball is drawn. Then,
P(E/E₁) = 3/5, P(E/E₂) = 2/5, P(E/E₃) = 4/5

By theorem of total probability, we have
P(E) = P(E/E₁) . P(E₁) + P(E/E₂) . P(E₂) + P(E/E₃) . P(E₃)
⇒ P(E) = (3/5 × 1/3) + (2/5 × 1/3) + (4/5 × 1/3)
⇒ P(E) = 9/15 = 3/5

Example 3: A card from a pack of 52 cards is lost. From the remaining cards of the pack, two cards are drawn and are found to be both hearts. Find the probability of the lost card being a heart.

Solution:

Let E₁, E₂, E_3, and E₄ be the events of losing a card of hearts, clubs, spades, and diamonds respectively.
Then P(E₁) = P(E₂) = P(E₃) = P(E₄) = 13/52 = 1/4.

Let E be the event of drawing 2 hearts from the remaining 51 cards. Then,

P(E|E₁) = probability of drawing 2 hearts, given that a card of hearts is missing
⇒ P(E|E₁) = ¹²C₂ / ⁵¹C₂ = (12 × 11)/2! × 2!/(51 × 50) = 22/425

P(E|E₂) = probability of drawing 2 clubs ,given that a card of clubs is missing
⇒ P(E|E₂) = ¹³C₂ / ⁵¹C₂ = (13 × 12)/2! × 2!/(51 × 50) = 26/425

P(E|E₃) = probability of drawing 2 spades ,given that a card of hearts is missing
⇒ P(E|E₃) = ¹³C₂ / ⁵¹C₂ = 26/425

P(E|E₄) = probability of drawing 2 diamonds ,given that a card of diamonds is missing
⇒ P(E|E₄) = ¹³C₂ / ⁵¹C₂ = 26/425

Therefore,

P(E₁|E) = probability of the lost card is being a heart, given the 2 hearts are drawn from the remaining 51 cards

⇒ P(E₁|E) = P(E₁) . P(E|E₁)/P(E₁) . P(E|E₁) + P(E₂) . P(E|E₂) + P(E₃) . P(E|E₃) + P(E₄) . P(E|E₄)
⇒ P(E₁|E) = (1/4 × 22/425) / {(1/4 × 22/425) + (1/4 × 26/425) + (1/4 × 26/425) + (1/4 × 26/425)}
⇒ P(E₁|E) = 22/100 = 0.22

Hence, The required probability is 0.22.

Example 4: Suppose 15 men out of 300 men and 25 women out of 1000 are good orators. An orator is chosen at random. Find the probability that a male person is selected.

Solution:

Given,

• Total Men = 300
• Total Women = 1000
• Good Orators among Men = 15
• Good Orators among Women = 25

Total number of good orators = 15 (from men) + 25 (from women) = 40
Probability of selecting a male orator:
P(Male Orator) = Numbers of male orators / total no of orators = 15/40 = 3/8

Example 5: A man is known to speak the lies 1 out of 4 times. He throws a die and reports that it is a six. Find the probability that it Bayes' actually a six.

Solution: 

In a throw of a die, let
E₁ = event of getting a six,
E₂ = event of not getting a six and
E = event that the man reports that it is a six.

Then, P(E₁) = 1/6, and P(E₂) = (1 - 1/6) = 5/6

P(E|E₁) = probability that the man reports that six occurs when six has actually occurred 
⇒ P(E|E₁) = probability that the man speaks the truth
⇒ P(E|E₁) = 3/4

P(E|E₂) = probability that the man reports that six occurs when six has not actually occurred

⇒ P(E|E₂) = probability that the man does not speak the truth
⇒ P(E|E₂) = (1 - 3/4) = 1/4

Probability of getting a six ,given that the man reports it to be six 

P(E₁|E) = P(E|E₁) × P(E₁)/P(E|E₁) × P(E₁) + P(E|E₂) × P(E₂)     [by Bayes theorem]
⇒ P(E₁|E) = (3/4 × 1/6)/{(3/4 × 1/6) + (1/4 × 5/6)}
⇒ P(E₁|E) = (1/8 × 3) = 3/8

Hence the probability required is 3/8.