Al-Imam Mohammad bin Saud University

CS433: Modeling and Simulation

Lecture 03: Probability Review

Dr. Anis Koubâa

05 March 2010
Goals of today

 Review the fundamental concepts of

probability
 Understand the difference between discrete
and continuous random variable

2
Topics

 Fundamental Laws of Probability

 Discrete Random Variable
 Probability Mass Function (PMF)
 Continuous Random Variable
 Probability Density Function (PDF)
 Cumulative Distribution Function (CDF)
 Mean and Variance

3
Probability Review
Fundamentals

 We measure the probability for Random Events

 How likely an event would occur
 The set of all possible events is called Sample Space
 In each experiment, an event may occur with a certain
probability (Probability Measure)
 Example:
 Tossing a dice with 6 faces
 The sample space is {1, 2, 3, 4, 5, 6}
 Getting the Event « 2 » in on experiment has a probability 1/6
5
Probability

 The probability of every set of possible events is between 0 and 1,

inclusive.
 The probability of the whole set of outcomes is 1.
 Sum of all probability is equal to one
 Example for a dice: P(1)+P(2)+P(3)+ P(4)+P(5)+P(6)=1
 If A and B are two events with no common outcomes, then the
probability of their union is the sum of their probabilities.
 Event E1={1},
 Event E2 ={6}
 P(E1 U E2)=P(E1)+P(E2)

6
 Complementary Event

 Complementary Event of A is not(A)

 P(A)=1-P(not A)
 The probability that event A will not happen is 1-P(A).
 Example
 Event E1={1}
 Probability to get a value different from {1} is 1-P(E1).

7
 Joint Probability

 Event Union (U = OR) Event Intersection (∩= AND)

 Joint Probability (A ∩ B)
 The probability of two events in conjunction. It is the probability of both events together.

p  A B   p  A  p  B   p  A B 

 Independent Events
 Two events A and B are independent if

p A B   p A p B 

8
Example on Independence
E1: Drawing Ball 1 P(E1): 1/3
E2: Drawing Ball 2 P(E2):1/3 p  A B   p  A   p  B 
E3: Drawing Ball 3 P(E3): 1/3

Case 1: Drawing with replacement of the ball

1 2
The second draw is independent of the first draw
3
1 1 1
p  E 1  E 2     p  E 1  p  E 2 
3 3 9

Case 2: Drawing without replacement of the ball

The second draw is dependent on the first draw
1 1 1
p  E 1  E 2      p  E 1  p  E 2 
3 2 6
1
Quiz: Show that in Case 2, we have p  E 1 E 2 
2
9
Conditional Probability
 Conditional Probability p(A|B) is the probability of some
event A, given the occurrence of some other event B.
p A B 
pA |B  pB |A 
p  B A 

pB PA

 If A and B are independent,

then p  A | B   p  A and
 pB |A  pB
 If A and B are independent, the conditional probability of A, given B is
simply the individual probability of A alone; same for B given A.
 p(A) is the prior probability;
 p(A|B) is called a posterior probability.
 Once you know B is true, the universe you care about shrinks to B.

10
Example on Independence
E1: Drawing Ball 1 P(E1): 1/3 p A B 
E2: Drawing Ball 2 P(E2):1/3 pA |B 
E3: Drawing Ball 3 P(E3): 1/3 pB
Case 1: Drawing with replacement of the ball
1 2
The second draw is independent of the first draw
3
1 1 1
p  E 1| E 2   p  E 1| E 2  
p  E 1 E 2 3  3 1
 
3 p  E 2 1 3
3
Case 2: Drawing without replacement of the ball
The second draw is dependent on the first draw
1 1 1
p  E 1| E 2   p  E 1| E 2  
p  E 1 E 2 3  2 1
 
2 p  E 2 1 2
3

11
Baye’s Rule

 We know that
p  A B   p  B A
 Using Conditional Probability definition, we have
p  A | B  P  B   p  B | A  P  A 

 The Bayes rule is:

pB |ApA
pA |B 
pB
12
Law of Total Probability

 In probability theory, the law of total probability means

the prior probability of A, P(A), is equal to
the expected value of the posterior probability of A.
 That is, for any random variable N,
p  A   E  p  A | N  
 where p(A|N) is the conditional probability of A given N.

13
Law of Total Probability

 Law of alternatives: The term law of total probability is

sometimes taken to mean the law of alternatives, which is a
special case of the law of total probability applying to
discrete random variables.
 if { Bn : n = 1, 2, 3, ... } is a finite partition of a probability
space and each set Bn is measurable, then for any event A we
have pA  n
p  A  Bn 

or, alternatively, (using Rule of Conditional Probability)

pA  pA |B
n
n pB
14
 Example: Law of Total Probability
Sample Space S  {1, 2,3, 4,5, 6, 7}

Partitions B 1  {1,5} B 2  {2,3, 6} B 3  {4, 7}

Event A  {3}

Law of Total Probability

p  A   p  {3}  p  A  B1   p  A  B 2   p  A  B 3 
 0  p  {3}  {2,3, 6}  0  p  {3}
15
 Five Minutes Break

You are free to discuss with your classmates about the

previous slides, or to refresh a bit, or to ask questions.

Administrative issues
 Groups Formation (mandatory to do it this week)
A random variable

► Random variable is a measurable function from a

sample space of events  to the measurable space S
of values of possible values of the variable

X : S
Ei  x i
Event Value

Each value/event has a probability of occurrence

17
A random variable

► Random Variable is also know as stochastic

variable.
► A random variable is not a variable. It is a function. It maps
from the sample space to the real numbers.
► random variable is defined as a quantity whose values
are random and to which a probability distribution is
assigned.

18
A random variable: Examples.

► The number of packets that arrives to the destination

► The waiting time of a customer in a queue
► The number of cars that enters the parking each hour
► The number of students that succeed in the exam

19
Random Variable Types
► Discrete Random Variable:
► possible values are discrete (countable sample space,
Integer values)
X :   {1, 2,3, 4,...}
Ei  x i
► Continuous Random Variable:
► possible values are continuous (uncountable space, Real
X :    1.4,32.3
values)

Ei  x i
20
Discrete Random Variable
 The probability distribution for discrete random variable
is called Probability Mass Function (PMF).
p xi   P  X  xi 
 Properties of PMF
0  p  xi  1 and pxi
i  1
 Cumulative Distribution Function (CDF)
p  X  x   x x
p (x i )
i

n
 Mean value
X  E  X   x
i 1
i  p xi 
21
Discrete Random Variable
 Mean (Expected) value
n
X  E  X   x
i 1
i p  xi 
 Variance
V X   X2  E X E X 
2
E X  
2
E X  2
General Equation

n
V X     x i  x  pxi 
2

i 1 For Discrete RV
2
 n   n 
V X     xi   p  xi    xi pxi  
2
  
 i 1   i 1 
22
 Discrete Random Variable: Example
 Random Variable: Grades of the students

Student ID 1 2 3 4 5 6 7 8 9 10

Grade 3 2 3 1 2 3 1 3 2 2

Probability Mass Function PDF

2
p  1  P  X  1   0.2 0  p  1  1
10
4
p  2  P  X  2   0.4 0  p  2  1
10
4
p  3  P  X  3   0.4 0  p  3  1
10 Grade

23
 Discrete Random Variable: Example
 Random Variable: Grades of the students
Student ID 1 2 3 4 5 6 7 8 9 10

Grade 3 2 3 1 2 3 1 3 2 2

Probability Mass Function Property CDF

px
i
i   p  1  p  2   p  3  1

Cumulative Distribution Function

p  X  x   x x
p (x i )
i

p  X  2   x p (x i )  p  1  p  2   0.2  0.4  0.6

Grade
i 2

p  X  3   x 2
p (x i )  p  1  p  2   p  3  1
i

24
 Discrete Random Variable: Example
 Random Variable: Grades of the students
Student ID 1 2 3 4 5 6 7 8 9 10

Grade 3 2 3 1 2 3 1 3 2 2

The Mean Value

x  p  x
i
i i   1 0.2  2  0.4  3  0.4  2.2

Grade
i
 2.2
10
25
Continuous Random Variable
 The probability distribution for continuous random
variable is called Probability Density Function (PDF).
f x 
 The probability of a given value is always 0 p  x  x i   0
 The sample space is infinite
 For continuous random variable, we compute p  a  x  b 
 Properties of PDF
1. f ( x)  0 , for all x in R X
2.  f ( x)dx  1
RX

3. f ( x)  0, if x is not in RX
26
Continuous Random Variable
 Cumulative Distribution Function (CDF)

p  X  x   x x
p (x i )
i

x
p  X  x    f  t  dt  0


 Mean/Expected value b
p  a  X  b    f  x  dx

a
X  E  X  

 x  f  x  dx
 Variance
   
V X    x    f  x  dx 
V  X    x 2  f  x  dx
2
x and    x2
 
   
27
Discrete versus Continuous Random Variables
Discrete Random Variable Continuous Random Variable

Finite Sample Space Infinite Sample Space

e.g. {0, 1, 2, 3} e.g. [0,1], [2.1, 5.3]

Probability Mass Function (PMF) Probability Density Function (PDF)

p xi   P X  xi  f x 
1. f ( x)  0 , for all x in R X
1. p (x i )  0, for all i
2.  i 1 p (x i )  1
 2.  f ( x)dx  1
RX

3. f ( x)  0, if x is not in RX

Cumulative Distribution Function (CDF) pX x 

p  X  x   x
x

x
p (x i ) p  X  x    f  t  dt  0
i 
b
p  a  X  b    f  x  dx
a

28
 Continuous Random Variables: Example

Example: modeling the waiting time in a queue

 People waiting for service in bank queue

 Time is a continuous random variable

 Random Time is typically modeled as exponential distribution
  is the mean value

1  x 
Exponential Distribution   exp   , x 0
f (x )     
Exp (µ) 0,
 otherwise

29
 Continuous Random Variables: Example

 We assume that with average waiting time of one customer

is 2 minutes
PDF: f (time)

 1  x / 2
e , x0
f ( x)   2
0, otherwise

time
30
 Continuous Random Variables: Example
 Probability that the customer waits exactly 3 minutes is:
1 3  x /2
P (x  3)  P (3  x  3)  3 e dx  0
2
 Probability that the customer waits between 2 and 3 minutes is:
1 3  x /2
P (2  x  3)   e dx  0.145
2 2
P(2  X  3)  F (3)  F (2)  (1  e  ( 3 / 2 ) )  (1  e 1 )  0.145 CDF

 The Probability that the customer waits less than 2 minutes

P (0  X  2)  F (2)  F (0)  F (2)  1  e 1  0.632 CDF

31
 Continuous Random Variables: Example

Expected Value and Variance

 The mean of life of the previous device is:


E (X ) 
1


xe x /2
dx   xe  x /2  
  e  x / 2dx  2
2 0
  0
0

 To compute variance of X, we first compute E(X2):


1 x / 2
E ( X )   x e dx   x e
 
2 2 x/ 2 2   e  x / 2 dx  8
2 0 0
0

 Hence, the variance and standard deviation of the device’s life are:
V ( X )  8  22  4
  V (X )  2
32
 Two Minutes Break

You are free to discuss with your classmates about the

previous slides, or to refresh a bit, or to ask questions.

Administrative issues
 Groups Formation (mandatory to do it this week)
Variance

 The standard deviation is defined as the square root

of the variance, i.e.:

X  2
X  V X  s

34
Coefficient of Variation

 The Coefficient of Variance of the random variable X is

defined as:

V X  X
CV X   
E X  X

35
 End of lecture
There are some additional slides afterwards that may help to
explain some concepts
Discrete Probability Distribution

 The probability distribution of a discrete random

variable is a list of probabilities associated with each of
its possible values.
 It is also sometimes called the probability function or the
probability mass function (PMF) for discrete random
variable.

37
Probability Mass Function (PMF)
 Formally
 the probability distribution or probability mass function
(PMF) of a discrete random variable X is a function that
gives the probability p(xi) that the random variable equals xi,
for each value xi:
p xi   P  X  xi 
 It satisfies the following conditions:
0  p  xi  1
px
i
i  1

38
Continuous Random Variable

 A continuous random variable is one which takes an infinite

number of possible values.
 Continuous random variables are usually measurements.
 Examples include height, weight, the amount of sugar in an orange,
the time required to run a mile.

39
Probability Density Function (PDF)
 For the case of continuous variables, we do not want to ask
what the probability of "1/6" is, because the answer is always
0...
 Rather, we ask what is the probability that the value is in the
interval (a,b).
 So for continuous variables, we care about the derivative of the
distribution function at a point (that's the derivative of an
integral). This is called a probability density function (PDF).
 The probability that a random variable has a value in a set A is
the integral of the p.d.f. over that set A.

40
Probability Density Function (PDF)
 The Probability Density Function (PDF) of a continuous
random variable is a function that can be integrated to obtain
the probability that the random variable takes a value in a
given interval.
 More formally, the probability density function, f(x), of a
continuous random variable X is the derivative of the
cumulative distribution function F(x):
d
f x   Fx 
dx
 Since F(x)=P(X≤x), it follows that:
b


F  b   F  a   P  a  X  b   f  x   dx
a
41
Cumulative Distribution Function (CDF)

 The Cumulative Distribution Function (CDF) is a

function giving the probability that the random variable X
is less than or equal to x, for every value x.
 Formally
 the cumulative distribution function F(x) is defined to be:

    x  ,
Fx  PX x 

42
Cumulative Distribution Function (CDF)

 For a discrete random variable, the cumulative distribution

function is found by summing up the probabilities as in the
example below.
    x  ,

Fx  PX x    P (X
x i x
 xi )   p (x
x i x
i )

 For a continuous random variable, the cumulative

distribution function is the integral of its probability density
function f(x).
b


F  a   F  b   P  a  X  b   f  x   dx
a
43
Cumulative Distribution Function (CDF)

► Example
► Discrete case: Suppose a random variable X has the following
probability mass function p(xi):
xi 0 1 2 3 4 5
p(xi) 1/32 5/32 10/32 10/32 5/32 1/32

► The cumulative distribution function F(x) is then:

xi 0 1 2 3 4 5
F(xi) 1/32 6/32 16/32 26/32 31/32 32/32

44
Mean or Expected Value

n
X  E  X   x
i 1
i  p  xi 


X  E  X   

x  f  x  dx
45
Example: Mean and variance

 When a die is thrown, each of the possible faces 1, 2, 3, 4, 5, 6

(the xi's) has a probability of 1/6 (the p(xi)'s) of showing. The
expected value of the face showing is therefore:

µ = E(X) = (1 x 1/6) + (2 x 1/6) + (3 x 1/6) + (4 x

1/6) + (5 x 1/6) + (6 x 1/6) = 3.5

 Notice that, in this case, E(X) is 3.5, which is not a possible

value of X.

46
Variance

 The variance is a measure of the 'spread' of a

distribution about its average value.
 Variance is symbolized by V(X) or Var(X) or σ2.
 The mean is a way to describe the location of a
distribution,
 the variance is a way to capture its scale or degree of
being spread out. The unit of variance is the square of the
unit of the original variable.

47
Variance
 The Variance of the random variable X is defined as:

V X  2
 X  E X E X 
2
E X   E X 
2 2

 where E(X) is the expected value of the random variable X.

 The standard deviation is defined as the square root
of the variance, i.e.:

 X  X2  V  X  s

48