Bayesian

Decision Theory

Chapter 2 (Duda, Hart & Stork)

CS 7616 - Pattern Recognition

Henrik I Christensen
Georgia Tech.
 Bayesian Decision Theory
• Design classifiers to recommend decisions that
minimize some total expected ”risk”.
– The simplest risk is the classification error (i.e., costs
are equal).
– Typically, the risk includes the cost associated with
different decisions.
 Terminology
• State of nature ω (random variable):
– e.g., ω1 for sea bass, ω2 for salmon

• Probabilities P(ω1) and P(ω2) (priors):

– e.g., prior knowledge of how likely is to get a sea bass
or a salmon

• Probability density function p(x) (evidence):

– e.g., how frequently we will measure a pattern with
feature value x (e.g., x corresponds to lightness)
 Terminology (cont’d)
• Conditional probability density p(x/ωj) (likelihood) :
– e.g., how frequently we will measure a pattern with
feature value x given that the pattern belongs to class ωj

e.g., lightness distributions

between salmon/sea-bass
populations
 Terminology (cont’d)

• Conditional probability P(ωj /x) (posterior) :

– e.g., the probability that the fish belongs to
class ωj given measurement x.
 Decision Rule Using Prior
Probabilities
Decide ω1 if P(ω1) > P(ω2); otherwise decide ω2

⎧ P(ω1 ) if we decide ω2
P(error ) = ⎨
⎩ P(ω2 ) if we decide ω1

or P(error) = min[P(ω1), P(ω2)]

• Favours the most likely class.

• This rule will be making the same decision all times.
– i.e., optimum if no other information is available
 Decision Rule Using Conditional
Probabilities
• Using Bayes’ rule, the posterior probability of category ωj
given measurement x is given by:

p( x / ω j ) P(ω j ) likelihood × prior

P(ω j / x) = =
p ( x) evidence
2
where
p( x) = ∑ p( x / ω j ) P(ω j ) (i.e., scale factor – sum of probs = 1)
j =1

Decide ω1 if P(ω1 /x) > P(ω2 /x); otherwise decide ω2

or
Decide ω1 if p(x/ω1)P(ω1)>p(x/ω2)P(ω2) otherwise decide ω2
 Decision Rule Using
Conditional pdf (cont’d)
2 1
p(x/ωj) P(ω1 ) = P (ω 2 ) =
3 3 P(ωj /x)
 Probability of Error
• The probability of error is defined as:

⎧ P(ω1 / x) if we decide ω2
P(error / x) = ⎨
⎩ P(ω2 / x) if we decide ω1
or P(error/x) = min[P(ω1/x), P(ω2/x)]

• What is the average probability error?

∞ ∞
P(error ) = ∫ P(error , x)dx = ∫ P(error / x) p( x)dx
−∞ −∞
• The Bayes rule is optimum, that is, it minimizes the
average probability error!
 Where do Probabilities Come From?

• There are two competitive answers to this

question:

(1) Relative frequency (objective) approach.

– Probabilities can only come from experiments.

(2) Bayesian (subjective) approach.

– Probabilities may reflect degree of belief and can be
based on opinion.
 Example (objective approach)
• Classify cars whether they are more or less than $50K:
– Classes: C1 if price > $50K, C2 if price <= $50K
– Features: x, the height of a car

• Use the Bayes’ rule to compute the posterior probabilities:

p ( x / Ci )P (C i )
P(Ci / x ) =
p( x)
• We need to estimate p(x/C1), p(x/C2), P(C1), P(C2)
 Example (cont’d)
• Collect data
– Ask drivers how much their car was and measure height.
• Determine prior probabilities P(C1), P(C2)
– e.g., 1209 samples: #C1=221 #C2=988

221
P(C1 ) = = 0.183
1209
988
P(C2 ) = = 0.817
1209
 Example (cont’d)
• Determine class conditional probabilities (likelihood)
– Discretize car height into bins and use normalized histogram

p( x / Ci )
 Example (cont’d)
• Calculate the posterior probability for each bin:
p( x = 1.0 / C1) P( C1)
P(C1 / x = 1.0) = =
p( x = 1.0 / C1) P( C1) + p( x =1.0 / C2) P( C2)
0.2081*0.183
= = 0.438
0.2081*0.183 + 0.0597 *0.817

P(Ci / x)
 A More General Theory
• Use more than one features.
• Allow more than two categories.
• Allow actions other than classifying the input to
one of the possible categories (e.g., rejection).
• Employ a more general error function (i.e., “risk”
function) by associating a “cost” (“loss” function)
with each error (i.e., wrong action).
 Terminology
• Features form a vector x ∈ R d
• A finite set of c categories ω1, ω2, …, ωc
• Bayes rule (i.e., using vector notation):

p (x / ω j ) P(ω j )
P(ω j / x) =
p( x)
c
where p(x) = ∑ p(x / ω j ) P(ω j )
j =1

• A finite set of l actions α1, α2, …, αl

• A loss function λ(αi / ωj)
– the cost associated with taking action αi when the correct
classification category is ωj
Conditional Risk (or Expected Loss)
• Suppose we observe x and take action αi

• Suppose that the cost associated with taking

action αi with ωj being the correct category is
λ(αi / ωj)

• The conditional risk (or expected loss) with

taking action αi is:
c
R(ai / x) = ∑ λ (ai / ω j ) P(ω j / x)
j =1
 Overall Risk
• Suppose α(x) is a general decision rule that
determines which action α1, α2, …, αl to take for
every x; then the overall risk is defined as:

R = ∫ R(a(x) / x) p(x)dx
• The optimum decision rule is the Bayes rule
 Overall Risk (cont’d)
• The Bayes decision rule minimizes R by:
(i) Computing R(αi /x) for every αi given an x
(ii) Choosing the action αi with the minimum R(αi /x)

• The resulting minimum overall risk is called Bayes

risk and is the best (i.e., optimum) performance
that can be achieved:
*
R =minR
 Example: Two-category
classification
• Define
– α1: decide ω1
(c=2)
– α2: decide ω2
– λij = λ(αi /ωj)

• The conditional risks are:

c
R(ai / x) = ∑ λ (ai / ω j ) P(ω j / x)
j =1
 Example: Two-category
classification (cont’d)
• Minimum risk decision rule:

or

or (i.e., using likelihood ratio)

>

likelihood ratio threshold

 Special Case:
Zero-One Loss Function
• Assign the same loss to all errors:

• The conditional risk corresponding to this loss function:

 Special Case:
Zero-One Loss Function (cont’d)
• The decision rule becomes:

or

or

• In this case, the overall risk is the average probability

error!
 Example
Assuming general loss:
>

Assuming zero-one loss:

Decide ω1 if p(x/ω1)/p(x/ω2)>P(ω2 )/P(ω1) otherwise decide ω2

θa = P(ω2 ) / P(ω1 )

P(ω2 )(λ12 − λ22 )

θb =
P(ω1 )(λ21 − λ11 )
(decision regions)

assume: λ12 > λ21

 Discriminant Functions
• A useful way to represent classifiers is through
discriminant functions gi(x), i = 1, . . . , c, where a feature
vector x is assigned to class ωi if:

gi(x) > gj(x) for all j ≠i

 Discriminants for Bayes Classifier
• Assuming a general loss function:

gi(x)=-R(αi / x)

• Assuming the zero-one loss function:

gi(x)=P(ωi / x)
 Discriminants for Bayes Classifier
(cont’d)
• Is the choice of gi unique?
– Replacing gi(x) with f(gi(x)), where f() is monotonically
increasing, does not change the classification results.

p(x / ωi ) P(ωi )
g i ( x) =
gi(x)=P(ωi/x) p ( x)
gi (x) = p(x / ωi ) P(ωi )
gi (x) = ln p(x / ωi ) + ln P(ωi )
we’ll use this
form extensively!
 Case of two categories
• More common to use a single discriminant function
(dichotomizer) instead of two:

• Examples:
g (x) = P(ω1 / x) − P(ω2 / x)
p (x / ω1 ) P(ω1 )
g (x) = ln + ln
p ( x / ω2 ) P(ω2 )
 Decision Regions and Boundaries
• Decision rules divide the feature space in decision regions
R1, R2, …, Rc, separated by decision boundaries.

decision boundary
is defined by:
g1(x)=g2(x)
 Discriminant Function for
Multivariate Gaussian Density

N(µ,Σ)

• Consider the following discriminant function:

gi (x) = ln p(x / ωi ) + ln P(ωi )

p(x/ωi)
 Multivariate Gaussian Density:
Case I
• Σi=σ2(diagonal)
– Features are statistically independent
– Each feature has the same variance

favours the a-priori

more likely category
Multivariate Gaussian Density:
Case I (cont’d)

wi=

)
)
 Multivariate Gaussian Density:
Case I (cont’d)
• Properties of decision boundary:
– It passes through x0
– It is orthogonal to the line linking the means.
– What happens when P(ωi)= P(ωj) ?
– If P(ωi)= P(ω
≠ j), then x0 shifts away from the most likely category.
– If σ is very small, the position of the boundary is insensitive to P(ωi)
and P(ωj)

)
)
Multivariate Gaussian Density:
Case I (cont’d)

If P(ωi)= P(ω
≠
j), then x0 shifts away
from the most likely category.
Multivariate Gaussian Density:
Case I (cont’d)

If P(ωi)= P(ω
≠
j), then x0 shifts away
from the most likely category.
Multivariate Gaussian Density:
Case I (cont’d)

If P(ωi)= P(ω
≠
j), then x0 shifts away
from the most likely category.
 Multivariate Gaussian Density:
Case I (cont’d)
• Minimum distance classifier
– When P(ωi) are equal, then:

g i ( x) = − || x − µi ||2

max
 Multivariate Gaussian Density:
Case II
• Σi= Σ
Multivariate Gaussian Density:
Case II (cont’d)
 Multivariate Gaussian Density:
Case II (cont’d)
• Properties of hyperplane (decision boundary):
– It passes through x0
– It is not orthogonal to the line linking the means.
– What happens when P(ωi)= P(ωj) ?
– If P(ωi)= P(ω
≠ j), then x0 shifts away from the most likely category.
Multivariate Gaussian Density:
Case II (cont’d)

If P(ωi)= P(ω
≠
j), then x0 shifts away
from the most likely category.
Multivariate Gaussian Density:
Case II (cont’d)

If P(ωi)= P(ω
≠
j), then x0 shifts away
from the most likely category.
 Multivariate Gaussian Density:
Case II (cont’d)
• Mahalanobis distance classifier
– When P(ωi) are equal, then:

max
 Multivariate Gaussian Density:
Case III
• Σi= arbitrary

hyperquadrics;

e.g., hyperplanes, pairs of hyperplanes, hyperspheres,

hyperellipsoids, hyperparaboloids etc.
 Example - Case III

decision boundary:

P(ω1)=P(ω2)

boundary does
not pass through
midpoint of μ1,μ2
Multivariate Gaussian Density:
Case III (cont’d)

non-linear
decision
boundaries
 Multivariate Gaussian Density:
Case III (cont’d)
• More examples
 Error Bounds
• Exact error calculations could be difficult – easier to
estimate error bounds!

or
min[P(ω1/x), P(ω2/x)]

P(error)
 Error Bounds (cont’d)
• If the class conditional distributions are Gaussian, then

where:

| |
 Error Bounds (cont’d)

• The Chernoff bound corresponds to β that minimizes e-κ(β)

– This is a 1-D optimization problem, regardless to the dimensionality
of the class conditional densities.
loose bound loose bound
tight bound
 Error Bounds (cont’d)
• Bhattacharyya bound
– Approximate the error bound using β=0.5
– Easier to compute than Chernoff error but looser.

• The Chernoff and Bhattacharyya bounds will not be good

bounds if the distributions are not Gaussian.
Example

Bhattacharyya error:
k(0.5)=4.06
P(error ) ≤ 0.0087
 Receiver Operating
Characteristic (ROC) Curve
• Every classifier employs some kind of a threshold.

θa = P(ω2 ) / P(ω1 )

P(ω2 )(λ12 − λ22 )

θb =
P(ω1 )(λ21 − λ11 )

• Changing the threshold affects the performance of the

system.
• ROC curves can help us evaluate system performance for
different thresholds.
 Example: Person Authentication
• Authenticate a person using biometrics (e.g., fingerprints).
• There are two possible distributions (i.e., classes):
– Authentic (A) and Impostor (I)

A
I
 Example: Person Authentication
(cont’d)
• Possible decisions:
– (1) correct acceptance (true positive):
• X belongs to A, and we decide A correct rejection
correct acceptance
– (2) incorrect acceptance (false positive):
• X belongs to I, and we decide A
– (3) correct rejection (true negative): A
• X belongs to I, and we decide I I
– (4) incorrect rejection (false negative):
• X belongs to A, and we decide I false negative false positive
Error vs Threshold
ROC
False Negatives vs Positives
 Next Lecture
• Linear Classification Methods
– Hastie et al, Chapter 4

• Paper list will available by Weekend

– Bidding to start on Monday
 Bayes Decision Theory:
Case of Discrete Features

• Replace with ∑ P(x / ω j )

∫ p(x / ω j )dx x

• See section 2.9

 Missing Features
• Consider a Bayes classifier using uncorrupted data.
• Suppose x=(x1,x2) is a test vector where x1 is missing and the
value of x2 is -
x̂2 how can we classify it?
– If we set x1 equal to the average value, we will classify x as ω3
p( xˆ2 / ω2 )
– But is larger; maybe we should classify x as ω2 ?
 Missing Features (cont’d)
• Suppose x=[xg,xb] (xg: good features, xb: bad features)
• Derive the Bayes rule using the good features:
Marginalize
p p posterior
probability
over bad
features.
 Compound Bayesian
Decision Theory
• Sequential decision
(1) Decide as each fish emerges.

• Compound decision
(1) Wait for n fish to emerge.
(2) Make all n decisions jointly.
– Could improve performance when consecutive states
of nature are not be statistically independent.
 Compound Bayesian
Decision Theory (cont’d)
• Suppose Ω=(ω(1), ω(2), …, ω(n))denotes the
n states of nature where ω(i) can take one of
c values ω1, ω2, …, ωc (i.e., c categories)
• Suppose P(Ω) is the prior probability of the n
states of nature.
• Suppose X=(x1, x2, …, xn) are n observed
vectors.
 Compound Bayesian
Decision Theory (cont’d)

P P
acceptable! i.e., consecutive states of nature may
not be statistically independent!