0% found this document useful (0 votes)
134 views58 pages

Irs Unit 4 CH 1

The document discusses text classification using the Naive Bayes algorithm. It introduces text classification and Bayes' rule, then describes the Naive Bayes approach. The Naive Bayes classifier makes a key assumption that features are conditionally independent given the class. This allows it to estimate probabilities efficiently from training data. The document provides examples of tasks that use text classification like spam filtering, document categorization, and standing queries for personalized information retrieval.

Uploaded by

Zeenath
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
134 views58 pages

Irs Unit 4 CH 1

The document discusses text classification using the Naive Bayes algorithm. It introduces text classification and Bayes' rule, then describes the Naive Bayes approach. The Naive Bayes classifier makes a key assumption that features are conditionally independent given the class. This allows it to estimate probabilities efficiently from training data. The document provides examples of tasks that use text classification like spam filtering, document categorization, and standing queries for personalized information retrieval.

Uploaded by

Zeenath
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
You are on page 1/ 58

Part 9: Text Classification;

The Naïve Bayes algorithm


Francesco Ricci

Most of these slides comes from the


course:
Information Retrieval and Web Search,
Christopher Manning and Prabhakar
Raghavan
1
Content

  Introduction to Text Classification


  Bayes rule
  Naïve Bayes text classification  
Feature independence assumption  
Multivariate and Multinomial approaches
  Smoothing (avoid overfitting)  
Feature selection   Chi square and
Mutual Information   Evaluating NB
classification.

2
Standing queries

  The path from information retrieval to


text classification:   You have an
information need, say:   "Unrest in the
Niger delta region"
  You want to rerun an appropriate query
periodically to find new news items on this
topic
  Youwill be sent new documents that are
found   I.e., it’s classification not ranking
  Such queries are called standing queries  

Long used by “information professionals”   A

modern mass instantiation is Google Alerts.

4
Google alerts

5
Spam filtering: Another text
classification task
From: "" <[email protected]>
Subject: real estate is the only way... gem oalvgkay

Anyone can buy real estate with no money down

Stop paying rent TODAY !

There is no need to spend hundreds or even thousands for similar courses

I am 22 years old and I have already purchased 6 properties using the methods
outlined in this truly INCREDIBLE ebook.

Change your life NOW !

=================================================
Click Below to order:
http://www.wholesaledaily.com/sales/nmd.htm
=================================================
6
Categorization/Classification
  Given:

  A description of an instance, x ∈ X, where X


is the instance language or instance space  
Issue: how to represent text documents – the
representation determines what information is
used for solving the classification task
  A fixed set of classes:
C = {c1, c2,…, cJ}
  Determine:

  Theclass of x: c(x)∈C, where c(x) is a


classification function whose domain is X
and whose range is C   We want to know how
to build classification functions (“classifiers”).
7
Document Classification
“planning
Test language
Data: proof
intelligence ”

(AI ) (Programming ) (HCI )


Classes:
ML Planning Semantics Garb.Coll. Multimedia GUI

Training learning planning programming garbage ... ...


Data: intelligence temporal semantics collection algorithm
reasoninglanguage memory reinforcement plan proof...
optimization network... language... region...

(Note: in real life there is often a hierarchy, not


present in the above problem statement; and also,
you get papers on "ML approaches to Garb. Coll.")

8
More Text Classification Examples

  Many search engine functionalities use classification


  Assign labels to each document or web-page:
  Labels are most often topics such as Yahoo-categories
e.g., "finance," "sports," "news>world>asia>business"
  Labels may be genres
e.g., "editorials" "movie-reviews" "news“
  Labels may be opinion on a person/product

e.g., “like”, “hate”, “neutral”


  Labels may be domain-specific

e.g., "interesting-to-me" : "not-interesting-to-me”


e.g., “contains adult language” : “doesn’t”
e.g., language identification: English, French, Chinese, …
e.g., “link spam” : “not link spam”
e.g., "key-phrase" : "not key-phrase"

9
Classification Methods (1)

  Manual classification   Used by Yahoo!


(originally; now present but
downplayed), Looksmart, about.com, ODP,
PubMed   Very accurate when job is done
by experts   Consistent when the problem
size and team
is small
  Difficult
and expensive to scale  
Means we need automatic classification
methods for big problems.

10
Classification Methods (2)   Hand-coded
rule-based systems   One technique used by CS
dept’s spam filter, Reuters, CIA, etc.   Companies
(Verity) provide “IDE” for writing such rules
  Example:assign category if document contains a
given Boolean combination of words
  Standing queries: Commercial systems have
complex query languages (everything in IR query
languages + accumulators)
  Accuracyis often very high if a rule has been
carefully refined over time by a subject expert  
Building and maintaining these rules is expensive!

11
Verity topic (a classification rule)
  Note:   maintenance
issues
(author, etc.)   Hand-

weighting of
terms
  Butit is easy to explain
the results.
Classification
Methods (3)
  Supervised learning
of a document-label
assignment function
12
  Manysystems partly rely on machine learning
(Autonomy, MSN, Verity, Enkata, Yahoo!, …)   k-
Nearest Neighbors (simple, powerful)   Naive

Bayes (simple, common method)   Support-

vector machines (new, more powerful)   … plus


many other methods
  No
free lunch: requires hand-classified training
data
  Notethat many commercial systems use a mixture
of methods.
Recall a few probability basics

  For events a and b:

13
  Bayes’ Rule

p(a,b) =p(a∩b) =p(a | b)p(b) =p(b | a)p(a)


Prio
p(b | a)p(a) p(b | a)p(a)
r
p(a | b) = =
p(b) ∑x=a,a p(b | x)p(x)
Posterior

  Odds:

p(a) p(a)
14
O(a)= =
p(a) 1−p(a)
Bayes’ Rule Example

P(C,E) =P(C | E)P(E) =P(E |C)P(C)


P(E |C)P(C)
P(C | E) =
P(E)
P(pass exam | attend classes) = ?

= P(pass exam) * P(attend classes | pass exam)/P(attend classes)


15
= 0.7 *
Initial estimation

Correction based on a ratio


0.9/0.78
= 0.7 * 1.15 = 0.81

16
Example explained
Pass 70% Not pass 30%

Attend 50%

Attend 90% = P(attend|pass)

Not attend 50%

Not attend 10%

P(pass) = 0.7
P(attend) = P(attend| pass)P(pass) + P(attend | not pass)P(not pass)
= 0.9*0.7 + 0.5*0.3 = 0.63 + 0.15 = 0.78 p(pass
| attend) = p(pass)*p(attend | pass)/p(attend)
17
= 0.7 * 0.9/0.78 = 0.81

Bayesian Methods

  Our focus this lecture


  Learning and classification methods based on
probability theory
  Bayes theorem plays a critical role in
probabilistic learning and classification
  Uses prior probability of each category given
no information about an item
  Obtainsa posterior probability distribution over
the possible categories given a description of an
item.

18
Naive Bayes Classifiers

  Task: Classify a new instance D based on a tuple


of attribute values into one of
D=x1,x2,,xn
the classes cj ∈ C

cMAP = argmaxP(c j | x1,x2,,xn )


c ∈C
j

P(x1, x2,, xn | cj )P(cj )


Maximum A
Posteriori = argmax c j∈ C
class P(x1, x2,, xn )
19
= argmaxP(x1, x2,, xn | cj )P(cj )
cj∈C

20
Naïve Bayes Assumption
  P(cj)

  Can be estimated from the frequency of classes in


the training examples
  P(x1,x2,…,xn|cj)   O(|X|n|C|) parameters (assuming X
finite)   Could only be estimated if a very, very large
number of training examples was available – or?
  Naïve Bayes Conditional Independence Assumption:
  Assume that the probability of observing the
conjunction of attributes is equal to the product of
the individual probabilities P(xi|cj).
n

cMAP = argmaxP(cj )∏P(xi | cj ) 19 cj∈C i=1


The Naïve Bayes Classifier
Flu

X1 X2 X3 X4 X5
runnynose sinus cough fever muscle-ache  

Conditional Independence Assumption:


features detect term presence and are
independent of each other given the class:

P(x1,, x5 |C) =P(x1 |C)•P(x2 |C)••P(x5 |C)   This


model is appropriate for binary
variables
= many
  Multivariate Bernoulli model variables

22
=only 2 values – T or F
Learning the Model
C

X1 X2 X3 X4 X5 X6

  First attempt: maximum likelihood estimates


  simply use the frequencies in the data

Pˆ(xi | c j ) =
Pˆ(cj )= N(C=cj )
N
N(
23
Estimated conditional
Xi =xi ,C=c j ) probability that the
attribute Xi (e.g. Fever)
N(C=c j ) has the value xi (True or
False) – we will also write
P(Fever | cj) instead of
P(Fever = T | cj)
Problem with Max Likelihood
Flu

1 2 3 4 5
X X X X X
runnynose sinus cough fever muscle-ache

P(x1,, x5 |C) =P(x1 |C)•P(x2 |C)••P(x5 |C)


  What
if we have seen no training cases where patient had flu and
muscle aches?
24
Pˆ(X5 =T |C = flu) = N(X5 =T,C = flu) = 0
N(C = flu)
  Zeroprobabilities cannot be conditioned away, no matter the
other evidence!

ℓ= argmaxc Pˆ(c)∏i Pˆ(xi | c)

25
Smoothing to Avoid Overfitting

Pˆ(xi | c j ) = N(Xi
=xi ,C=c j ) +1 N(C=c j )
+k
# of values of X i

  Somewhat more subtle version


overall fraction in
data where Xi=x i

26
P ˆ (xi |c j ) =

N(Xi = xi,C = c j )+ mP(Xi

= xi) N(C = c j )+ m
extent of
“smoothing”
Example
docID words in document in c = China? training set 1
Chinese Beijing Chinese yes 2 Chinese Chinese Shanghai yes
3 Chinese Macao yes
4 Tokyo Japan Chinese no test set 5 Chinese
Chinese Chinese Tokyo Japan ?

Pˆ(Chinese|c) = (3+1)/(3+2) = 4/5


27
Pˆ(Japan|c) = Pˆ(Tokyo|c) = (0+1)/(3+2) = 1/5

Pˆ(Beijing|c) = Pˆ(Macao|c) = Pˆ(Shanghai|c) = (1+1)/(3+2) = 2/5

Pˆ(Chinese|c) = (1+1)/(1+2) = 2/3

Pˆ(Japan|c) = Pˆ(Tokyo|c) = (1+1)/(1+2) = 2/3

Pˆ(Beijing|c) = Pˆ(Macao|c) = Pˆ(Shanghai|c) = (0+1)/(1+2) = 1/3

) ) )
Pˆ(c|d5) ∝ Pˆ(c · Pˆ(Chinese|c · Pˆ(Japan|c · Pˆ(Tokyo|c)

·(1 −Pˆ(Beijing|c)) · (1 −Pˆ(Shanghai|c)) · (1 −Pˆ(Macao|c))

Exercise
docID words in document in c = China? training set 1
Chinese Beijing Chinese yes 2 Chinese Chinese Shanghai yes
3 Chinese Macao yes
28
4 Tokyo Japan Chinese no test set 5 Chinese
Chinese Chinese Tokyo Japan ?

Pˆ(Chinese|c) = (3+1)/(3+2) = 4/5

Pˆ(Japan|c) = Pˆ(Tokyo|c) = (0+1)/(3+2) = 1/5

Pˆ(Beijing|c) = Pˆ(Macao|c) = Pˆ(Shanghai|c) = (1+1)/(3+2) = 2/5

Pˆ(Chinese|c) = (1+1)/(1+2) = 2/3

Pˆ(Japan|c) = Pˆ(Tokyo|c) = (1+1)/(1+2) = 2/3

Pˆ(Beijing|c) = Pˆ(Macao|c) = Pˆ(Shanghai|c) = (0+1)/(1+2) = 1/3

Estimate the probability that the test document does not


belong to class c

29
Exercise
docID words in document in c = China? training set 1
Chinese Beijing Chinese yes 2 Chinese Chinese Shanghai yes
3 Chinese Macao yes
4 Tokyo Japan Chinese no test set 5 Chinese
Chinese Chinese Tokyo Japan ?

Pˆ(Chinese|c) = (3+1)/(3+2) = 4/5

Pˆ(Japan|c) = Pˆ(Tokyo|c) = (0+1)/(3+2) = 1/5

Pˆ(Beijing|c) = Pˆ(Macao|c) =
Pˆ(Shanghai

ˆ
P (Chinese|

Pˆ(Japan|c) = Pˆ(Tokyo|c) = (1+1)/(1+2) = 2/3

Pˆ(Beijing|c) = Pˆ(Macao|c) = Pˆ(Shanghai|c) = (0+1)/(1+2) = 1/3

30

Multinomial Naive Bayes Classifiers: Basic
method

  Attributes (Xi) are text positions, values (xi) are


words:

cNB = argmaxP (xi |c j )


cj ∈C

= argmaxP(c j )P(X1 ="Our"|c j )P(Xn ="text."|c j )


 cj ∈C

  Still too many possibilities

31
  Assume that classification is independent of the
positions of the words

P(Xi =w | c) =P(X j =w | c)
Multinomial Naïve Bayes: Learning

  From training corpus, extract Vocabulary   Calculate

required P(cj) and P(xk | cj) terms   For each class cj in
C do   docsj ← subset of documents for which the target
class is cj
|docsj |
P(cj )←
total # documents

32
  Textj ← single document containing all
docsj   for each word xk in Vocabulary  

njk ← number of occurrences of xk in Textj


  nj ← number of words in Textj
Assume is = 1;
n jk +α
P(xk | cj ) ← nj +α|Vocabulary | this is for
smoothing
Multinomial Naïve Bayes:
Classifying

  positions ← all word positions in current


document which contain tokens found
in Vocabulary

  Return cNB such that:


33
cNB = argmax P(cj ) ∏P(xi | cj ) cj∈C
i∈positions

Example
docID words in document in c = China? training set 1
Chinese Beijing Chinese yes 2 Chinese Chinese Shanghai yes
3 Chinese Macao yes
4 Tokyo Japan Chinese no test set 5 Chinese
Chinese Chinese Tokyo Japan ?

Pˆ(Chinese|c) = (5+1)/(8+6) = 6/14 = 3/7

Pˆ(Tokyo|c) = Pˆ(Japan|c) = (0+1)/(8+6) = 1/14

Pˆ(Chinese|c) = (1+1)/(3+6) = 2/9

34
Pˆ(Tokyo|c) = Pˆ(Japan|c) = (1+1)/(3+6) = 2/9

Pˆ(c|d5) ∝ 3/4 · (3/7)3 · 1/14 · 1/14 ≈ 0.0003.

)
Pˆ(c|d5) ∝ 1/4 · (2/9 3 · 2/9 · 2/9 ≈ 0.0001.

Naive Bayes: Time Complexity


Number of
  Training
Time: if Ld is the average length of a conditional
document in D probabilities to
estimate
  O(|D|Ld + |C||V|))

Scan the documents to compute the


vocabulary and the frequencies of words

  Assumes that V and all docsj , nj, and njk are
35
computed in O(|D|Ld) time during one pass through
all of the data
  Generally just O(|D|Ld) since usually |C||V| < |D|Ld
  TestTime: O(|C| Lt) - where Lt is the average length of
a test document
  Very efficient overall, linearly proportional to the time
needed to just read in all the data.
Underflow Prevention: log space

  Multiplying
lots of probabilities, which are between 0
and 1 by definition, can result in floating-point
underflow
  Since
log(xy) = log(x) + log(y), it is better to
perform all computations by summing logs of
probabilities rather than multiplying probabilities

36
  Classwith highest final un-normalized log probability
score is still the most probable

cNB = argmax( logP(cj) + ∑logP(x | c ) )


i j

cj∈C
i∈positions

  Note that model is now just max of sum of weights…

Sounds
familiar?
Summary - Two Models: Multivariate Bernoulli

  One feature Xw for each word in


dictionary   Xw = true in document d if w
37
appears in d   Naive Bayes
assumption:   Given the document’s
topic (class),
appearance of one word in the document tells
us nothing about chances that another word
appears (independence)
Summary - Two Models: Multinomial

  One feature Xi for each word positions in document


  feature’s values are all words in dictionary
  Value of Xi is the word in position i   Naïve
Bayes assumption:   Given the document’s
topic (class), word in one
position in the document tells us nothing about
words in other positions
38
  Second assumption:   Word appearance does
not depend on position - for
all positions i,j, word w, and class c

P(Xi =w | c) =P(X j =w | c)
  Just
have one multinomial feature predicting all
words.
Parameter estimation

  Multivariate Bernoulli model:

39
fraction of documents of
ˆ
P (Xw = true |c j ) topic cj in which word w
appears
=
  Multinomial model:

fraction of times in which


Pˆ(Xi =w| cj ) = word w appears
across all documents of topic cj

  Can
create a mega-
document for topic j by concatenating all
documents in this topic   Use frequency of w

40
Classification

  Multinomial vs Multivariate Bernoulli?

  Multinomial model is almost always more


effective in text applications!
  See results figures later

  See
IIR sections 13.2 and 13.3 for worked
examples with each model
Feature Selection: Why?

41
  Text collections have a large number of features
  10,000 – 1,000,000 unique words … and
more
  May make using a particular classifier unfeasible
  Some classifiers can’t deal with 100,000 of
features
  Reduces training time   Training time for
some methods is quadratic or
worse in the number of features
  Can improve generalization
(performance)   Eliminates noise
features   Avoids overfitting.

42
Feature selection: how?

  Twoideas:
  Hypothesis testing statistics:   Are
we confident that the value of one
categorical variable is associated with the
value of another
  Chi-square test (
http://faculty.vassar.edu/lowry/
webtext.html chapter 8)
  Information theory:   How much
information does the value of one categorical
variable give you about the value of another
  Mutual information

43
χ2 statistic (CHI) – testing
independence of class and term
Term = jaguar Term ≠ jaguar

Class = auto 2 500

Class ≠ auto 3 9500

  Ifthe term "jaguar" is independent from the class "auto"


we should have:   P(C(d)=auto, d contains jaguar) =
P(C(d)= auto) * P(d contains jaguar)
  2/10005=? 502/10005 * 5/10005   0.00019 =? 0.0501 *

0.00049 = 0.000025 NOT REALLY!   To be independent


we should have more documents that contains jaguar but are
not in class auto (38).
44
χ2 statistic (CHI)
  χ2is interested in (fo – fe)2/fe summed over all table entries: is the
observed number what you’d expect given the marginals?

2 2 2 2

χ (j,a) =∑(O − E) /E = (2 −.25) /.25+(3− 4.75) /4.75


+(500 −502)2 /502+(9500 −9498)2 /9498 =12.9 (p < .001)
  The null hypothesis (the two variables are independent) is
rejected with confidence .999,
  since 12.9 > 10.83 (the critical value for .999 confidence for a 1
degree of freedom χ2 distribution ).

Term = jaguar Term ≠ jaguar


expected: fe

Class = auto 2 (0.25) 500 (502)

observed: fo
Class ≠ auto 3 (4.75) 9500 (9498)

45
Expected value, for instance in the up-left cell is:
P(c=auto)*P(T=jaguar)* #of cases = 502/10005 * 5/10005 *
10005 = 0.2508
χ2 statistic (CHI)

There is a “simpler” formula for 2x2 χ2:

A = #(t,c) C = #(¬t,c)

B = #(t,¬c) D = #(¬t, ¬c)

N=A+B+C+D
46
Feature selection via Mutual
Information

  Intraining set, choose k words which give most info


on the knowledge of the categories   The Mutual
Information between a word, class is:

p(U =ew,C=ec)
I(U,C)= ∑ ∑ p(U =ew,C=ec)log e ∈{0,1} e ∈{0,1} p(U
w c

=ew)p(C=ec)
  U=1 (U=0) means the document (does not) contains
w   C=1 (C=0) the document is (not) in class c   For

each word w and each category c   I(X,Y) = H(X) –


H(X|Y)
47
=-Σi p(xi)logp(xi) + Σi p(yj) H(X|Y=yj)
  H is called the Entropy

Feature selection via MI (contd.)

  Foreach category we build a list of k most


discriminating terms
  For example (on 20 Newsgroups):
  sci.electronics: circuit, voltage, amp, ground,
copy, battery, electronics, cooling, …
  rec.autos: car, cars, engine, ford, dealer,
mustang, oil, collision, autos, tires, toyota, …
  Greedy: does not account for
correlations between terms   Why?
48
Feature Selection

  Mutual Information   Clear information-


theoretic interpretation   May select very
slightly informative frequent
terms that are not very useful for classification
  Chi-square  
Statistical foundation
  May select rare statistically correlated but
uninformative terms
  Just use the commonest terms?
  No particular foundation   In
practice, this is often 90% as good.

49
Example

50
Feature selection for NB

  In
general feature selection is necessary for
multivariate Bernoulli NB
  Otherwise you suffer from noise, multi-counting

  “Feature selection” really means something

different for multinomial NB - it means


dictionary truncation
  The multinomial NB model only has 1 feature
  This
“feature selection” normally isn’t needed
for multinomial NB, but may help a fraction with
quantities that are badly estimated.
51
Evaluating Categorization

  Evaluation must be done on test data that are


independent of the training data (usually a disjoint
set of instances).
  Classification
accuracy: c/n where n is the total
number of test instances and c is the number of test
instances correctly classified by the system
  Adequateif one class per document (and positive
and negative examples have similar cardinalities)
  Otherwise F measure for each class
  Resultscan vary based on sampling error due to
different training and test sets
  Average results over multiple training and test sets
(splits of the overall data) for the best results.

52
WebKB Experiment (1998)

  Classify webpages from CS departments into:


  student, faculty, course, project
  Train on ~5,000 hand-labeled web pages

53
  Results:

Student Faculty Person Project Course Departmt
Extracted 180 66 246 99 28 1
Correct 130 28 194 72 25 1
Accuracy: 72% 42% 79% 73% 89% 100%

Actually this is not accuracy but …

  Cornell, Washington, U.Texas, Wisconsin


  Crawl and classify a new site (CMU)

54
Most relevant features: MI

55
Naïve Bayes on spam email

56
Naïve Bayes Posterior Probabilities

  Classification results of naïve Bayes (the


class
with maximum posterior probability) are
usually fairly accurate
  However, due to the inadequacy of the
conditional independence assumption, the
actual posterior-probability numerical
estimates are not
  Output probabilities are commonly very close
to 0 or 1
  Correct estimation ⇒ accurate prediction, but

57
correct probability estimation is NOT necessary
for accurate prediction (just need right ordering
of probabilities).
Naive Bayes is Not So Naive
  Naïve
Bayes: First and Second place in KDD-CUP 97 competition,
among 16 (then) state of the art algorithms
Goal: Financial services industry direct mail response prediction model:
Predict if the recipient of mail will actually respond to the advertisement
– 750,000 records.
  Robust to Irrelevant Features
Irrelevant Features cancel each other without affecting results
Instead Decision Trees can heavily suffer from this.
  Very good in domains with many equally important features
Decision Trees suffer from fragmentation in such cases – especially if
little data
  Agood dependable baseline for text classification (but not the
best)!
  Optimal if the Independence Assumptions hold: If assumed

58
independence is correct, then it is the Bayes Optimal Classifier for
problem
  Very Fast: Learning with one pass of counting over the data; testing
linear in the number of attributes, and document collection size
  Low Storage requirements
Resources

  IIR 13
  Tom Mitchell, Machine Learning. McGraw-
Hill, 1997.   Clear simple explanation of
Naïve Bayes

59

You might also like