Text

Classification
with Naïve Bayes

The Task of Text

Classification
 Text Classification: definition
• Input:
• a document d
• a fixed set of classes C = {c1, c2,…, cJ}

• Output: a predicted class c Î C

 Naïve Bayes Intuition
• Simple (“naïve”) classification method based on
Bayes rule
• Relies on very simple representation of document
• Bag of words
 The Bag of Words Representation
it
it it 6 6
I I I 5 5
I love
I loveI love
this movie!
this this movie!
movie! It'sIt's It's
sweet, sweet,
sweet, thethe the 4 4
fairy always love it it it
fairy
butbut
but with with
satirical
with satirical
satiricalhumor. humor.
humor. The The
The fairy always lovelove
always toto to to to 3 3
dialogue itititwhimsical
whimsical to
it and and 3 3
dialogue is great
dialogue is is great
andand
great and
the thethe and whimsical
seen
it it I I I
are and
adventure scenes are fun... and
and seen are areanyone seen 2
adventure
adventure scenes scenes are are fun...
fun... friend seen anyone
anyone seen seen2
friend dialogue
It manages
It manages to to
be be whimsical
whimsical friend happydialogue
happy dialogue yetyet yet 1 1
It manages to be whimsical happy recommend
recommend would
andand romantic
romantic whilewhile laughing
laughing adventure
adventure recommend would 1 1
and romantic while laughing adventure sweet of satirical
satirical would
whimsical1 1
at at
thethe conventions
conventions of of thethe who sweet
who sweet of it it
satirical
movie whimsical
at thefairy
conventions
fairy tale genre. of Ithewould who I toof
movie timeswhimsical
1 1
tale genre. I would it it I but
I
but
to yet movieI I it
romantic
to romantic times
sweet times1 1
fairy recommend
tale genre. itI to
recommend would
it to
justjust about
about several
several it again but yet romantic humor sweet
I
recommendanyone.
anyone. I've
it to
I've seen
just
seen it several
itabout
several several
the again it thethehumor
ityet satirical
satiricalsweet1 1
the again seen humor
would adventure
anyone. times,
times,I'veand and I'mI'm
seen it always
alwaysseveral happy
happy to to seen
scenes I it the
would
the manages adventure 1 1
satirical
thescenes I genre
times,
to
to and
seesee itI'm it again
againalways
whenever
whenever happy I I fun thethe the
seen times
would
manages genre 1 1
adventure
have a friend who hasn't fun I to Iscenes I
and timesthe and fairy 1 1
to seehave
it a friend
again who
whenever hasn't I and theaboutandmanages
about while
fairy genre
seen it yet! fun whenever times while humor 1 1
humor
seen it yet! wheneverIconventions have and
have a friend who hasn't and about have havefairy 1 1
have
conventions
with while
seen it yet! whenever
with greathumor 1 1
have great
4
conventions …… have……
with great
 Multinomial Naïve Bayes Independence
Assumptions
P(x1, x2 ,…, xn | c)

• Bag of Words assumption: Assume position doesn’t

matter
• Conditional Independence: Assume the feature
probabilities P(xi|cj) are independent given the class c.
P(x1,…, xn | c) = P(x1 | c)• P(x2 | c)• P(x3 | c)•...• P(xn | c)
 Multinomial Naïve Bayes Classifier

cMAP = argmax P(x1, x2 ,…, xn | c)P(c)

c∈C

cNB = argmax P(c j )∏ P(x | c)

c∈C x∈X
Text Classification
and Naïve Bayes

Parameter
Estimation and
Smoothing
 Sec.13.3

Learning the Multinomial Naïve Bayes Model

• First attempt: maximum likelihood estimates

• simply use the frequencies in the data
doccount(C = c j )
P̂(c j ) =
N doc
count(wi , c j )
P̂(wi | c j ) =
∑ count(w, c j )
w∈V
 Parameter estimation

P̂(wi | c j ) =
count(wi , c j ) fraction of times word wi appears
∑ count(w, c j ) among all words in documents of topic cj
w∈V

• Create mega-document for topic j by concatenating all docs in

this topic
• Use frequency of w in mega-document
 Sec.13.3

Problem with Maximum Likelihood

• What if we have seen no training documents with the word
fantastic and classified in the topic positive (thumbs-up)?
count("fantastic", positive)
P̂("fantastic" positive) = = 0
∑ count(w, positive)
w∈V

• Zero probabilities cannot be conditioned away, no matter

the other evidence!

cMAP = argmax c P̂(c)∏ P̂(xi | c)

i
Laplace (add-1) smoothing for Naïve Bayes

count(wi , c) +1
P̂(wi | c) =
∑ (count(w, c))+1)
w∈V

count(wi , c) +1
=
# &
%% ∑ count(w, c)(( + V
$ w∈V '
 Multinomial Naïve Bayes: Learning

• From training corpus, extract Vocabulary

• Calculate P(cj) terms • Calculate P(wk | cj) terms
• For each cj in C do • Textj ¬ single doc containing all docsj
docsj ¬ all docs with class =cj • For each word wk in Vocabulary
nk ¬ # of occurrences of wk in Textj
| docs j |
P(c j ) ← nk + α
| total # documents| P(wk | c j ) ←
n + α | Vocabulary |
Text Classification
and Naïve Bayes

Precision, Recall, and

the F measure
The 2-by-2 contingency table

correct not correct

selected tp fp
not selected fn tn
 Precision and recall
• Precision: % of selected items that are correct
Recall: % of correct items that are selected

correct not correct

selected tp fp
not selected fn tn
 A combined measure: F
• A combined measure that assesses the P/R tradeoff is F measure
(weighted harmonic mean):
1 ( β 2 + 1) PR
F= =
1 1 β 2
P+R
α + (1 − α )
P R
• The harmonic mean is a very conservative average
• People usually use balanced F1 measure
• i.e., with b = 1 (that is, a = ½): F = 2PR/(P+R)
Text Classification
and Naïve Bayes

Text Classification:
Evaluation
 Cross-Validation
Iteration

• Break up data into 10 folds 1 Test Training

• (Equal positive and negative

inside each fold?) 2 Test Training

• For each fold

• Choose the fold as a 3 Training Test Training
temporary test set
• Train on 9 folds, compute 4 Training Test
performance on the test fold
• Report average
5 Training Test
performance of the 10 runs
 Development Test Sets and Cross-validation

Training set Development Test Set Test Set

• Metric: P/R/F1 or Accuracy

• Unseen test set Training Set Dev Test
• avoid overfitting (‘tuning to the test set’)
• more conservative estimate of performance Training Set Dev Test
• Cross-validation over multiple splits
• Handle sampling errors from different datasets Dev Test Training Set
• Pool results over each split
• Compute pooled dev set performance Test Set
Text Classification
and Naïve Bayes

Text Classification:
Practical Issues
 Sec. 15.3.1

The Real World

• Gee, I’m building a text classifier for real, now!
• What should I do?

21
 Sec. 15.3.1
No training data?
Manually written rules
If (wheat or grain) and not (whole or bread) then
Categorize as grain

• Need careful crafting

• Human tuning on development data
• Time-consuming: 2 days per class

22
 Sec. 15.3.1

Very little data?

• Use Naïve Bayes
• Naïve Bayes is a “high-bias” algorithm (Ng and Jordan 2002 NIPS)
• Get more labeled data
• Find clever ways to get humans to label data for you
• Try semi-supervised training methods:
• Bootstrapping, EM over unlabeled documents, …

23
 Sec. 15.3.1

A reasonable amount of data?

• Perfect for all the clever classifiers
• SVM
• Regularized Logistic Regression
• You can even use user-interpretable decision trees
• Users like to hack
• Management likes quick fixes

24
 Sec. 15.3.1

Accuracy as a function of data size

• With enough data

• Classifier may not matter

25
Brill and Banko on spelling correction
 Basic Text
Processing

Regular Expressions
 Regular expressions
• A formal language for specifying text strings
• How can we search for any of these?
• woodchuck
• woodchucks
• Woodchuck
• Woodchucks
 Regular Expressions: Disjunctions
• Letters inside square brackets []
Pattern Matches
[wW]oodchuck Woodchuck, woodchuck
[1234567890] Any digit

• Ranges [A-Z]
Pattern Matches
[A-Z] An upper case letter Drenched Blossoms
[a-z] A lower case letter my beans were impatient
[0-9] A single digit Chapter 1: Down the Rabbit Hole
 Regular Expressions: Negation in Disjunction
• Negations [^Ss]
• Carat means negation only when first in []

Pattern Matches
[^A-Z] Not an upper case letter Oyfn pripetchik
[^Ss] Neither ‘S’ nor ‘s’ I have no exquisite reason”
[^e^] Neither e nor ^ Look here
a^b The pattern a carat b Look up a^b now
 Regular Expressions: More Disjunction
• Woodchucks is another name for groundhog!
• The pipe | for disjunction
Pattern Matches
groundhog|woodchuck
yours|mine yours
mine
a|b|c = [abc]
[gG]roundhog|[Ww]oodchuck
 Regular Expressions: ? * + .

Pattern Matches
colou?r Optional color colour
previous char
oo*h! 0 or more of oh! ooh! oooh! ooooh!
previous char
o+h! 1 or more of oh! ooh! oooh! ooooh!
previous char
Stephen C Kleene
baa+ baa baaa baaaa baaaaa
beg.n begin begun begun beg3n Kleene *, Kleene +
Regular Expressions: Anchors ^ $

Pattern Matches
^[A-Z] Palo Alto
^[^A-Za-z] 1 “Hello”
\.$ The end.
.$ The end? The end!
 Example
• Find me all instances of the word “the” in a text.
the
Misses capitalized examples
[tT]he
Incorrectly returns other or theology
[^a-zA-Z][tT]he[^a-zA-Z]
 Errors
• The process we just went through was based on fixing
two kinds of errors
• Matching strings that we should not have matched (there,
then, other)
• False positives (Type I)
• Not matching things that we should have matched (The)
• False negatives (Type II)
 Errors cont.
• In NLP we are always dealing with these kinds of
errors.
• Reducing the error rate for an application often
involves two antagonistic efforts:
• Increasing accuracy or precision (minimizing false positives)
• Increasing coverage or recall (minimizing false negatives).
 Summary
• Regular expressions play a surprisingly large role
• Sophisticated sequences of regular expressions are often the first model
for any text processing text
• For many hard tasks, we use machine learning classifiers
• But regular expressions are used as features in the classifiers
• Can be very useful in capturing generalizations

36
 Basic Text
Processing

Regular Expressions
 Basic Text
Processing

Word tokenization
 Text Normalization
• Every NLP task needs to do text
normalization:
1. Segmenting/tokenizing words in running text
2. Normalizing word formats
3. Segmenting sentences in running text
 How many words?
• I do uh main- mainly business data processing
• Fragments, filled pauses
• Seuss’s cat in the hat is different from other cats!
• Lemma: same stem, part of speech, rough word sense
• cat and cats = same lemma
• Wordform: the full inflected surface form
• cat and cats = different wordforms
 How many words?
they lay back on the San Francisco grass and looked at the stars and their

• Type: an element of the vocabulary.

• Token: an instance of that type in running text.
• How many?
• 15 tokens (or 14)
• 13 types (or 12) (or 11?)
 How many words?
N = number of tokens
Church and Gale (1990): |V| > O(N½)
V = vocabulary = set of types
|V| is the size of the vocabulary

Tokens = N Types = |V|

Switchboard phone 2.4 million 20 thousand
conversations
Shakespeare 884,000 31 thousand
Google N-grams 1 trillion 13 million
 Simple Tokenization in UNIX
• (Inspired by Ken Church’s UNIX for Poets.)
• Given a text file, output the word tokens and their frequencies
tr -sc ’A-Za-z’ ’\n’ < shakes.txt Change all non-alpha to newlines
| sort Sort in alphabetical order
| uniq –c Merge and count each type

1945 A 25 Aaron
72 AARON 6 Abate
1 Abates
19 ABBESS
5 Abbess
5 ABBOT 6 Abbey
... ... 3 Abbot
.... …
 The first step: tokenizing
tr -sc ’A-Za-z’ ’\n’ < shakes.txt | head

THE
SONNETS
by
William
Shakespeare
From
fairest
creatures
We
...
 The second step: sorting
tr -sc ’A-Za-z’ ’\n’ < shakes.txt | sort | head

A
A
A
A
A
A
A
A
A
...
 More counting
• Merging upper and lower case
tr ‘A-Z’ ‘a-z’ < shakes.txt | tr –sc ‘A-Za-z’ ‘\n’ | sort | uniq –c

• Sorting the counts

tr ‘A-Z’ ‘a-z’ < shakes.txt | tr –sc ‘A-Za-z’ ‘\n’ | sort | uniq –c | sort –n –r
23243 the
22225 i
18618 and
16339 to
15687 of
12780 a
12163 you
10839 my What happened here?
10005 in
8954 d
 Issues in Tokenization
• Finland’s capital ® Finland Finlands Finland’s ?
• what’re, I’m, isn’t ® What are, I am, is not
• Hewlett-Packard ® Hewlett Packard ?
• state-of-the-art ® state of the art ?
• Lowercase ® lower-case lowercase lower case ?
• San Francisco ® one token or two?
• m.p.h., PhD. ® ??
 Tokenization: language issues
• French
• L'ensemble ® one token or two?
• L ? L’ ? Le ?
• Want l’ensemble to match with un ensemble

• German noun compounds are not segmented

• Lebensversicherungsgesellschaftsangestellter
• ‘life insurance company employee’
• German information retrieval needs compound splitter
 Tokenization: language issues
• Chinese and Japanese no spaces between words:
• 莎拉波娃现在居住在美国东南部的佛罗⾥达。
• 莎拉波娃 现在 居住 在 美国 东南部 的 佛罗⾥达
• Sharapova now lives in US southeastern Florida
• Further complicated in Japanese, with multiple alphabets
intermingled
• Dates/amounts in multiple formats

フォーチュン500社は情報不足のため時間あた$500K(約6,000万円)

Katakana Hiragana Kanji Romaji

End-user can express query entirely in hiragana!
 Word Tokenization in Chinese
• Also called Word Segmentation
• Chinese words are composed of characters
• Characters are generally 1 syllable and 1 morpheme.
• Average word is 2.4 characters long.
• Standard baseline segmentation algorithm:
• Maximum Matching (also called Greedy)
 Maximum Matching
Word Segmentation Algorithm
• Given a wordlist of Chinese, and a string.
1) Start a pointer at the beginning of the string
2) Find the longest word in dictionary that matches the string
starting at pointer
3) Move the pointer over the word in string
4) Go to 2
 Max-match segmentation illustration

• Thecatinthehat the cat in the hat

• Thetabledownthere the table down there

theta bled own there

• Doesn’t generally work in English!

• But works astonishingly well in Chinese

• 莎拉波娃现在居住在美国东南部的佛罗里达。
• 莎拉波娃 现在 居住 在 美国 东南部 的 佛罗里达
• Modern probabilistic segmentation algorithms even better
 Basic Text
Processing

Word tokenization
 Basic Text
Processing

Word Normalization and

Stemming
 Normalization

• Need to “normalize” terms

• Information Retrieval: indexed text & query terms must have same form.
• We want to match U.S.A. and USA
• We implicitly define equivalence classes of terms
• e.g., deleting periods in a term
• Alternative: asymmetric expansion:
• Enter: window Search: window, windows
• Enter: windows Search: Windows, windows, window
• Enter: Windows Search: Windows

• Potentially more powerful, but less efficient

 Case folding
• Applications like IR: reduce all letters to lower case
• Since users tend to use lower case
• Possible exception: upper case in mid-sentence?
• e.g., General Motors
• Fed vs. fed
• SAIL vs. sail
• For sentiment analysis, MT, Information extraction
• Case is helpful (US versus us is important)
 Lemmatization
• Reduce inflections or variant forms to base form
• am, are, is ® be
• car, cars, car's, cars' ® car
• the boy's cars are different colors ® the boy car be different color
• Lemmatization: have to find correct dictionary headword form
• Machine translation
• Spanish quiero (‘I want’), quieres (‘you want’) same lemma as querer
‘want’
 Morphology
• Morphemes:
• The small meaningful units that make up words
• Stems: The core meaning-bearing units
• Affixes: Bits and pieces that adhere to stems
• Often with grammatical functions
 Stemming
• Reduce terms to their stems in information retrieval
• Stemming is crude chopping of affixes
• language dependent
• e.g., automate(s), automatic, automation all reduced to automat.

for example compressed for exampl compress and

and compression are both compress ar both accept
accepted as equivalent to as equival to compress
compress.
 Porter’s algorithm
The most common English stemmer
Step 1a Step 2 (for long stems)
sses ® ss caresses ® caress ational® ate relational® relate
ies ® i ponies ® poni izer® ize digitizer ® digitize
ss ® ss caress ® caress ator® ate operator ® operate
s ® ø cats ® cat …
Step 1b Step 3 (for longer stems)
(*v*)ing ® ø walking ® walk al ® ø revival ® reviv
sing ® sing able ® ø adjustable ® adjust
(*v*)ed ® ø plastered ® plaster ate ® ø activate ® activ
… …
 Viewing morphology in a corpus
Why only strip –ing if there is a vowel?

(*v*)ing ® ø walking ® walk

sing ® sing

61
 Viewing morphology in a corpus
Why only strip –ing if there is a vowel?
(*v*)ing ® ø walking ® walk
sing ® sing

tr -sc 'A-Za-z' '\n' < shakes.txt | grep ’ing$' | sort | uniq -c | sort –nr
1312 King 548 being
548 being 541 nothing
541 nothing 152 something
388 king 145 coming
375 bring 130 morning
358 thing 122 having
307 ring 120 living
152 something 117 loving
145 coming 116 Being
130 morning 102 going

tr -sc 'A-Za-z' '\n' < shakes.txt | grep '[aeiou].*ing$' | sort | uniq -c | sort –nr

62
 Dealing with complex morphology is
sometimes necessary
• Some languages requires complex morpheme segmentation
• Turkish
• Uygarlastiramadiklarimizdanmissinizcasina
• `(behaving) as if you are among those whom we could not civilize’
• Uygar `civilized’ + las `become’
+ tir `cause’ + ama `not able’
+ dik `past’ + lar ‘plural’
+ imiz ‘p1pl’ + dan ‘abl’
+ mis ‘past’ + siniz ‘2pl’ + casina ‘as if’
 Basic Text
Processing

Word Normalization and

Stemming
 Basic Text
Processing

Sentence Segmentation
and Decision Trees
 Sentence Segmentation
• !, ? are relatively unambiguous
• Period “.” is quite ambiguous
• Sentence boundary
• Abbreviations like Inc. or Dr.
• Numbers like .02% or 4.3
• Build a binary classifier
• Looks at a “.”
• Decides EndOfSentence/NotEndOfSentence
• Classifiers: hand-written rules, regular expressions, or machine-learning
Determining if a word is end-of-sentence:
a Decision Tree
 More sophisticated decision tree features
• Case of word with “.”: Upper, Lower, Cap, Number
• Case of word after “.”: Upper, Lower, Cap, Number

• Numeric features
• Length of word with “.”
• Probability(word with “.” occurs at end-of-s)
• Probability(word after “.” occurs at beginning-of-s)
 Implementing Decision Trees
• A decision tree is just an if-then-else statement
• The interesting research is choosing the features
• Setting up the structure is often too hard to do by hand
• Hand-building only possible for very simple features, domains
• For numeric features, it’s too hard to pick each threshold
• Instead, structure usually learned by machine learning from a training
corpus
 Decision Trees and other classifiers
• We can think of the questions in a decision tree
• As features that could be exploited by any kind of
classifier
• Logistic regression
• SVM
• Neural Nets
• etc.