Natural Language Processing

2024/2023
What is Natural Language Processing (NLP)
• The process of computer analysis of input provided in a human
language (natural language), and conversion of this input into a useful
form of representation.
• The field of NLP is primarily concerned with getting computers to
perform useful and interesting tasks with human languages.
• The field of NLP is secondarily concerned with helping us come to a
better understanding of human language.

2
Forms of Natural Language
• The input/output of a NLP system can be:
• written text
• speech
• We will mostly concerned with written text (not speech).
• To process written text, we need:
• lexical, syntactic, semantic knowledge about the language
• discourse information, real world knowledge
• To process spoken language, we need everything required to process
written text, plus the challenges of speech recognition and speech
synthesis.
3
4
Components of NLP
• Natural Language Understanding
• Mapping the given input in the natural language into a useful representation.
• Different level of analysis required:
morphological analysis,
syntactic analysis,
semantic analysis,
discourse analysis, …
• Natural Language Generation
• Producing output in the natural language from some internal representation.
• Different level of synthesis required:
deep planning (what to say),
syntactic generation
• NL Understanding is much harder than NL Generation.
But, still both of them are hard.

8
Why is NLP hard?
• Reason (1) – human language is ambiguous.

• Task: Pronoun Resolution

• Jack saw Sam yesterday. He went back to the restaurant to get another meal.
• Jack saw Sam yesterday. He clearly had eat too much.
Why is NLP hard?
• Reason (1) – human language is ambiguous

• Task: Preposition Attachment (aka PP-attachment)

• I ate the bread with pecans.

• I ate the bread with fingers.

Why is NLP hard?
• Reason (2) – requires reasoning beyond what is explicitly mentioned (A,B) , and
some of the reasoning requires world knowledge (C)

• I couldn’t submit my homework because my horse ate it.

• Implies that…
A. I have a horse.
B. I did my homework.
C. My homework was done on a soft object (such as papers) as opposed to a hard/heavy object
(such as a computer). – it’s more likely that my horse ate papers than a computer.
Why is NLP hard?
• Reason (3) – Language is difficult even for humans.

• Learning mother tongue (native language)

• -- you might think it’s easy, but…
• compare 5 year old V.S. 10 year old V.S. 20 year old

• Learning foreign languages

•– even harder
Ambiguity
Why is Language Ambiguous?
• Having a unique linguistic expression for every possible
conceptualization that could be conveyed would make language
overly complex and linguistic expressions unnecessarily long.
• Allowing resolvable ambiguity permits shorter linguistic expressions,
i.e. data compression.
• Language relies on people’s ability to use their knowledge and
inference abilities to properly resolve ambiguities.
• Infrequently, disambiguation fails, i.e. the compression is lossy.
Natural Languages vs. Computer Languages

• Ambiguity is the primary difference between natural and computer

languages.
• Formal programming languages are designed to be unambiguous, i.e.
they can be defined by a grammar that produces a unique parse for
each sentence in the language.
• Programming languages are also designed for efficient (deterministic)
parsing.
Natural Language Tasks
• Processing natural language text involves many various syntactic,
semantic and pragmatic tasks in addition to other problems.
Why NL Understanding is hard?
• Natural language is extremely rich in form and structure, and very
ambiguous.
• How to represent meaning,
• Which structures map to which meaning structures.
• One input can mean many different things. Ambiguity can be at different
levels.
• Lexical (word level) ambiguity -- different meanings of words
• Syntactic ambiguity -- different ways to parse the sentence
• Interpreting partial information -- how to interpret pronouns
• Contextual information -- context of the sentence may affect the meaning of that
sentence.
• Many input can mean the same thing.
• Interaction among components of the input is not clear.

17
Ambiguity
• “Get the cat with the gloves.”

18
Ambiguity
• “Get the cat with the gloves.”

19
20
Knowledge of Language
• Phonology – concerns how words are related to the sounds that realize
them.

• Morphology – concerns how words are constructed from more basic

meaning units called morphemes. A morpheme is the primitive unit of
meaning in a language.
• Syntax – concerns how can be put together to form correct sentences and
determines what structural role each word plays in the sentence and what
phrases are subparts of other phrases.
• Semantics – concerns what words mean and how these meaning combine in
sentences to form sentence meaning. The study of context-independent
meaning.
21
Knowledge of Language (cont.)
• Pragmatics – concerns how sentences are used in different situations
and how use affects the interpretation of the sentence.

• Discourse – concerns how the immediately preceding sentences

affect the interpretation of the next sentence. For example, interpreting
pronouns and interpreting the temporal aspects of the information.

• World Knowledge – includes general knowledge about the world.

What each language user must know about the other’s beliefs and
goals.

22
Phonology
•Red and Read
•Flower and Flour
•I and Eye
•Write and Right
•Knows and Nose
•Hear and Here
•Weight and Wait
•A part and Apart
•Piece and Peace
•ate and eight

23
What is Morphology?

24
Morphology for NLP

25
Morphemes
• Smallest meaning bearing units constituting a word

26
Inflection vs Derivation morphemes
• Inflection is the morphological system for making word forms of
words,
• derivation is one of the morphological systems for making new
words.
• Derivation is formally similar to inflection because both processes
make use of affixation.

27
Inflectional morphemes:
• Create new forms of the same word with the addition of grammatical
properties. The basic meaning and the category is the same.
• - s (plural) Number
• - ‘s (possessive) Possessive
• - ed (past tense)
• - ing (progressive) Tense
• - er (comparison)
• - est (superlative) Comparison

28
Derivational morphemes:
• Creating new words (with the addition of affixes) with a different
meaning that may belong to a different grammatical category.
• Write (V) re- + write rewrite (V)
• Write (V) writ + -er writer (N)
•V N
•N ADJ.
•N V
•N N

29
Inflection vs Derivation morphemes

30
Ambiguity
I made her duck.

• How many different interpretations does this sentence have?

• What are the reasons for the ambiguity?
• The categories of knowledge of language can be thought of as
ambiguity resolving components.
• How can each ambiguous piece be resolved?
• Does speech input make the sentence even more ambiguous?
• Yes – deciding word boundaries
31
 Ambiguity (cont.)
• Some interpretations of : I made her duck.

32
Ambiguity is Everywhere
• Lexical category: part of speech
‣ Duck can be a Noun or Verb
• V: Duck! I caused her to quickly lower her head or body.
• N: I cooked waterfowl for her benefit
‣ Her can be possessive (of her) or dative (for her)
• Possessive: I cooked waterfowl belonging to her.
• Dative: I cooked waterfowl for her benefit
• Lexical Semantics:
‣ Make can mean create or cook
• create: I made the (plaster) duck statue she owns
• cook: I cooked waterfowl for her benefit
 Ambiguity (cont.)
• Some interpretations of : I made her duck.
1. I cooked duck for her.
2. I cooked duck belonging to her.
3. I created a toy duck which she owns.
4. I caused her to quickly lower her head or body.
5. I used magic and turned her into a duck.
• duck – morphologically and syntactically ambiguous:
noun or verb.
• her – syntactically ambiguous: dative or possessive.
• make – semantically ambiguous: cook or create.
• make – syntactically ambiguous:
• Transitive – takes a direct object.
• Di-transitive – takes two objects.
• Takes a direct object and a verb.

34
Ambiguity (cont.)
• Grammar: Make can be:
‣ Transitive: (verb has a noun direct object)
• I cooked [waterfowl belonging to her]

‣ Ditransitive: (verb has 2 noun objects)

• I (magically) made [her] (into) [undifferentiated waterfowl]

‣ Action-transitive (verb has a direct object and another verb)

• I caused [her] [to move her body]

35
 Ambiguity (cont.)
• Some interpretations of : I made her duck.
1. I cooked duck for her.
2. I cooked duck belonging to her.
3. I created a toy duck which she owns.
4. I caused her to quickly lower her head or body.
5. I used magic and turned her into a duck.
• duck – morphologically and syntactically ambiguous:
noun or verb.
• her – syntactically ambiguous: dative or possessive.
• make – semantically ambiguous: cook or create.
• make – syntactically ambiguous:
• Transitive – takes a direct object. => 2
• Di-transitive – takes two objects. => 5
• Takes a direct object and a verb. => 4

36
Syntactic Ambiguity
• Structural ambiguity: one sentence can have many syntactic
representations
Another Ambiguity to “I made her duck” is
phonetics

38
Word2vec Architecture
• There are two architectures used by Word2vec:

• Continuous bag-of-words model: predicts the middle word based on

surrounding context words. The context consists of a few words before and
after the current (middle) word. This architecture is called a bag-of-words
model as the order of words in the context is not important.

• Continuous skip-gram model: predicts words within a certain range before

and after the current word in the same sentence.
Bag-of-Words Model
• The bag-of-words model is a way of representing text data when
modeling text with machine learning algorithms.

• The bag-of-words model is simple to understand and implement and

has seen great success in problems such as language modeling and
document classification.
What is a Bag-of-Words?
• A bag-of-words model, or BoW for short, is a way of extracting
features from text for use in modeling, such as with machine learning
algorithms.
• The approach is very simple and flexible, and can be used in a myriad
of ways for extracting features from documents.
• A bag-of-words is a representation of text that describes the
occurrence of words within a document. It involves two things:
• A vocabulary of known words.
• A measure of the presence of known words.
Steps of a Bag-of-Words?
• It is called a “bag” of words, because any information about the order
or structure of words in the document is discarded. The model is only
concerned with whether known words occur in the document, not
where in the document.
• Step 1: Collect Data
• Step 2: Design the Vocabulary
• Step 3: Create Document Vectors
Step 1: Collect Data
• It was the best of times,
• it was the worst of times,
• it was the age of wisdom,
• it was the age of foolishness,
Step 2: Design the Vocabulary
• Now we can make a list of all of the words in our model vocabulary.
• The unique words here (ignoring case and punctuation) are:
• “it”
• “was”
It was the best of times, • “the”
it was the worst of times, • “best”
it was the age of wisdom, • “of”
it was the age of foolishness, • “times”
• “worst”
• “age”
• “wisdom”
• “foolishness”
• That is a vocabulary of 10 words from a corpus containing 24 words.
Step 3: Create Document Vectors
• The next step is to score the words in each document.

• The objective is to turn each document of free text into a vector that we
can use as input or output for a machine learning model.

• Because we know the vocabulary has 10 words, we can use a fixed-length

document representation of 10, with one position in the vector to score
each word.

• The simplest scoring method is to mark the presence of words as a boolean

value, 0 for absent, 1 for present.
• The scoring of the document would look as follows:
• “it” = 1
• “was” = 1
• “the” = 1
• “best” = 1
• “of” = 1
• “times” = 1
• “worst” = 0
• “age” = 0
• “wisdom” = 0
• “foolishness” = 0
 “it” = 1
“was” = 1
“the” = 1
“best” = 1
“of” = 1
“times” = 1
“worst” = 0
“age” = 0
“wisdom” = 0
“foolishness” = 0

1 "it was the worst of times" = [1, 1, 1, 0, 1, 1, 1, 0, 0, 0]

2 "it was the age of wisdom" = [1, 1, 1, 0, 1, 0, 0, 1, 1, 0]
3 "it was the age of foolishness" = [1, 1, 1, 0, 1, 0, 0, 1, 0, 1]
Continuous Bag of Words
• Let us draw a simple Word2vec example diagram to understand the
continuous bag of word architecture.
Skip-Gram Model
• Skip-Gram approach is used to predict a sentence given an input
word.
• One can treat it as the reverse of the Continuous bag of word model
where the input is the word and model provides the context or the
sequence.
Confusion matrix
➢True Positive:
• You predicted positive and it’s true.
➢True Negative:
• You predicted negative and it’s true.
➢False Positive: (Type 1 Error)
• You predicted positive and it’s false.
➢False Negative: (Type 2 Error)
• You predicted negative and it’s false.