BIRLA INSTITUTE OF

TECHNOLOGY & SCIENCE, PILANI

WORK INTEGRATED LEARNING PROGRAMMES
COURSE HANDOUT
Part A: Content Design
Course Title Natural Language Processing
Course No(s)
Credit Units 4 units
Course Author Dr. Chetana Gavankar
Version No 1.0
Date September 2022

Course Objectives
No Course Objective

CO1 To learn the fundamental concepts and techniques of natural language processing (NLP)
including Language Models, Word Embedding, Part pf speech Tagging, Parsing

CO2 To learn computational properties of natural languages and the commonly used algorithms
for processing linguistic information

CO3 To introduce basic mathematical models and methods used in NLP applications to
formulate computational solutions.

CO4 To introduce students research and development work in Natural language Processing
Text Book(s)
T1 Jurafsky and Martin, SPEECH and LANGUAGE PROCESSING: An Introduction to
Natural Language Processing, Computational Linguistics, and Speech Recognition,
McGraw Hill
T2 Manning and Schütze, Foundations of Statistical Natural Language Processing, MIT Press.
Cambridge, MA

Reference Book(s) & other resources

R1 Allen James, Natural Language Understanding
R2 Neural Machine Translation by Philipp Koehn
R3 Semantic Web Primer (Information Systems) By Antoniou, Grigoris; Van Harmelen, Frank
Modular Content Structure

1. Natural Language Understanding and Generation

 The Study of Language.
 Applications of Natural Language Understanding.
 Evaluating Language Understanding Systems.
 The Different Levels of Language Analysis.
 The Organization of Natural Language Understanding Systems.

2. N-gram Language Modelling

 N-Grams
 Generalization and Zeros.
 Smoothing
 The Web and Stupid Backoff
 Evaluating Language Models
 Smoothing
 The Web and Stupid Backoff

3 Neural networks and Neural language Models

 Units
 The XOR problem
 Feed-Forward Neural Networks
 Training Neural Nets
 Neural Language Models -expand spend more time

4. Part-of-Speech Tagging
 (Mostly) English Word Classes
 The Penn Treebank Part-of-Speech Tag set
 Part-of-Speech Tagging
 Markov Chains
 The Hidden Markov Model
 HMM Part-of-Speech Tagging
 Part-of-Speech Tagging for Morphological Rich Languages

5. Hidden Markov Models and MEMM

 The Hidden Markov Model
 Likelihood Computation: The Forward Algorithm
 Decoding: The Viterbi Algorithm
 HMM Training: The Forward-Backward Algorithm
 Maximum Entropy Markov Models
 Bidirectionality

6. Topic Modelling
 Mathematical foundations for LDA : Multinomial and Dirichlet distributions
 Intuition behind LDA
 LDA Generative model
 Latent Dirichlet Allocation Algorithm and Implementation
 Gibbs Sampling

7. Vector semantics and Embedding

  Lexical semantics
 Vector semantics
 Word and Vectors
 TFIDF
 Word2Vec, Skip gram and CBOW
 Glove
 Visualizing Embedding’s

8. Grammars and Parsing.

 Grammars and Sentence Structure.
 What Makes a Good Grammar
 A Top-Down Parser.
 Bottom-Up Chart Parser.
 Top-Down Chart Parsing.
 Finite State Models and Morphological Processing.
 Grammars and Logic Programming.
9. Statistical Constituency Parsing
 Probabilistic Context-Free Grammars
 Probabilistic CKY Parsing of PCFGs
 Ways to Learn PCFG Rule Probabilities
 Problems with PCFGs
 Improving PCFGs by Splitting Non-Terminals
 Probabilistic Lexicalized CFGs
10. Dependency Parsing
 Dependency Relations
 Dependency Formalisms
 Dependency Treebanks
 Transition-Based Dependency Parsing
 Graph-Based Dependency Parsing
 Dependency parser using neural network

11. Encoder-Decoder Models, Attention and Contextual Embeddings

 Neural Language Models and Generation
 Encoder-Decoder Networks, Attention
 Applications of Encoder-Decoder Networks
 Self-Attention and Transformer Networks
 BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
 Contextual Word Representations: A Contextual Introduction
 The Illustrated BERT, ELMo, and co.
 XLM

12. Word sense disambiguation

 Word Senses
 Relations between Senses
 WordNet: A Database of Lexical Relations
 Word Sense Disambiguation
 Alternate WSD algorithms and Tasks
 Using Thesauruses to Improve Embedding’s
 Word Sense Induction
13. Semantic web ontology and Knowledge Graph
 Introduction to semantic web
 Semantic web ontology
 Semantic web languages
 Ontology Engineering
 Ontology Learning
 Knowledge graph –construction of graph

14. Introduction to NLP Applications

 Brief introduction of state of art applications
 Text Summarization
 Machine Translation

Part B: Contact Session Plan

Academic Term
Course Title
Course No
Lead Instructor
Course Contents

Contact List of Topic Title Topic # Text /

session (from content structure in Part A) (from Ref Book /
content External
structure resource
in Part A)

1 Natural Language Understanding and Generation Chapter1 T2

1.1 The Study of Language.
1.2 Applications of Natural Language Understanding.
1.3 Evaluating Language Understanding Systems.
1.4 The Different Levels of Language Analysis.
1.5 The Organization of Natural Language Understanding
Systems.

2 N-gram Language Modelling Chapter 3 T1

 N-Grams
 Generalization and Zeros.
 Smoothing
 The Web and Stupid Backoff
 Evaluating Language Models
 Smoothing
 The Web and Stupid Backoff

3 Neural Network and Neural Language Modelling Chapter 4 R2

 Units
 The XOR problem
 Feed-Forward Neural Networks
 Training Neural Nets
  Neural Language Models

4 Vector semantics and Embedding Chapter 6 T1 and

 Lexical semantics lecture notes
 Vector semantics
 Word and Vectors https://www.
youtube.com
 TFIDF
/watch?v=hQ
 Word2Vec, Skip gram and CBOW wFeIupNP0
 Glove
 Visualizing Embedding’s

5 Part-of-Speech Tagging Chapter8 T1 and class

 (Mostly) English Word Classes notes
 The Penn Treebank Part-of-Speech Tag set
 Part-of-Speech Tagging
 Markov Chains
 The Hidden Markov Model
 HMM Part-of-Speech Tagging
 Part-of-Speech Tagging for Morphological Rich
Languages

6 Hidden Markov Model Algorithms Appendix T1 and class

 Likelihood Computation: The Forward Algorithm chapter A notes
 Decoding: The Viterbi Algorithm
 HMM Training: The Forward-Backward Algorithm
 Maximum Entropy Markov Model
 Bidirectionality

7 Topic modelling Class Notes

 Mathematical foundations for LDA
 Multinomial and Dirichlet distributions
 Intuition behind LDA
 LDA Generative model
 Latent Dirichlet Allocation Algorithm and
Implementation
 Gibbs Sampling

Review of M1 to M7

9 Grammars and Parsing Chapter3 T2

 Grammars and Sentence Structure.
 What Makes a Good Grammar
 A Top-Down Parser.
 A Bottom-Up Chart Parser.
 Top-Down Chart Parsing.
 Finite State Models and Morphological Processing.
 Grammars and Logic Programming.
 Parsing
 10 Statistical Constituency Parsing Chapter 14 T1
 Probabilistic Context-Free Grammars
 Probabilistic CKY Parsing of PCFGs
 Ways to Learn PCFG Rule Probabilities
 Problems with PCFGs
 Improving PCFGs by Splitting Non-Terminals
 Probabilistic Lexicalized CFGs

11 Dependency Parsing Chapter 19 T1 and class

 Dependency Relations notes
 Dependency Formalisms
 Dependency Treebanks
 Transition-Based Dependency Parsing
 Graph-Based Dependency Parsing
 Dependency parsers using neural network

12 Encoder-Decoder Models, Attention and Contextual Chapter10 T1

Embeddings https://colab.
 Neural Language Models and Generation research.goo
 Encoder-Decoder Networks, Attention gle.com/driv
 Applications of Encoder-Decoder Networks e/1iqs9Y5_z
LI6R6mAwl
 Self-Attention and Transformer Networks
napcxcUbKj
 BERT: Pre-training of Deep Bidirectional Transformers pv2CC?usp=
for Language Understanding sharing
 Contextual Word Representations: A Contextual
Introduction
 The Illustrated BERT, ELMo, and co.
 XLM

13 Word sense and word net Chapter15 T1

 Word Senses
 Relations between Senses
 WordNet: A Database of Lexical Relations
 Word Sense Disambiguation
 Alternate WSD algorithms and Tasks
 Using Thesauruses to Improve Embedding
 Word Sense Induction

14 Semantic web ontology and Knowledge Graphs Chapter 24 R1 and class

 Introduction notes
 Ontology and Ontologies
 Ontology Engineering
 Ontology Learning

15 State of art applications Class Notes

and web
references
 16 Review of session 9 to session 15

Detailed Plan for Lab work

Lab Lab Sheet Session

Lab Objective Reference
No. Access URL
1 Introduction to NLTK, Spacy and other open 1
source tools
2 Language Modelling- Neural 2,3

3 Part of speech tagging 4,5

4 Topic Modeling 7
5 Parsing-Dependency-neural 9,10,11
6 Wordnet, Ontology and Knowledge Graph 12,13,14

Evaluation Scheme
Evaluation Name Type Weight Duration Day, Date, Session,
Component (Quiz, Lab, Project, (Open book, Time
Midterm exam, End Closed book,
semester exam, etc) Online, etc.)

EC – 1 Quiz 10% To be announced

EC – 2 Assignment 20% To be announced

EC – 3 Mid-term Exam Open book 30% To be announced

EC – 4 End Semester Exam Open book 40% To be announced

Important Information
Syllabus for Mid-Semester Test (Closed Book): Topics in Weeks 1-8 (1-18 Hours)
Syllabus for Comprehensive Exam (Open Book): All topics given in plan of study

Notes
 Quiz and Assignments timelines will be announced on the canvas portal.
 Deadlines for evaluation components will NOT be extended and the student is requested not
to wait for the deadline to start working on Quiz/Assignment
 Syllabus for Mid-Semester Test (Closed Book): Topics in Session Nos. 1 to 8
 Syllabus for Comprehensive Exam (Open Book): All topics (Session Nos. 1 to 16)
 Strictly NO MAKEUPS for Quiz and Assignments and all submissions after the announced
deadlines will not be considered for evaluation.
 All assignments will be subjected to plagiarism check, and if violated will be subject to
disciplinary action apart from nullifying all the marks/grades assigned.

Important links and information:

Canvas: Students are expected to visit the Canvas portal on a regular basis and stay up to date with
the latest announcements and deadlines.

Contact sessions: Students should attend the online lectures as per the schedule provided.
Evaluation Guidelines:
1. EC-1 consists of Assignments and Quizzes. Announcements regarding the same will be made
in a timely manner.
2. For Closed Book tests: No books or reference material of any kind will be permitted.
Laptops/Mobiles of any kind are not allowed. Exchange of any material is not allowed.
3. For Open Book exams: Use of prescribed and reference text books, in original (not photocopies)
is permitted. Class notes/slides as reference material in filed or bound form is permitted.
However, loose sheets of paper will not be allowed. Use of calculators is permitted in all exams.
Laptops/Mobiles of any kind are not allowed. Exchange of any material is not allowed.
4. If a student is unable to appear for the Regular Test/Exam due to genuine exigencies, the student
should follow the procedure to apply for the Make-Up Test/Exam. The genuineness of the
reason for absence in the Regular Exam shall be assessed prior to giving permission to appear
for the Make-up Exam. Make-Up Test/Exam will be conducted only at selected exam centres.
It shall be the responsibility of the individual student to be regular in maintaining the self-study schedule
as given in the course handout, attend the lectures, and take all the prescribed evaluation components
such as Assignment/Quiz, Mid-Semester Test and Comprehensive Exam according to the evaluation
scheme provided in the handout.

Learning Outcomes:
No Learning Outcomes

LO1 Should have a good understanding of the field of natural language processing.

LO2 Should have knowledge of important techniques like language modelling, parsing, used
in natural language processing

LO3 Should be able to apply NLP algorithms along with deep learning algorithms for state of
art areas like word embedding