BIRLA INSTITUTE OF TECHNOLOGY & SCIENCE, PILANI
WORK INTEGRATED LEARNING PROGRAMMES
COURSE HANDOUT
Part A: Content Design
Course Title Natural Language Processing
Course No(s)
Credit Units
Course Author Vijayalakshmi
Version No 1.0
Date
Course Objectives
No Course Objective
CO1 To learn the fundamental concepts and techniques of natural language processing (NLP)
CO2 To learn computational properties of natural languages and the commonly used algorithms
for processing linguistic information
Text Book(s)
T1 Speech and Language processing: An introduction to Natural Language Processing,
Computational Linguistics and speech Recognition by Daniel Jurafsky and James H.
Martin[2nd edition]
Speech and Language processing: An introduction to Natural Language Processing,
Computational Linguistics and speech Recognition by Daniel Jurafsky and James H.
Martin[3rd edition]
T2 Foundations of statistical Natural language processing by Christopher D.Manning and
Hinrich schutze
Reference Book(s) & other resources
Handbook of Natural Language Processing, Second Edition—NitinIndurkhya, Fred J.
R1 Damerau, Fred J. Damerau
R2 Natural Language Processing with Python by Steven Bird, Ewan Klein, Edward Lopper
Modular Content Structure
1. Introduction
1.1 Knowledge in speech and language processing
1.2 Ambiguity
1.3 Models and Algorithms
1.4 Language ,thought and understanding
1.5 The State of the art
2. Regular Expressions, Text Normalization, Edit Distance
2.1 Regular Expressions
2.1.1 Basic Regular Expression Patterns
2.1.2 Disjunction, Grouping, and Precedence
2.1.3 More Operators
2.1.4 Regular Expression Substitution, Capture Groups
2.1.5 Look ahead assertions
2.2 Words
2.3 Corpora
2.4 Text Normalization
2.5 Minimum Edit Distance
3. Text classifications
3.1 Naive Bayes and Sentiment Classication
3.1.1 Naive Bayes Classiers
3.1.2 Naïve Bayes for other texts classification task
3.1.3 Naïve Bayes as a language model
3.1.4 Evaluation: Precision, Recall, F Measure
3.1.5 Statistical significance testing
3.2 Logistic Regression
3.2.1 Classication: the sigmoid
3.2.2 Learning in Logistic Regression
3.2.3 The cross-entropy loss function
3.2.4 Gradient Descent
3.2.5 Regularization
3.2.6 Multinomial logistic regression
3.2.7 Interpreting models
4. Language models
4.1 N gram language models
4.1.1 N-Grams
4.1.2 Evaluating Language Models
4.1.3 Generalization and Zeros
4.1.4 Unknown Words
4.1.5 Smoothing
4.2 Neural Networks and Neural Language Models
4.2.1 The XOR problem
4.2.2 Feed-Forward Neural Networks
4.2.3Training Neural Nets
4.2.4 Neural Language Models
5. Lexical Analysis
5.1Lexical semantics
5.2Vector semantics
5.3 Words and vectors
5.4 Cosine for measuring similarity
5.5 TF-IDF : Weighing terms in the vector
5.6 Application of the tf-idf vector model
5.7 Word2vec
5.8 Visualizing Embedding
5.9 Semantic properties of embedding
5.10Bias and Embedding
5.11 Evaluating Vector Models
6. Word Disambiguation
6.1 Supervised Disambiguation
6.1.1 Bayesian classification
6.1.2FlipFlop algorithm
6.2 Dictionary-Based Disambiguation
6.2.1 Thesaurus-based disambiguation
6.2.2 Disambiguation based on translations in a second-language corpus
6.2.3 One sense per discourse, one sense per collocation
6.3 Unsupervised Disambiguation
6.4 Evaluation
6.4.1 Pseudo words
6.4.2 Upper and lower bounds on performance
7. Grammar
7.1 Introduction to Markov models and Hidden Markov models
7.2 Part-of-Speech Tagging
7.2.1 The Information Sources in Tagging
7.2.2 Markov Model Taggers
7.2.3 The probabilistic model
7.2.4 The Viterbi algorithm
7.2.5 Variations
7.3 Hidden Markov Model Taggers
7.3.1 Applying HMMs to POS tagging
7.3.2 The effect of initialization on HMM training
7.4 Transformation-Based Learning of Tags
7.4.1 Transformations
7.4.2 The learning algorithm
7.4.3 Relation to other models
7.4.4 Automata
8. Probabilistic Context Free Grammars-Statistical Parsing
8.1 Some Features of PCFGs
8.2 Questions for PCFGs
8.3 The Probability of a String
8.3.1 using inside probabilities
8.3.2 using outside probabilities
8.3.3 Finding the most likely parse for a sentence
8.3.4 Training a PCFG 398
8.4 Problems with the Inside-Outside Algorithm
9 Syntactic parsing and Dependency parsing
9.1 CKY parsing
9.2 Partial parsing
9.3Dependence parsing
9.3.1Dependency Relations
9.3.2Dependency Formalisms
9.3.3 Dependency Treebanks
9.3.4 Transition-Based Dependency Parsing
9.3.5 Graph-Based Dependency Parsing
9.3.6 Evaluation
10. Statistical Machine translation
10.1 Introduction
10.2 Approaches
10.3 Language models
10.4 Word alignment
10.5 Translation models
10.5.1 IBM models
10.5.2 Phrase Based systems
10.5.3 Syntax based systems
10.6 Direct translation models
11. Applications
11.1 Question answering system
11.1.1 IR-based Factoid Question Answering
11.1.2 Knowledge-based Question answering
11.1.3 Using multiple information sources: IBM’s Watson
11.1.4 Evaluation of Factoid Answers
11.2 Dialog Systems and Chatbots
11.2.1 Chatbots
11.2.1.1 Rule based Chatbots
11.2.1.2 Corpus based Chatbots
11.2.1.3 Evaluation of Chatbots
11.2.2 Frame Based Dialog Agents
11.2.2.1 Natural language understanding for filling slots
11.2.2.2 Evaluation
11.2.2.3 Voice XML
11.2.2.4 Evaluating Dialog Systems
11.2.2.5 Dialog System Design
11.3WordNet: Word Relations, Senses, and Disambiguation
11.3.1 Word Senses
11.3.2 WordNet: A Database of Lexical Relations
11.3.3 Word Similarity: Thesaurus Methods
11.3.4 Word Sense Disambiguation: Overview
11.3.5 Supervised Word Sense Disambiguation
11.3.6 WSD: Dictionary and Thesaurus Methods
11.3.7 Semi-Supervised WSD: Bootstrapping
11.3.8 Unsupervised Word Sense Induction
Learning Outcomes:
No Learning Outcomes
LO1 Should have a good understanding of the field of natural language processing.
LO2 Should have an algorithms and techniques used in this field.
LO3 Should also understand the how natural language processing is used in Machine
translation and Information extraction.
Part B: Contact Session Plan
Academic Term
Course Title Natural Language processing
Course No
Lead Instructor
Course Contents
Contact List of Topic Title Topic # Text/Ref
Hour (from content structure in Part A) (from Book/external
content resource
structure in
Part A)
1-2 Introduction Chapter1 T1[2nd edition]
Knowledge in speech and language processing,
Ambiguity, Models and Algorithms, Language,
thought and understanding, The State of the art.
2-4 Regular Expression: Chapter2 T1[3rd edition]
Basic Regular Expression Pattern, Disjunction,
Grouping, and Precedence, More Operators, Regular
Expression Substitution, Capture Groups, Look ahead
assertions, Words , Corpora , Text Normalization ,
Minimum Edit Distance
5-6 Text classifications: Chapter 4 T1[3rd edition]
Naive Bayes and Sentiment Classication: Naive
Bayes Classiers, Naïve Bayes for other texts
classification task, Naïve Bayes as a language model
Evaluation: Precision, Recall, F Measure, Statistical
significance testing
7-8 ogistic Regression: Classication: the sigmoid ,
L Chapter5 T1[3rd edition]
Learning in Logistic Regression ,The cross-entropy
loss function , Gradient Descent, Regularization ,
Multinomial logistic regression , Interpreting models
9-10 Language models: Chapter3 T1[3rd edition]
N gram language models: N-Grams , Evaluating
Language Models , Generalization and
Zeros,Unknown Words, Smoothing
11-12 Neural Networks and Neural Language Models: Chapter7 T1[3rd edition]
The XOR problem , Feed-Forward Neural Networks
Training Neural Nets, Neural Language Models
13-14 Lexical analysis: Chapter 6 T1[3rd edition]
Lexical semantics, Vector semantics, Words and
vectors, Cosine for measuring similarity, TF-IDF :
Weighing terms in the vector, Application of the tf-idf
vector model ,Word2vec,Visualizing Embedding,
Semantic properties of embedding, Bias and
Embedding, Evaluating Vector Models.
15-16 Computational Lexical semantics Chapter7 T2
Word Disambiguation
Supervised Disambiguation :Bayesian classification
,An information-theoretic approach
Dictionary-Based Disambiguation : Thesaurus-based
disambiguation , Disambiguation based on translations
in a second-language corpus ,One sense per discourse,
one sense per collocation
Unsupervised Disambiguation
Evaluation :Pseudo words, Upper and lower bounds
on performance
17-18 Grammar Chapter9 T2
Introduction to Markov Models
Markov Models, Hidden Markov Models, Why use
HMMs? ,General form of an HMM ,The Three
Fundamental Questions for HMMs , Finding the
probability of an observation ,Finding the best state
sequence
MMs: Implementation, Properties, and Variants
19-20 Part-of-Speech Tagging Chapter10 T2
Chapter 8 can T1[3rd edition]
The Information Sources in Tagging , Markov Model also be
Taggers, The probabilistic model , The Viterbi referred
algorithm, Variations, Hidden Markov Model Taggers
, Applying HMMs to POS tagging
The effect of initialization on HMM training
:Transformation-Based Learning of Tags ,
Transformations, The learning algorithm , Relation to
other models, Automata
21-22 Probabilistic Context Free Grammars Chapter 11 T2
Some Features of PCFGs :Questions for PCFGs ,The
Probability of a String , using inside probabilities ,
using outside probabilities , Finding the most likely
parse for a sentence, Training a PCFG ,Problems with
the Inside-Outside Algorithm
23-24 Syntactic Parsing : Chapter 11 T1[3rd edition]
CKY Parsing
Partial Parsing
Dependency Parsing:
Dependency Relations, Dependency Formalisms, Chapter13 T1[3rd edition]
Dependency Treebanks, Transition-Based Dependency
Parsing, Graph-Based Dependency Parsing ,Evaluation
24-25 Statistical Machine translation : Introduction Chapter 17,18 R2
Approaches, Language models, Word alignment
Translation models :
IBM models, Phrase Based systems, Syntax based
systems, Direct translation models
Example :Chinese Machine translation
25-26 Applications: Chapter 23 T1[3rd edition]
Question answering system: IR-based Factoid R1
Question Answering, Knowledge-based Question
answering, Using multiple information sources: IBM’s
Watson, Evaluation of Factoid Answers.
27-28 Dialog Systems and Chatbots Chapter 24 T1[3rd edition]
Chatbots :Rule based Chatbots, Corpus based
Chatbots, Evaluation of Chatbots
Frame Based Dialog Agents: Natural language
understanding for filling slots, Evaluation ,Voice
XML, Evaluating Dialog Systems, Dialog System
Design
29-30 WordNet: Word Relations, Senses, and Appendix c T1[3rd edition]
Disambiguation , Word Senses , WordNet: A Database
of Lexical Relations
Word Similarity: Thesaurus Methods
Word Sense Disambiguation: Overview, Supervised
Word Sense Disambiguation WSD: Dictionary and
Thesaurus Methods ,Semi-Supervised WSD:
Bootstrapping Unsupervised Word Sense Induction
31-32 Summary.
Evaluation Scheme
Evaluation Name Type Weight Duration Day, Date, Session,
Component (Quiz, Lab, Project, (Open book, Time
Midterm exam, End Closed book,
semester exam, etc) Online, etc.)
EC – 1 Quizzes / Assignment 20% To be announced
EC – 2 Mid-term Exam Closed book 30% 2 hours To be announced
EC – 3 End Semester Exam Open book 50% 3 hours To be announced
Note - Evaluation components can be tailored depending on the proposed model.
Important Information
Syllabus for Mid-Semester Test (Closed Book): Topics in Weeks 1-8 (1-18 Hours)
Syllabus for Comprehensive Exam (Open Book): All topics given in plan of study
Evaluation Guidelines:
1. EC-1 consists of either two Assignments or three Quizzes. Announcements regarding the
same will be made in a timely manner.
2. For Closed Book tests: No books or reference material of any kind will be permitted.
Laptops/Mobiles of any kind are not allowed. Exchange of any material is not allowed.
3. For Open Book exams: Use of prescribed and reference text books, in original (not
photocopies) is permitted. Class notes/slides as reference material in filed or bound form is
permitted. However, loose sheets of paper will not be allowed. Use of calculators is permitted
in all exams. Laptops/Mobiles of any kind are not allowed. Exchange of any material is not
allowed.
4. If a student is unable to appear for the Regular Test/Exam due to genuine exigencies, the
student should follow the procedure to apply for the Make-Up Test/Exam. The genuineness of
the reason for absence in the Regular Exam shall be assessed prior to giving permission to
appear for the Make-up Exam. Make-Up Test/Exam will be conducted only at selected exam
centres on the dates to be announced later.
It shall be the responsibility of the individual student to be regular in maintaining the self-study
schedule as given in the course handout, attend the lectures, and take all the prescribed evaluation
components such as Assignment/Quiz, Mid-Semester Test and Comprehensive Exam according to the
evaluation scheme provided in the handout.
</DIV>
