CS626: Speech, Natural Language Processing and the Web

Announcement

The first assignment is uploaded on teams, moodle and course page.
Assignment2 is uploaded
Assignment3 is uploaded
Solutions for sample quiz Sample quiz solution
Solutions for quiz1 Quiz1 solution
Solutions for Midsem Midsem solution

Course Details

CS626: Speech and Natural Language Processing and the Web
Department of Computer Science and Engineering
Indian Institute of Technology Bombay

Time Table and Venue

Monday: 8:30 AM to 9:25 AM
Tuesday: 9:30 AM to 10:25 AM
Thursday: 10:35 PM to 11:30 AM

Course Description

The general approach in the course will be covering (i) a language phenomenon, (ii) the corresponding language processing task, and (iii) techniques based on deep learning, classical machine learning and knowledge base. On one hand we will understand the language processing task in detail using linguistics, cognitive science, utility etc., on the other hand we will delve deep into techniques for solving the problem. The topics are given now.

Sound: Biology of Speech Processing; Place and Manner of Articulation; Peculiarities of Vowels and Consonants; Word Boundary Detection; Argmax based computations; Hidden Markov Model and Speech Recognition; deep neural nets for speech processing.
Morphology: Morphology fundamentals; Isolating, Inflectional, Agglutinative morphology; Infix, Prefix and Postfix Morphemes, Morphological Diversity of Indian Languages; Morphology Paradigms; Rule Based Morphological Analysis: Finite State Machine Based Morphology; Automatic Morphology Learning; Deep Learning based morphology analysis.
Shallow Parsing: Part of Speech (POS) Tagging; HMM based POS tagging; Maximum Entropy Models and POS; Random Fields and POS; DNN for POS.
Parsing: Constituency and Dependency Parsing; Theories of Parsing; Scope Ambiguity and Attachment Ambiguity Resolution; Rule Based Parsing Algorithms; Probabilistic Parsing; Neural Parsing.
Meaning: Lexical Knowledge Networks, Wordnet Theory and Indian Language Wordnets; Semantic Roles; Word Sense Disambiguation; Metaphors.
Discourse and Pragmatics: Coreference Resolution; Cohesion and Coherence.
Applications: Machine Translation; Sentiment and Emotion Analysis; Text Entailment; Question Answering; Code Mixing; Analytics and Social Networks, Information Retrieval and Cross Lingual Information Retrieval (IR and CLIR)

References

Allen, James, Natural Language Understanding, Second Edition, Benjamin/Cumming, 1995.
Charniack, Eugene, Statistical Language Learning, MIT Press, 1993
Jurafsky, Dan and Martin, James, Speech and Language Processing, Speech and Language Processing (3rd ed. draft), Draft chapters in progress, October 16, 2019.
Manning, Christopher and Heinrich, Schutze, Foundations of Statistical Natural Language Processing, MIT Press, 1999.
Jacob Eisenstein, Introduction to Natural Language Processing, MIT Press, 2019.
Deep Learning, Ian Goodfellow, Yoshua Bengio, and Aaron Courville, MIT Press, 2016.
Radford, Andrew et. al., Linguistics, an Introduction, Cambridge University Press, 1999.
Pushpak Bhattacharyya, Machine Translation, CRC Press, 2017.

Journals: Computational Linguistics, Natural Language Engineering, Machine Learning, Machine Translation, Artificial Intelligence
Conferences: Annual Meeting of the Association of Computational Linguistics (ACL), Computational Linguistics (COLING), European ACL (EACL), Empirical Methods in NLP (EMNLP), Annual Meeting of the Special Interest Group in Information Retrieval (SIGIR), Human Language Technology (HLT).

Pre-requisites

Data Structures and Algorithms, Python (or similar language) Programming skill

Lecture Slides

Lecture	Topics	Readings and useful links
Course Overview	Introduction Motivation	Lecture1
Lecture2,3,4 (Week of 17th August)	POS Tagging (Rule based POS tagging, Mathematics of POS tagging)	Lecture2
Lecture5,6,7 (Week of 24th August)	POS Tagging (HMM, Viterbi decoding)	Lecture5
Lecture8,9,10 (Week of 31st August)	Shallow parsing, CRF, Morphology brief	Lecture8
Lecture 11,12,13 (Week of 7th September)	MEMM, CRF, Feature Engineering	Lecture11_final
Lecture 14,15,16 (Week of 14th September)	Deep parsing	Lecture14_final
Lecture 17,18,19 (Week of 21st September)	Deep parsing	Lecture17_final
Lecture 20,21,22 (Week of 28th September)	Dependency Parsing, Technique of Probabilistic Parsing, Difficult Parsing Phenomena	Lecture20_final
Lecture 23,24,25 (Week of 12th October)	Start of Neural Network	Lecture23_v1
Lecture 26,27,28 (Week of 19th October)	Neural Network	Lecture26_final
Lecture 29,30,31 (Week of 26th October)	Softmax FFNN-BP and Neural Dependency Parsing	Lecture29_final
Lecture 32,33,34 (Week of 2nd November)	Deeper into Projectivity and Neural Dependency Parsing	Lecture32_final
Lecture 35,36,37 (Week of 9th November)	RNN, Seq2seq, Machine Translation	Lecture35_final Lecture36_MachineTranslation
Lecture 38,39,40 (Week of 16th November)	RNN, Seq2seq, Data Driven Machine Translation (SMT and NMT)	Lecture38_v1 Lecture38_final

Lecture videos

Lecture videos are regularly uploaded on MSTeams.

Assignments

Date	Assignment#	Topic	Deadline	Link
23/08/2020	Assignment1	POS Tagging	15/09/2020 11:59 PM	Assignment1
14 September 2020	Assignment2	Chunking	No deadline	Assignment2
27 September 2020	Assignment3	Parsing	No deadline	Assignment3

ReadingList

Sag, Ivan A., Timothy Baldwin, Francis Bond, Ann Copestake, and Dan Flickinger. "Multiword expressions: A pain in the neck for NLP." In International conference on intelligent text processing and computational linguistics, pp. 1-15. Springer, Berlin, Heidelberg, 2002.
Mitchell P. Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a Large Annotated Corpus of English: The Penn Treebank. Comput. Linguist., 19(2):313–330, June.
Xuezhe Ma and Eduard H. Hovy. 2016. End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF. CoRR, abs/1603.01354. (comment: POS and NER)
Anders Søgaard and Yoav Goldberg. 2016. Deep multi-task learning with low level tasks supervised at lower layers. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 231–235, Berlin, Germany, August. Association for Computational Linguistics.
Chapter2 of Machine Translation by Professor Pushpak
Smoothing from Manning and Schutz, "Foundations of Statistical Natural Language Processing", Page 199 (general) and Page 354 (POS specific).
Sha, Fei, and Fernando Pereira. "Shallow parsing with conditional random fields." In Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, pp. 213-220. 2003.
Ratnaparkhi, Adwait. "A maximum entropy model for part-of-speech tagging." In Conference on empirical methods in natural language processing. 1996.
Harshada Gune, Mugdha Bapat, Mitesh Khapra and Pushpak Bhattacharyya, Verbs are where all the Action Lies: Experiences of Shallow Parsing of a Morphologically Rich Language, Computational Linguistics Conference (COLING 2010), Beijing, China, August 2010.
Erik F. Tjong Kim Sang and Sabine Buchholz, Introduction to the CoNLL-2000 Shared Task: Chunking. In: Proceedings of CoNLL-2000, Lisbon, Portugal, 2000.
John Lafferty, Andrew McCallum, and Fernando C.N. Pereira, "Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data", ICML 2001.
Toutanova, Kristina; Manning, Christopher D. (2000). "Enriching the Knowledge Sources Used in a Maximum Entropy Part-of-Speech Tagger". Proc. J. SIGDAT Conf. on Empirical Methods in NLP and Very Large Corpora (EMNLP/VLC-2000). pp. 63–70.

CS626: Speech, Natural Language Processing and the Web

Announcement

Course Details

CS626: Speech and Natural Language Processing and the Web
Department of Computer Science and Engineering
Indian Institute of Technology Bombay

Time Table and Venue

Course Description

References

Pre-requisites

Course Instructors

Teaching Assistants

Lecture Slides

Lecture videos

Assignments

ReadingList

Contact Us

CS626: Speech, Natural Language Processing and the Web

Announcement

Course Details

CS626: Speech and Natural Language Processing and the Web Department of Computer Science and Engineering Indian Institute of Technology Bombay

Time Table and Venue

Course Description

References

Pre-requisites

Course Instructors

Teaching Assistants

Lecture Slides

Lecture videos

Assignments

ReadingList

Contact Us

CS626: Speech and Natural Language Processing and the Web
Department of Computer Science and Engineering
Indian Institute of Technology Bombay