Syllabus
CS 478
Natural Language Processing (NLP)
Instructor
Ziyu Yao (ziyuyao [at] gmu [dot] edu)Office Hours: by appointments (Virtual or in person at ENGR4415).
Teaching Assistant
Saurabh Srivastava (ssrivas6@gmu.edu)Office Hours: Thu 2-4pm, ENGR 4456 D3 or online (email the TA for Zoom appointment) TBD
Meets
Monday and Wednesday, 3:00 to 4:15 PM, Exploratory Hall L102.Course Web Page
https://nlp.cs.gmu.edu/course/cs478-fall24/.We will use Blackboard for materials distribution, Gradescope for assignment submission and grading, and Piazza for Q&A (sign up link here).
Course Description
Massive amounts of information in our daily life are expressed in natural language. In this class, we will study building computing systems that can process, understand, and communicate in natural language. This field is called natural language processing, or NLP. This class will focus on introducing foundamental concepts in NLP, and will cover techniques and necessary programming skills for building machine learning/deep learning-based NLP models. In the last several classes, we will further study cutting-edge research problems in NLP, including text generation, question answering, neural network interpretation, interactive learning, multilingual NLP, and so on.Prerequisites
CS310 (Data Structure), CS330 (Formal Methods and Models), and proficiency in Python programming. Please contact the instructor if you have questions about the necessary background.Class Format
The class will be in-person. Each class will take the following format:- Reading: Before the class, you will be pointed to some reading materials (see "Reading Materials" in the course schedule) . Reading is not required but highly recommended to do before the class.
- Summary/Elaboration/Q&A: In the class, the instructor will summarize important points from the reading material, elaborating on details that were not included in the reading while fielding any questions. New material on cutting-edge methods, or a deep look into one salient method will also be covered.
- Quiz: In some classes, there will be in-class quizzes evaluating student learning performance. The quiz will be based on the content in the previous lectures.
Grading
There will be no midterm or final exam. Your final grade will be dependent on:Quizzes and Class Participation: 30%. We will have weekly in-class quizzes, each containing 2-3 short-answer questions covering contents introduced in previous lectures. The instructor will specify the range one class before the quiz. Each quiz question is worth 1% of the total grade, with the remaining credits for class participation (based on student attendance and ad-hoc presentation).
Assignments: 70%. There will be six assignments in this class. Each assignment must be completed independently.
- A1: Introduction (5%). The assignment will ask you a few questions about the class logistic and help you get familiar with classic NLP topics.
- A2: Understand n-gram language models (10%).
- A3: Build a feedforward neural net for sentiment classification (20%). This assignment will have three submission checkpoints.
- A4: Build a BERT-based sentiment classifier (10%).
- A5: Explore Large Language Models (15%).
- A6: Analyze a BERT-based sentiment classifier (10%).
Your final letter grade will be given based on:
| Letter Grade | Points (out of 100) |
|---|---|
| A+ | 100+ (w/ extra credits) |
| A | 95-100 |
| A- | 90-94 |
| B+ | 85-89 |
| B | 82-84 |
| B- | 78-81 |
| C+ | 74-77 |
| C | 72-73 |
| C- | 70-71 |
| D | 60-69 |
| F | 0-59 |
Late Day Policy for Assignments:
In case there are unforeseen circumstances that don’t let you turn in your assignment on time, 5 late days total over the six assignments will be allowed. Notes that the late days cannot be used fractionally, e.g., submitting the assignment 1 hour late will incur 1 late day.
Assignments that are late beyond the allowed late days will be graded down by 5% per day.
In the case of a serious illness or other excused absence, as defined by university policies (including providing necessary evidence), coursework submissions will be accepted late by the same number of days as the excused absence. However, students should inform the instructor as soon as they can, ideally before the due date.
Class Attendance Policy:
As we will have quizzes for some classes, attendence is highly suggested. If you miss a quiz, you lose the credits (i.e., no make ups for quiz). However, in the case of a serious illness or other excused absence, as defined by university policies (including providing necessary evidence), students will be excused and the quiz can be taken home. However, students should inform the instructor as soon as they can, ideally before the quiz date.
Readings
Students should be able to understand the course content just by following the lecture and by doing the readings. However, the following textbooks serve as good references.- Jurafsky and Martin, Speech and Language Processing, 3rd edition [online; version Aug 20, 2024] (Referred to as "JM");
- Jacob Eisenstein, Natural Language Processing [online] (Referred to as "Eisenstein");
- Yoav Goldberg, Neural Network Methods in Natural Language Processing [publisher] [online primer pdf] (Referred to as "Goldberg-Publisher/Primer"); Note that the "publisher" version can be downloaded if you use the school VPN.
Tentative Schedule
| # | Date | Topic | Reading Materials | Assignment Dates |
|---|---|---|---|---|
| 1 | 08/26 | Introduction and Class Outline | A1 Out | |
| 2 | 08/28 | Text Processing and Probability Basics | JM Ch2 | |
| 3 | 09/02 | Labor Day - No Class | ||
| 4 | 09/04 | N-gram Language Models | JM Ch3.1-3.4, 3.6, 10.2 | A1 Due; A2 Out |
| 5 | 09/09 | ML/NLP Experimental Design | JM Ch4.7-4.8 | |
| 6 | 09/11 | Classification 1 | JM Ch5.1-5.6, Ch5.8, Eisenstein Ch2.3-2.4, 2.6 | |
| 7 | 09/16 | Classification 2 | ||
| 8 | 09/18 | Classification 3 & Intro to Neural Nets | JM Ch7.1-7.2 | A2 Due; A3 Out |
| 9 | 09/23 | Neural 1: Feedforward Neural Networks | JM Ch7.3-7.7 & Goldberg-Primer Ch6.1-6.3
Blog by Michael Nielsen, DL book. PyTorch basics |
|
| 10 | 09/25 | Neural 1: Feedforward Neural Networks (cont.) | ||
| 11 | 09/30 | Neural 2: Word Embeddings | JM Ch6; Mikolov et al., 2013a&b | A3.1 Due |
| 12 | 10/02 | Neural 2: Word Embeddings (Cont.) | JM Ch6; Mikolov et al., 2013a&b | |
| 13 | 10/07 | Neural 3: RNN-based Neural Language Models | JM Ch8.1-8.5; "understand LSTM" blog by Olah; "gradient vanishing" blog by Nielsen; Karpathy et al. 2015 | |
| 14 | 10/09 | Neural 4: Seq2Seq & Attention, Transformers | JM Ch8.7-8.8, 9.1-9.2, Eisenstein 18.3.1; Attention-based NMT; Transformer paper and Alammar's blog |
A3.2 Due |
| 15 | 10/14 | Fall Break - No Class | ||
| 16 | 10/16 | Assginment and Quiz Q&A; Intro to Contextual Representations | Peters et al., 2018 (ELMo); Devlin et al., 2019 (BERT) | A4 Out |
| 17 | 10/21 | Contextual Representations | Peters et al., 2018 (ELMo); Devlin et al., 2019 (BERT); JM Ch11.1-11.4. | |
| 18 | 10/23 | Prompt-based Learning | Liu et al. survey 2021, JM Ch10.1, 10.3, 12.1, 12.4, 12.6 | A3.3 Due |
| 19 | 10/28 | Emerging Topics about LLM Agents | Lilian Weng's blog; Chain-of-Thought; Automatic Prompt Engineering; Stanford Smallville | |
| 20 | 10/30 | Sequence: POS tagging (HMM) & NER (CRF) | JM Ch17.1-17.5, 8.4, Eisenstein Ch7.5.3 | A4 Due; A5 Out |
| 21 | 11/04 | Parsing 1: Syntactic Parsing | JM Ch19.1-19.2, 18.1-18.3, 18.6, 18.8 | |
| 22 | 11/06 | Parsing 2: Semantic Parsing | Eisenstein Ch13 | |
| 23 | 11/11 | Interpreting and Debugging NLP Models 1 & A6 Preview | LIME, e-SNLI, EMNLP20 tutorial, CheckList | |
| 24 | 11/13 | Interpreting and Debugging NLP Models 2 | Mechanistic Interpretability (Rai et al. 2024) | |
| 25 | 11/18 | Text Generation & Machine Translation | JM Ch15.1-15.4, 13.2, 13.6; Eisenstein 18.1-18.2; Maynez et al., 2020; See et al., 2017; BERTScore | |
| 26 | 11/20 | Question Answering | JM Ch14.1-14.2; ACL20 tutorial;
QA over text: Chen et al., 2017 (DrQA); Lee et al., 2019 (ORQA); Zhu et al., 2021 (survey). QA over structured data:Pasupat&Liang, 2015 (Table QA); Yih et al., 2015 (KBQA); Rajpurkar et al., 2016 |
A5 Due; A6 Out |
| 27 | 11/25 | Retrieval-Augmented Generation (RAG) | JM Ch14.3, Lewis et al., 2021 | |
| 28 | 11/27 | Thanksgiving - No Class | ||
| 29 | 12/02 | Human-AI Interaction | Wang et al., 2016, Hancock et al., 2019, guidelines for human-AI interaction; Materials by Daume III; InstructGPT/RLHF | |
| 30 | 12/04 | AI/NLP Ethics | Zhao et al., 2017, Greshake et al., 2023, OWASP's Top10 LLM Risks | |
| 31 | 12/09 | Select A5 Presentations | A6 Due |