Syllabus
CS 678
Advanced Natural Language Processing
Instructors
Ziyu Yao (ziyuyao [at] gmu [dot] edu)Office Hours: Email for appointments.
TA
Anjishnu Mukherjee (amukher6 [at] gmu [dot] edu)Office Hours: Wednesday 9-11am ENGR 4456
Meets
Monday, 7:20 pm - 10:00 pm, Sandbridge Hall 107.Course Web Page
https://nlp.cs.gmu.edu/course/cs678-spring24/.We will use Blackboard and Gradescope for course materials/assignments/grading, and Piazza for Q&A (sign up link: https://piazza.com/gmu/spring2024/cs678).
Course Description
Massive amounts of information in our daily life are expressed in natural language. In this class, we will study building computing systems that can process, understand, and communicate in natural language. The class will start with an introduction to the foundations of natural language processing (NLP), and then focus on cutting-edge research problems in NLP. Each section will introduce a particular problem or phenomenon in natural language, describe why it is difficult to model, and demonstrate recent models that were designed to tackle this problem. In the process of doing so, the class will cover different techniques that are useful in creating and applying neural network models. The class will include assignments, short quizzes, and a final project.Learning Outcomes
- Be familiar fundamental NLP problems and the methods powering the current state-of-the-art in language technologies, such as large pre-trained neural language models;
- Understand the limitations of current technologies and be able to make informed decisions about addressing them using advanced machine learning techniques;
- Design, implement, and evaluate a computing-based solution to address a language-related problem using state-of-the-art tools.
Prerequisites
CS 580 (Intro to AI) or CS 584 (Data Mining). You should be proficient in (a) Algorithms and Data Structures and (b) Probability and Statistics (STAT 344) or equivalent. Students should be experienced with writing substantial programs in Python. Please contact the instructor if you have questions about the necessary background.Class Format
The class will be in-person. As the class aims to provide skills necessary to familiarize the students with, and to do cutting-edge NLP research, the classes and assignments will be at least partially implementation-focused. In general, each class will take the following format:- Reading: Before some lectures, you will be pointed to some reading materials (see "Reading Materials" in course schedule) that you should read before coming to class that day.
- Summary/Elaboration/Questions: The instructor will summarize the important points of the reading material, elaborate on details that were not included in the reading while fielding any questions. Finally, new material on cutting-edge methods, or a deep look into one salient method will be covered.
- Demo/Code Walk: In some classes we will walk through some demonstration code that implements a simple version of the main concepts presented in the reading material.
Grading
There will be no midterm or final exam. Your final grade will be dependent on:Four homework assignments (50%): In the first weeks of the semester, we will have three programming assignments (each worth 10% of your grade, to be completed independently) to ensure that everybody gets a minimal hands-on experience with building a state-of-the-art NLP model or system. The fourth assignment (worth 20% of your grade) will be a longer one, where you will experience the full pipeline of building a dataset and an NLP solution for a problem. These assignments will be useful (if not necessary) for implementing your projects later in the class.
- HW0: Preliminaries and introduction (10%)
- HW1: Implementing a small BERT model (10%)
- HW2: Implementing a tool-augmented LLM agent (10%)
- HW3: Text classification from scratch (20%) -- As it is a longer assignment, in practice HW3 will involve two checkpoints: HW3-Ckpt1 and HW3-Ckpt2.
Quizzes (20%): To make sure everyone learns the core concepts taught in the class, we will have 4 take-home quizzes (each worth 5% of your grade). These will be released on Blackboard, and you will have 6 days to complete them (individually). They will only have a few questions with an answer that will not require more than 4-5 sentences.
Project (30%): The bulk of your grade will be based on a group research project related to the topics we will discuss in class. The groups will be of 2-3 people. If you intend to complete the project independently or with more than 3 people in your group, you MUST ask for permission from the instructor.
Please check out this webpage for details and requirements for the project.
Briefly, the project will consist of the following milestones:
- Checkpoint 1: Baseline Model Reproduction (12%): In the first checkpoint, you will pick a paper that you have interest in exploring from the allowed publication venues (IMPORTANT: see Project Instruction for clarification on what papers are allowed and the checkpoint requirements). Checkpoint 1 then involves (1) doing a small literature survey about the research topic of the picked paper; (2) reproducing the NLP model in the picked paper, meaning that you must get the same numbers as the picked paper on the same dataset; and (3) writing a report clearly describing the NLP problems you are exploring, as well as the reproduced model result.
- Checkpoint 2: Model Analysis and Improvement (18%): In this checkpoint, you will perform a deeper analysis and improvement upon the baseline model you have reproduced in Checkpoint 1. There are three kinds of analysis or improvement you can consider: (a) Robustness analysis: you will use an existing tool to evaluate whether the model performs robustly to challenging input variations. Accordingly, can you propose an approach to enhance the robustness of this model? (b) Multilingual NLP: you will implement a multilingual version of your baseline model. That is, say if the baseline model was only trained on English data, can you make it work on a different language (e.g., by synthesizing training data in the target language and retraining the model on it)? Do you observe equal performance of the model across different languages? (c) You are also welcome to propose your own analysis and improvement. However, your plan should be clarified in the Checkpoint 1 report for the instructor feedback. As such, Checkpoint 2 involves mainly three components: (1) model analysis and improvement; (2) report writing; and (3) final-class project presentation. (IMPORTANT: see Project Instruction for requirements on the checkpoint implementation, report, and in-class presentation.)
| Letter Grade | Points (out of 100) |
|---|---|
| A | 94-100 |
| A- | 90-93 |
| B+ | 86-89 |
| B | 83-85 |
| B- | 80-82 |
| C+ | 76-79 |
| C | 73-75 |
| C- | 70-72 |
| D | 60-69 |
| F | 0-59 |
Late Day Policy:
- For Assignments: In case there are unforeseen circumstances that don’t let you turn in your assignments on time, 5 late days total over the four assignments will be allowed without penalty. If you are late beyond the allowed late days, your assignment will be graded down one half-grade per day late (e.g., Assume your assignment receives a score of 95/100 before penalty. If you have used out your allowed late days and are now 1 more day’s late, then your final score becomes 90/100.)
- For Quizzes: No late days will be allowed. If you are late, you will receive zero grade.
- For Project Checkpoints: Any late days will cause penalty (downgraded one half-grade per day late). However, for Checkpoint 2, late days may not be feasible given the school deadline for instructors to complete grading.
- Contact the instructor IMMEDIATELY and BEFORE DUE DAYS if you know that you will very likely to be late for illness, etc. If you have any questions about the late day policy, similarly reach out to the instructor IMMEDIATELY.
Readings
For each topic/class the instructor will provide a list of papers as suggested readings. Students should be able to understand the course content just by following the lecture and by doing the readings. However, the following textbooks serve as good references.- Jurafsky and Martin, Speech and Language Processing, 3rd edition [online] (Referred to as "JM");
- Jacob Eisenstein, Natural Language Processing [online] (Referred to as "Eisenstein");
- Yoav Goldberg, Neural Network Methods in Natural Language Processing [publisher] [online primer pdf] (Referred to as "Goldberg-Publisher/Primer"); Note that the "publisher" version can be downloaded if you use the school VPN.
Tentative Schedule
We will try to cover a lot of ground in the first weeks in order to lay the foundations for the projects, but then we will focus more on specific NLP tasks and Linguistics phenomena. Pay attention to the Assignments/Quizzes/Project Checkpoints due dates. See the current deadline calendar HERE.| Date | Topic | Assignment Details | Reading Materials |
|---|---|---|---|
| 01/22 | Introduction and Class Outline; Intro to Language Modeling and Neural Network (Basics & Feedforward NN) | HW0&HW1 released | JM Ch3 (Language Modeling), Ch7 (NNs and FFNN); Eisenstein Ch2; & Goldberg-Primer Ch6.1-6.3 (FFNN); Introduction to Pytorch |
| 01/29 | ML Basics: Binary/Multiclass Classification; Word Embeddings | HW0 due 02/02 | JM Ch5 (Logistic Regression), Ch6 (Vector Semantics) Prof. Durrett's lecture note 1 & 2 ; |
| 02/05 | Contextual Representations (ELMo, Self-Attention, BERT); Attention and Transformers | Quiz1 released, due 02/11 | JM Ch10.1-10.2 (Transformers), Ch11 (MLMs and pre-training); Goldberg-Publisher Ch10.4;
Bahdanau et al. 2015 (attention); Vaswani et al., 2017 (Transformer); Vaswani et al., 2017 (Transformer); Peters et al., 2018 (ELMo); Devlin et al., 2019 (BERT) An easy-to-read blog post on Transformer language models |
| 02/12 | NN Architectures: Recurrent Neural Networks, Encoder-Decoder/Decoder-only | HW1 due 02/16; HW2 released | JM Ch9 (RNNs and LSTMs, Encoder-Decoder) Mikolov et al., 2013a & 2013b; An easy-to-read blog post on attention |
| 02/19 | Language Generation; Large Language Models (LLMs) & Scaling Laws | Quiz2 released, due 02/25 | Exploring the Limits of Transfer Learning... (Raffel et al., JMLR 2020, "T5"); Nucleus sampling (Holtzmann et al., ICLR 2020); Scaling Laws for Neural Language Models (Kaplan et al., 2020); Training Compute-Optimal Large Language Models (Hoffmann et al., 2022); PaLM: Scaling Language Modeling with Pathways (Chowdhery et al., 2022) |
| 02/26 | LLM Advancement: Prompt-based Learning, Tool Augmentation, and More | HW3 released | Pre-train, Prompt, and Predict... (Liu et al., 2021, survey paper); Prefix tuning... Prompts for Generation (Li & Liang, ACL 2021); Chain-of-Thought Prompting (Wei et al., 2022); Chameleon Plug-and-Play (Lu et al., 2023) |
| 03/04 | Spring Recess (NO CLASS) | HW2 due 03/08 | |
| 03/11 | NLP Interpretability and Explainability | LIME; e-SNLI; Hewitt&Liang'19; CheckList | |
| 03/18 | Sequence Labeling and Syntactic Parsing | HW3-Ckpt1 due 03/18; Quiz3 released, due 03/24 | JM Ch8; JM Ch12.1-12.2, 12.6, 13.1-13.4 (constituency), 14 (dependency); Chen&Manning, 2014; Dozat&Manning, 2017 |
| 03/25 | Semantic Parsing | HW3-Ckpt2 due 03/29 | Eisenstein Ch12-13; Zettlemoyer&Collins, 2005; Berant et al., 2013; Dong&Lapata, 2016; Spider (Yu et al., 2018); UnifiedSKG (Xie et al., 2022) |
| 04/01 | Machine Translation and Multilingual NLP | Eisenstein 18.1-18.2 Neural Machine Translation... with Subword Units (Sennrich et al., ACL 2016); Beyond English-centric multilingual machine translation (Fan et al., 2020); MAD-X: Multi-task cross lingual transfer (Pfeiffer et al., EMNLP 2020); Hershcovich et al., 2022; Liu et al., 2021; Bird, 2020; Lent et al., 2021; |
|
| 04/08 | Language Generation (Dialog, Summarization, etc.) | Project Checkpoint 1 due 04/15 | Holtzman et al., 2020; Ranzato et al., 2016; Maynez et al., 2020; Sellam et al., 2020; See et al., 2017 |
| 04/15 | Question Answering and Dataset Biases | Quiz4 released, due 04/21 | JM Ch23; ACL20 tutorial QA over text: Chen et al., 2017 (DrQA); Lee et al., 2019 (ORQA); Zhu et al., 2021 (survey); QA over structured data:Pasupat&Liang, 2015 (Table QA); Yih et al., 2015 (KBQA); Rajpurkar et al., 2016 |
| 04/22 | Ethical Concerns in NLP; Wrap-up |
Interactivity: Wang et al., 2016; Hancock et al., 2019; guidelines for human-AI interaction; InstructGPT&RLHF; Ethics: Zhao et al., 2017; Rudinger et al., 2018; Gebru et al., 2018 |
|
| 04/29 | Project Presentation | Final Project Report due 05/03 |
Honor Code
The class enforces the GMU Honor Code, and the more specific honor code policy special to the Department of Computer Science. You will be expected to adhere to this code and policy.- IMPORTANT NOTE about the use of ChatGPT:Unless permission to do so is granted by the instructor, you (or your group, if a group assignment) may not use assistive technology, artificial intelligence, or other tools to complete assignments which can generate, translate, or otherwise create/correct code or answers (many types of assistive technology may be permitted, but you must ask permission).