Syllabus
CS 678
Advanced Natural Language Processing
Instructors
Kenton Murray (kmurra23 [at] gmu [dot] edu)Office Hours: Mondays 2:00-3:00 ENGR 4708 or Email for appointments.
TA
Primary TA: Daking Rai (drai2 [at] gmu [dot] edu)Office Hours: Thursday 11:00-1:00, ENGR 4456 D8
Meets
Monday, 4:30 pm - 7:10 pm, Exploratory Hall L111.Course Web Page
https://nlp.cs.gmu.edu/course/cs678-fall26/.We will use Canvas and Piazza for Q&A (sign up link: https://piazza.com/gmu/fall2026/cs678).
Course Description
Massive amounts of information in our daily life are expressed in natural language. In this class, we will study building computing systems that can process, understand, and communicate in natural language. The class will start with an introduction to the foundations of natural language processing (NLP), and then focus on cutting-edge research problems in NLP. Each section will introduce a particular problem or phenomenon in natural language, describe why it is difficult to model, and demonstrate recent models that were designed to tackle this problem. In the process of doing so, the class will cover different techniques that are useful in creating and applying neural network models. The class will include assignments, short quizzes, and a final project.Learning Outcomes
- Be familiar fundamental NLP problems and the methods powering the current state-of-the-art in language technologies, such as large pre-trained neural language models;
- Understand the limitations of current technologies and be able to make informed decisions about addressing them using advanced machine learning techniques;
- Design, implement, and evaluate a computing-based solution to address a language-related problem using state-of-the-art tools.
Prerequisites
CS 580 (Intro to AI) or CS 584 (Data Mining). You should be proficient in (a) Algorithms and Data Structures and (b) Probability and Statistics (STAT 344) or equivalent. Students should be experienced with writing substantial programs in Python. Please contact the instructor if you have questions about the necessary background.Class Format
The class will be in-person. As the class aims to provide skills necessary to familiarize the students with, and to do cutting-edge NLP research, the classes and assignments will be at least partially implementation-focused. In general, each class will take the following format:- Reading: Before some lectures, you will be pointed to some reading materials (see "Reading Materials" in course schedule) that you should read before coming to class that day.
- Summary/Elaboration/Questions: The instructor will summarize the important points of the reading material, elaborate on details that were not included in the reading while fielding any questions. Finally, new material on cutting-edge methods, or a deep look into one salient method will be covered.
- Demo/Code Walk: In some classes we will walk through some demonstration code that implements a simple version of the main concepts presented in the reading material.
Grading
There will be no midterm or final exam. Your final grade will be dependent on:Class participation (25%): (1) There will be 3 in-class quizzes throughout the semester, comprising 15% of the total grade. Each quiz will ask you 3-4 short-answer questions, designed to evaluate if you understand the lecture and homework. (2) You will review in-class presentations from your classmates on their projects checkpoint 1 (10%). We will assign you to peer-review 3 random groups. You are expected to provide construct feedback to the assigned groups.
Homework assignments (25%):
- HW1: Preliminaries and n-gram language modeling (8%)
- HW2: Post-training an open-weight language model (9%)
- HW3: Implementing a tool-augmented LLM agent (8%)
Project (50%): The bulk of your grade will be based on a group research project related to the topics we will discuss in class. The groups will be of 2-3 people. If you intend to complete the project independently or with more than 3 people in your group, you MUST ask for permission from the instructor.
Briefly, the project will consist of the following milestones (refer to Project Instruction for requirements on the checkpoint implementation, report, and in-class presentation):
- Checkpoint 1: Project Kickstart (20%): By the first checkpoint, you will make a team, decide a topic, perform literature study, collect datasets, and outline a feasible plan for the project. This checkpoint also involves an in-class presentation, which is mandatory for all team members.
- Checkpoint 2: Final Project Submission (30%): You are expected to finish the project development, evaluation, and report. This checkpoint also involves an in-class presentation, which is mandatory for all team members.
| Letter Grade | Points (out of 100) |
|---|---|
| A | 94-100 |
| A- | 90-93 |
| B+ | 86-89 |
| B | 83-85 |
| B- | 80-82 |
| C+ | 76-79 |
| C | 73-75 |
| C- | 70-72 |
| D | 60-69 |
| F | 0-59 |
Late Day Policy:
In case there are unforeseen circumstances that don’t let you turn in your work on time, 5 late days total over the assignments and project checkpoint 1 will be allowed without penalty. If you are late beyond the allowed late days, your assignment or project checkpoint 1 will be graded down one half-grade per day late (e.g., Assume your assignment receives a score of 95/100 before penalty. If you have used out your allowed late days and are now 1 more day’s late, then your final score becomes 90/100.)
For project checkpoint 2 (final report submission), no late days are allowed. All quizzes and in-class activities (presentations and peer-review) need to be completed in class and there is no lateness allowed. As a result, for project checkpoint 1, even if you can apply the allowable late days to the report submission, you still need to make the in-class presentation on time.
Contact the instructor IMMEDIATELY and BEFORE DUE DAYS if you know that you will very likely to be late beyond the allowable late days due to illness or other incidents. If you have any questions about the late day policy, similarly reach out to the instructor IMMEDIATELY.
Readings
For each topic/class the instructor will provide a list of papers as suggested readings. Students should be able to understand the course content just by following the lecture and by doing the readings. We will mainly be reading rcently published papers in the field. All of these will be available online for free. We will also make use of the following textbooks.- Jurafsky and Martin, Speech and Language Processing, 3rd edition [online; January 12, 2025 release] (Referred to as "JM");
- Jacob Eisenstein, Natural Language Processing [online] (Referred to as "Eisenstein");
- Yoav Goldberg, Neural Network Methods in Natural Language Processing [publisher] [online primer pdf] (Referred to as "Goldberg-Publisher/Primer"); Note that the "publisher" version can be downloaded if you use the school VPN.
Tentative Schedule
| Date | Topic | Assignment Details | Reading Materials |
|---|---|---|---|
| 08/24 | Introduction and ML/NN Basics | JM Ch4.7, Ch5 (Logistic Regression); Eisenstein Ch2 | |
| 08/31 | Language Modeling and Tokenization; Neural Language Modeling and Word Embedding | An Empirical Study of Smoothing Techniques for Language Modeling; JM Ch3 (Language Modeling), Ch7 (FFNN); Goldberg-Primer Ch6.1-6.3 (FFNN); Andrej Karpathy's tokenization course | |
| 09/07 | No Class (Labor Day) | HW1 Out | |
| 09/14 | Interpretability (Guest Lectures) | ||
| 09/21 | NN Architectures: Seq2Seq, Attention, and Transformers | HW1 Due; HW2 Out | The Annotated Transformer; JM Ch8 (Enc-Dec), Ch9 (Transformers); Eisenstein Ch18.3.1; "understand LSTM" blog by Olah; "gradient vanishing" blog by Nielsen; Attention-based NMT; Transformer paper and Alammar's blog |
| 09/28 | Large Language Models (LLMs) | JM Ch10-11; BERT; T5; Scaling Laws for Neural Language Models (Kaplan et al., 2020) | |
| 10/05 | LLM Advancement 1: Prompting and Alignment (Post-Training) | JM Ch12; Pre-train, Prompt, and Predict... (Liu et al., 2021); Chain-of-Thought Prompting (Wei et al., 2022) | |
| 10/12 | No Class (Fall Break) | ||
| 10/19 | LLM Advancement 2: LLM Interpretability | LIME; EMNLP20 tutorial; Mechanistic Interpretability (Rai et al. 2024) | |
| 10/26 | LLM Advancement 3: LLM Agents | HW2 Due; HW3 Out | Lilian Weng's blog; Stanford Smallville |
| 11/02 | Linguistic Annotation: Syntactic and Semantic Parsing | JM Ch17.1-17.3 (POS tagging), 18.1-18.2 (const par), 19.1 (dep par), Appendix F (sem par) | |
| 11/09 | Project Checkpoint 1 Presentation | Project Ckpt 1 Report Due 11/10, Peer-Review Feedback Due 11/11 | |
| 11/16 | Applied NLP 1: Chatbot and Dialogue Systems | JM Ch15 | |
| 11/23 | Applied NLP 2: Question Answering, Info Retrieval, and RAG | HW3 Due | JM Ch14; ACL20 tutorial |
| 11/30 | Applied NLP 3: AI/LLM Safety | OWASP's Top10 LLM Risks | |
| 12/07 | Final Project Presentation | Project Checkpoint 2 Due on 12/14 |
Honor Code
The class enforces the GMU Honor Code, and the more specific honor code policy special to the Department of Computer Science. You will be expected to adhere to this code and policy.- IMPORTANT NOTE about the use of AI: Unless permission to do so is granted by the instructor, you (or your group, if a group assignment) may not use assistive technology, artificial intelligence, or other tools to complete assignments which can generate, translate, or otherwise create/correct code or answers (many types of assistive technology may be permitted, but you must ask permission).