Syllabus

CS 678

Advanced Natural Language Processing

Instructors

Ziyu Yao (ziyuyao [at] gmu [dot] edu)
Office Hours: Email for appointments.

TA

Primary TA: Saurabh Srivastava (ssrivas6 [at] gmu [dot] edu)
Office Hours: TBD
Secondary TA: Hao Yan (hyan5 [at] gmu [dot] edu)
Office Hours: TBD

Meets

Tuesday, 4:30 pm - 7:10 pm, Horizon Hall 1014.

Course Web Page

https://nlp.cs.gmu.edu/course/cs678-fall25/.
We will use Canvas and Gradescope for course materials/assignments/grading, and Piazza for Q&A (sign up link: https://piazza.com/gmu/fall2025/cs678).

Course Description

Massive amounts of information in our daily life are expressed in natural language. In this class, we will study building computing systems that can process, understand, and communicate in natural language. The class will start with an introduction to the foundations of natural language processing (NLP), and then focus on cutting-edge research problems in NLP. Each section will introduce a particular problem or phenomenon in natural language, describe why it is difficult to model, and demonstrate recent models that were designed to tackle this problem. In the process of doing so, the class will cover different techniques that are useful in creating and applying neural network models. The class will include assignments, short quizzes, and a final project.

Learning Outcomes

  • Be familiar fundamental NLP problems and the methods powering the current state-of-the-art in language technologies, such as large pre-trained neural language models;
  • Understand the limitations of current technologies and be able to make informed decisions about addressing them using advanced machine learning techniques;
  • Design, implement, and evaluate a computing-based solution to address a language-related problem using state-of-the-art tools.

Prerequisites

CS 580 (Intro to AI) or CS 584 (Data Mining). You should be proficient in (a) Algorithms and Data Structures and (b) Probability and Statistics (STAT 344) or equivalent. Students should be experienced with writing substantial programs in Python. Please contact the instructor if you have questions about the necessary background.

Class Format

The class will be in-person. As the class aims to provide skills necessary to familiarize the students with, and to do cutting-edge NLP research, the classes and assignments will be at least partially implementation-focused. In general, each class will take the following format:
  • Reading: Before some lectures, you will be pointed to some reading materials (see "Reading Materials" in course schedule) that you should read before coming to class that day.
  • Summary/Elaboration/Questions: The instructor will summarize the important points of the reading material, elaborate on details that were not included in the reading while fielding any questions. Finally, new material on cutting-edge methods, or a deep look into one salient method will be covered.
  • Demo/Code Walk: In some classes we will walk through some demonstration code that implements a simple version of the main concepts presented in the reading material.

Grading

There will be no midterm or final exam. Your final grade will be dependent on:

Class participation (20%): (1) There will be 4 in-class quizzes throughout the semester, comprising 15% of the total grade. Each quiz will ask you 3-4 short-answer questions, designed to evaluate if you understand the lecture contents. (2) You will review in-class presentations from your classmates for project checkpoint 1 (5%). We will assign you to peer-review 3 random groups. You are expected to provide construct feedback to the assigned groups.

Homework assignments (50%):

  • HW1: Preliminaries and n-gram language modeling (10%)
  • HW2: Post-training an open-weight language model (20%)
  • HW3: Implementing a tool-augmented LLM agent (20%)

Project (30%): The bulk of your grade will be based on a group research project related to the topics we will discuss in class. The groups will be of 2-3 people. If you intend to complete the project independently or with more than 3 people in your group, you MUST ask for permission from the instructor.
Briefly, the project will consist of the following milestones (refer to Project Instruction for requirements on the checkpoint implementation, report, and in-class presentation):

  • Checkpoint 1: Project Kickstart (10%): By the first checkpoint, you will make a team, decide a topic, perform literature study, collect datasets, and outline a feasible plan for the project. This checkpoint also involves an in-class presentation, which is mandatory for all team members.
  • Checkpoint 2: Final Project Submission (20%): You are expected to finish the project development, evaluation, and report. This checkpoint also involves an in-class presentation, which is mandatory for all team members.
Letter Grade Points (out of 100)
A 94-100
A- 90-93
B+ 86-89
B 83-85
B- 80-82
C+ 76-79
C 73-75
C- 70-72
D 60-69
F 0-59

Late Day Policy:
In case there are unforeseen circumstances that don’t let you turn in your work on time, 5 late days total over the assignments and project checkpoint 1 will be allowed without penalty. If you are late beyond the allowed late days, your assignment or project checkpoint 1 will be graded down one half-grade per day late (e.g., Assume your assignment receives a score of 95/100 before penalty. If you have used out your allowed late days and are now 1 more day’s late, then your final score becomes 90/100.)

For project checkpoint 2 (final report submission), no late days are allowed. All quizzes and in-class activities (presentations and peer-review) need to be completed in class and there is no lateness allowed. As a result, for project checkpoint 1, even if you can apply the allowable late days to the report submission, you still need to make the in-class presentation on time.

Contact the instructor IMMEDIATELY and BEFORE DUE DAYS if you know that you will very likely to be late beyond the allowable late days due to illness or other incidents. If you have any questions about the late day policy, similarly reach out to the instructor IMMEDIATELY.

Readings

For each topic/class the instructor will provide a list of papers as suggested readings. Students should be able to understand the course content just by following the lecture and by doing the readings. We will mainly use the following textbooks.
  • Jurafsky and Martin, Speech and Language Processing, 3rd edition [online; January 12, 2025 release] (Referred to as "JM");
  • Jacob Eisenstein, Natural Language Processing [online] (Referred to as "Eisenstein");
  • Yoav Goldberg, Neural Network Methods in Natural Language Processing [publisher] [online primer pdf] (Referred to as "Goldberg-Publisher/Primer"); Note that the "publisher" version can be downloaded if you use the school VPN.

Tentative Schedule

Date Topic Assignment Details Reading Materials
08/26 Introduction and ML/NN Basics JM Ch4.7, Ch5 (Logistic Regression); Eisenstein Ch2
09/02 Language Modeling and Tokenization HW1 Out JM Ch3 (Language Modeling), Ch7 (FFNN); Goldberg-Primer Ch6.1-6.3 (FFNN); Andrej Karpathy's tokenization course
09/09 Neural Language Modeling and Word Embedding JM Ch8 (RNN, LSTMs), Ch6 (Vector Semantics); Eisenstein Ch6; Mikolov et al., 2013a&b
09/16 NN Architectures: Seq2Seq, Attention, and Transformers HW1 Due; HW2 Out JM Ch8 (Enc-Dec), Ch9 (Transformers); Eisenstein Ch18.3.1; "understand LSTM" blog by Olah; "gradient vanishing" blog by Nielsen; Attention-based NMT; Transformer paper and Alammar's blog
09/23 Large Language Models (LLMs) JM Ch10-11; BERT; T5; Scaling Laws for Neural Language Models (Kaplan et al., 2020)
09/30 LLM Advancement 1: Prompting and Alignment (Post-Training) JM Ch12; Pre-train, Prompt, and Predict... (Liu et al., 2021); Chain-of-Thought Prompting (Wei et al., 2022)
10/07 LLM Advancement 2: LLM Interpretability LIME; EMNLP20 tutorial; Mechanistic Interpretability (Rai et al. 2024)
10/14 LLM Advancement 3: LLM Agents HW2 Due; HW3 Out Lilian Weng's blog; Stanford Smallville
10/21 Linguistic Annotation: Syntactic and Semantic Parsing JM Ch17.1-17.3 (POS tagging), 18.1-18.2 (const par), 19.1 (dep par), Appendix F (sem par)
10/28 Project Checkpoint 1 Presentation Project Ckpt 1 Report Due 10/28, Peer-Review Feedback Due 10/31
11/04 No Class (Election Day)
11/11 Applied NLP 1: Chatbot and Dialogue Systems JM Ch15
11/18 Applied NLP 2: Question Answering, Info Retrieval, and RAG HW3 Due JM Ch14; ACL20 tutorial
11/25 Applied NLP 3: AI/LLM Safety OWASP's Top10 LLM Risks
12/02 Final Project Presentation Project Checkpoint 2 Due on 12/09

Honor Code

The class enforces the GMU Honor Code, and the more specific honor code policy special to the Department of Computer Science. You will be expected to adhere to this code and policy.
  • IMPORTANT NOTE about the use of ChatGPT: Unless permission to do so is granted by the instructor, you (or your group, if a group assignment) may not use assistive technology, artificial intelligence, or other tools to complete assignments which can generate, translate, or otherwise create/correct code or answers (many types of assistive technology may be permitted, but you must ask permission).

Note to Students

Take care of yourself! As a student, you may experience a range of challenges that can interfere with learning, such as strained relationships, increased anxiety, substance use, global pandemics, feeling down, difficulty concentrating and/or lack of motivation. All of us benefit from support during times of struggle. There are many helpful resources available on campus and an important part of having a healthy life is learning how to ask for help. Asking for support sooner rather than later is almost always helpful. GMU services are available, and treatment does work. You can learn more about confidential mental health services available on campus at: https://caps.gmu.edu/. Support is always available (24/7) from Counseling and Psychological Services: 703-527-4077.

Disabilities

If you have a documented learning disability or other condition which may affect academic performance, make sure this documentation is on file with the Office of Disability Services and come talk to me about accommodations. I will work with you to ensure that accommodations are provided as appropriate. If you suspect that you may have a disability and would benefit from accommodations but are not yet registered with the Office of Disability Services, I encourage you to contact them at ods@gmu.edu.
Next