Syllabus
CS 795
Interpretability and Explainability of LLMs
Instructor
Ziyu Yao (ziyuyao [at] gmu [dot] edu)Office Hours: Email for appointments.
TA
Daking Rai (drai2 [at] gmu [dot] edu)Office Hour: Every Thursday, from 11:00 AM - 1:00 PM Location: ENGR 4456, D8
Meets
Wednesday, 4:30 pm – 7:10 pm, Innovation Hall 133.Course Web Page
https://nlp.cs.gmu.edu/course/cs795-fall26/.We will use Canvas and Gradescope for course materials/assignments/grading, and Piazza for Q&A (sign-up link).
Course Description
In this seminar course, we will study the cutting-edge topic of LLM interpretability and explainability. The course will cover different subtopics under mechanistic interpretability of LLMs, ranging from probing and feature analysis, superposition and sparse autoencoders, causal interventions and circuit analysis, to activation steering, the faithfulness of model explanations, and interpretability for AI safety. The format consists of weekly paper presentations and discussions, plus a team research project.Learning Outcomes
- Be familiar with the major research directions, methods, and open problems in LLM interpretability and explainability;
- Critically read, evaluate, and present cutting-edge research papers, including assessing their strengths, weaknesses, reproducibility, and impact;
- Design, implement, and evaluate an original research project on the interpretability or explainability of LLMs.
Prerequisites
No official prerequisites, but students are expected to be comfortable with deep learning and NLP fundamentals and to be proficient in Python. CS 678 (Advanced NLP) is strongly recommended. Please contact the instructor if you have questions about the necessary background.Class Format
The class will be in-person and discussion-driven. Starting from the 4th class, each meeting will center on the discussion of two papers.- Reading and review: All students read the assigned papers before class and submit a short review of each paper before the start of that class (see "Grading" below).
- Role-play presentation: A team of students presents each paper in assigned roles (see "Role-Play Paper Presentation" below).
- Discussion: Everyone else is expected to critically engage — asking questions, supporting or challenging the presenters' arguments, and sharing comments.
Grading
There will be no midterm or final exam. Your final grade will be dependent on:Class participation (55%): Students are required to attend every class, unless there is a valid excuse and an early notification to the instructor. Starting from the 4th class, we will begin paper discussions, two papers in each class.
- Paper review (10%): Students are required to submit a review of the papers before the start of the class (11:59pm the day before the class). The review does not need to be long; a short paragraph describing what the paper does and a few points of comments/questions would be good.
- "Role-play" paper presentation (40%): Each student is expected to participate in (tentatively) 4 papers' role-play presentations throughout the semester. Students will be assigned to papers based on their interest; they will be allocated to different role types by the instructor.
- General in-class participation (5%): Students who are not presenting are expected to critically engage in paper discussions, including asking questions, offering comments, discussing future work, etc. This evaluation will be subjectively provided by the instructor based on her impression of everyone.
Course project (45%): Students work in groups of 2–3 on a project focusing on interpretability and explainability of LLMs. Note that projects that only shallowly involve LLM interpretability/explainability do not qualify. Students should use what they have learned from this class in their projects.
- Project proposal (15%): Students submit a report describing their project idea, datasets involved, preliminary methodology design, experimental plan, etc. There will be a mandatory in-class presentation in the middle of the semester.
- Final project report (30%): Students submit a final report, expanding from their proposal report, describing their finalized project idea, datasets involved, methodology design, experimental plan, experimental results, discussions, and future work. There will be a mandatory in-class presentation at the end of the semester.
| Letter Grade | Points (out of 100) |
|---|---|
| A | 94-100 |
| A- | 90-93 |
| B+ | 86-89 |
| B | 83-85 |
| B- | 80-82 |
| C+ | 76-79 |
| C | 73-75 |
| C- | 70-72 |
| D | 60-69 |
| F | 0-59 |
Late Policy: Paper reviews and any in-class components (paper presentations, project presentations) must be completed on time and cannot be made up. For written project reports, 3 late days in total will be allowed without penalty. Extra lateness will incur point deductions: actual grade = raw grade - 5 x N for N extra late days (grades in 1-100 scale). The final project report cannot be submitted later than Dec 13.
Contact the instructor IMMEDIATELY and BEFORE DUE DAYS if you know that you will very likely miss a class or a deadline due to illness or other incidents.
Role-Play Paper Presentation
The paper presentations will follow a "role playing" format. Specifically, a team of students will collaboratively present one paper in different roles:| Role | # of Student(s) | Responsibility | Presentation Requirement (Time) |
|---|---|---|---|
| Background Investigator | 1 | Provides *necessary context* about the paper, including its related work, especially work that served as a foundation for the current one, and justifies its motivation and historical significance. | Presents an overview of related work and a justification of the current work's motivation and significance. If any paper(s) served as a foundation for the current work, you should give details about one of them. (10 min) |
| Paper Reporter | 1 | You are the main presenter of this paper. Presents the paper in detail and helps the audience understand what it does and what it results in. | Presents the current paper. Considering the time limit, it is okay to selectively present only the critical parts of the paper rather than its full details, but your presentation should give the "review committee" (see the role description of Scientific Peer Reviewer) and the audience sufficient "meat". (25 min) |
| Scientific Peer Reviewer | 1 | Imagine this paper is submitted to a top AI/NLP venue and you are serving on the review committee deciding if this paper is good enough to be accepted. Analyzes the paper and helps the committee chair understand its strengths, weaknesses, and potential (positive or negative) impacts. | Presents at least 3 pros and at least 3 cons of the current paper, sorted by their severity. Discusses the impact of the work (connecting to Background Investigation). (10 min) |
| Forward Looker | 1 | Discusses (a) how this work may change (or has been changing) the research landscape along the investigated and relevant topics and (b) how future work can further improve this work (with respect to the weaknesses discovered by the Peer Reviewer or the audience). | Presents your vision and initiates the discussion. Gives an overview or a highlight of the citing papers. (10 min) |
Everyone else is expected to have critical engagement in the paper discussion — asking questions, supporting or challenging the presenter's argument, sharing comments, etc.
Tentative Schedule
| Date | Class | Theme | Readings and Materials |
|---|---|---|---|
| 08/26 | 1 | Introduction to CS795; Backgrounds of Language Models | |
| 09/02 | 2 | Fundamentals of LLMs | |
| 09/09 | 3 | Overview of Model Interpretability/Explainability | A Practical Review of Mechanistic Interpretability for Transformer-Based Language Models [Paper][ICML Tutorial] |
| 09/16 | 4 | Probing and feature analysis | Main Papers: Supplementary Reading: |
| 09/23 | 5 | Superposition and sparse autoencoders | Main Papers: Supplementary Reading: |
| 09/30 | 6 | Describing features and representations | Main Papers: Supplementary Reading: |
| 10/07 | 7 | Course project proposal presentation | Mandatory in-class presentation for all team members. Written proposal report submission due on 10/11. |
| 10/14 | 8 | Causal interventions and abstractions | Main Papers:
|
| 10/21 | 9 | Circuit analysis | Main Papers: Supplementary Reading: |
| 10/28 | 10 | Steering and activation engineering | Main Papers:
|
| 11/04 | 11 | Faithfulness of explanation | Main Papers: Supplementary Reading: |
| 11/11 | 12 | Interpreting LLM training | Main Papers: Supplementary Reading: |
| 11/18 | 13 | Interpretability for LLM safety | Main Papers: Supplementary Reading: |
| 11/25 | – | No Class (Thanksgiving recess) | |
| 12/02 | 14 | Course project presentation | Mandatory in-class presentation for all team members. Written final report submission due on 12/09. |
Other Resources
- LLM Interpretability course materials (Tel Aviv University)
- CMSC848R: Selected Topics in Information Processing; Language Model Interpretability (UMD)