Syllabus

CS 795

Interpretability and Explainability of LLMs

Instructor

Ziyu Yao (ziyuyao [at] gmu [dot] edu)
Office Hours: Email for appointments.

TA

Daking Rai (drai2 [at] gmu [dot] edu)
Office Hour: Every Thursday, from 11:00 AM - 1:00 PM Location: ENGR 4456, D8

Meets

Wednesday, 4:30 pm – 7:10 pm, Innovation Hall 133.

Course Web Page

https://nlp.cs.gmu.edu/course/cs795-fall26/.
We will use Canvas and Gradescope for course materials/assignments/grading, and Piazza for Q&A (sign-up link).

Course Description

In this seminar course, we will study the cutting-edge topic of LLM interpretability and explainability. The course will cover different subtopics under mechanistic interpretability of LLMs, ranging from probing and feature analysis, superposition and sparse autoencoders, causal interventions and circuit analysis, to activation steering, the faithfulness of model explanations, and interpretability for AI safety. The format consists of weekly paper presentations and discussions, plus a team research project.

Learning Outcomes

  • Be familiar with the major research directions, methods, and open problems in LLM interpretability and explainability;
  • Critically read, evaluate, and present cutting-edge research papers, including assessing their strengths, weaknesses, reproducibility, and impact;
  • Design, implement, and evaluate an original research project on the interpretability or explainability of LLMs.

Prerequisites

No official prerequisites, but students are expected to be comfortable with deep learning and NLP fundamentals and to be proficient in Python. CS 678 (Advanced NLP) is strongly recommended. Please contact the instructor if you have questions about the necessary background.

Class Format

The class will be in-person and discussion-driven. Starting from the 4th class, each meeting will center on the discussion of two papers.
  • Reading and review: All students read the assigned papers before class and submit a short review of each paper before the start of that class (see "Grading" below).
  • Role-play presentation: A team of students presents each paper in assigned roles (see "Role-Play Paper Presentation" below).
  • Discussion: Everyone else is expected to critically engage — asking questions, supporting or challenging the presenters' arguments, and sharing comments.

Grading

There will be no midterm or final exam. Your final grade will be dependent on:

Class participation (55%): Students are required to attend every class, unless there is a valid excuse and an early notification to the instructor. Starting from the 4th class, we will begin paper discussions, two papers in each class.

  • Paper review (10%): Students are required to submit a review of the papers before the start of the class (11:59pm the day before the class). The review does not need to be long; a short paragraph describing what the paper does and a few points of comments/questions would be good.
  • "Role-play" paper presentation (40%): Each student is expected to participate in (tentatively) 4 papers' role-play presentations throughout the semester. Students will be assigned to papers based on their interest; they will be allocated to different role types by the instructor.
  • General in-class participation (5%): Students who are not presenting are expected to critically engage in paper discussions, including asking questions, offering comments, discussing future work, etc. This evaluation will be subjectively provided by the instructor based on her impression of everyone.

Course project (45%): Students work in groups of 2–3 on a project focusing on interpretability and explainability of LLMs. Note that projects that only shallowly involve LLM interpretability/explainability do not qualify. Students should use what they have learned from this class in their projects.

  • Project proposal (15%): Students submit a report describing their project idea, datasets involved, preliminary methodology design, experimental plan, etc. There will be a mandatory in-class presentation in the middle of the semester.
  • Final project report (30%): Students submit a final report, expanding from their proposal report, describing their finalized project idea, datasets involved, methodology design, experimental plan, experimental results, discussions, and future work. There will be a mandatory in-class presentation at the end of the semester.
Letter Grade Points (out of 100)
A 94-100
A- 90-93
B+ 86-89
B 83-85
B- 80-82
C+ 76-79
C 73-75
C- 70-72
D 60-69
F 0-59

Late Policy: Paper reviews and any in-class components (paper presentations, project presentations) must be completed on time and cannot be made up. For written project reports, 3 late days in total will be allowed without penalty. Extra lateness will incur point deductions: actual grade = raw grade - 5 x N for N extra late days (grades in 1-100 scale). The final project report cannot be submitted later than Dec 13.

Contact the instructor IMMEDIATELY and BEFORE DUE DAYS if you know that you will very likely miss a class or a deadline due to illness or other incidents.

Role-Play Paper Presentation

The paper presentations will follow a "role playing" format. Specifically, a team of students will collaboratively present one paper in different roles:
Role # of Student(s) Responsibility Presentation Requirement (Time)
Background Investigator 1 Provides *necessary context* about the paper, including its related work, especially work that served as a foundation for the current one, and justifies its motivation and historical significance. Presents an overview of related work and a justification of the current work's motivation and significance. If any paper(s) served as a foundation for the current work, you should give details about one of them. (10 min)
Paper Reporter 1 You are the main presenter of this paper. Presents the paper in detail and helps the audience understand what it does and what it results in. Presents the current paper. Considering the time limit, it is okay to selectively present only the critical parts of the paper rather than its full details, but your presentation should give the "review committee" (see the role description of Scientific Peer Reviewer) and the audience sufficient "meat". (25 min)
Scientific Peer Reviewer 1 Imagine this paper is submitted to a top AI/NLP venue and you are serving on the review committee deciding if this paper is good enough to be accepted. Analyzes the paper and helps the committee chair understand its strengths, weaknesses, and potential (positive or negative) impacts. Presents at least 3 pros and at least 3 cons of the current paper, sorted by their severity. Discusses the impact of the work (connecting to Background Investigation). (10 min)
Forward Looker 1 Discusses (a) how this work may change (or has been changing) the research landscape along the investigated and relevant topics and (b) how future work can further improve this work (with respect to the weaknesses discovered by the Peer Reviewer or the audience). Presents your vision and initiates the discussion. Gives an overview or a highlight of the citing papers. (10 min)

Everyone else is expected to have critical engagement in the paper discussion — asking questions, supporting or challenging the presenter's argument, sharing comments, etc.

Tentative Schedule

Date Class Theme Readings and Materials
08/26 1 Introduction to CS795; Backgrounds of Language Models
09/02 2 Fundamentals of LLMs
09/09 3 Overview of Model Interpretability/Explainability A Practical Review of Mechanistic Interpretability for Transformer-Based Language Models [Paper][ICML Tutorial]
09/16 4 Probing and feature analysis Main Papers:
  • Language Models Represent Space and Time [paper] [code]
  • Finding Neurons in a Haystack: Case Studies with Sparse Probing [paper] [code]
Supplementary Reading:
  • Probing Classifiers: Promises, Shortcomings, and Advances [paper]
  • A Structural Probe for Finding Syntax in Word Representations [paper]
  • The Linear Representation Hypothesis and the Geometry of Large Language Models [paper]
09/23 5 Superposition and sparse autoencoders Main Papers:
  • Sparse Autoencoders Find Highly Interpretable Features in Language Models [paper] [code]
  • Are Sparse Autoencoders Useful? A Case Study in Sparse Probing [paper] [code]
Supplementary Reading:
  • Toy Models of Superposition [paper]
  • Towards Monosemanticity: Decomposing Language Models With Dictionary Learning [paper]
  • SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability [paper]
  • Transcoders Find Interpretable LLM Feature Circuits [paper]
09/30 6 Describing features and representations Main Papers:
  • Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models [paper] [code]
  • Activation Oracles [paper]
Supplementary Reading:
  • Language Model Inversion [paper]
  • SelfIE: Self-Interpretation of Large Language Model Embeddings [paper]
  • LatentQA: Teaching LLMs to Decode Activations Into Natural Language [paper]
  • Automatically Interpreting Millions of Features in Large Language Models [paper] [code]
10/07 7 Course project proposal presentation Mandatory in-class presentation for all team members.
Written proposal report submission due on 10/11.
10/14 8 Causal interventions and abstractions Main Papers:
  • Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias [paper] [code]
  • Causal Abstractions of Neural Networks [paper] [code] [tool]
Supplementary Reading:
  • Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representations [paper]
10/21 9 Circuit analysis Main Papers:
  • Interpretability in the Wild: A Circuit for Indirect Object Identification in GPT-2 Small [paper] [code]
  • Locating and Editing Factual Associations in GPT [paper] [code]
Supplementary Reading:
  • In-context Learning and Induction Heads [paper]
  • Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristics [paper]
  • Data-driven Circuit Discovery for Interpretability of Language Models (DCD) [paper]
  • Transcoders Find Interpretable LLM Feature Circuits [paper]
10/28 10 Steering and activation engineering Main Papers:
  • Steering Llama 2 via Contrastive Activation Addition [paper] [code]
  • Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior [paper] [code]
Supplementary Reading:
  • Inference-Time Intervention: Eliciting Truthful Answers from a Language Model [paper]
  • Failure by Interference: Language Models Make Balanced Parentheses Errors When Faulty Mechanisms Overshadow Sound Ones [paper]
  • Representation Engineering: A Top-Down Approach to AI Transparency [paper]
  • Function Vectors in Large Language Models [paper]
  • Steering off Course: Reliability Challenges in Steering Language Models [paper]
11/04 11 Faithfulness of explanation Main Papers:
  • Measuring Faithfulness in Chain-of-Thought Reasoning [paper] [code: NA]
  • Measuring Chain of Thought Faithfulness by Unlearning Reasoning Steps [paper] [code]
Supplementary Reading:
  • Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting [paper]
  • Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety [paper]
11/11 12 Interpreting LLM training Main Papers:
  • Progress Measures for Grokking via Mechanistic Interpretability [paper] [code]
  • Mechanistic Data Attribution: Tracing the Training Origins of Interpretable LLM Units [paper] [code]
Supplementary Reading:
  • LLM Circuit Analyses Are Consistent Across Training and Scale [paper]
  • Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalization [paper]
  • Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Tracking [paper]
11/18 13 Interpretability for LLM safety Main Papers:
  • Persona Vectors: Monitoring and Controlling Character Traits in Language Models [paper] [code]
  • Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models [paper] [code]
Supplementary Reading:
  • LLMs Encode Harmfulness and Refusal Separately [paper]
  • Representation Engineering: A Top-Down Approach to AI Transparency [paper]
11/25 – No Class (Thanksgiving recess)
12/02 14 Course project presentation Mandatory in-class presentation for all team members.
Written final report submission due on 12/09.

Other Resources

Academic Standards and the Use of AI

The class enforces the GMU Academic Standards, including a specification to the use of Generative AI tools. In this class, AI tools are forbidden from direct use for your submission writing; the paper reviews and the project proposals/reports must be written by you (except trivial grammar fixes and language refinement). However, you are encouraged to use AI tools for study or assisting purposes, such as using them to understand concepts in papers, brainstorm project ideas, evaluate project reports for self-improvement, and coding for the course project. Note that it should be you doing the homework and that you are fully responsible for everything you submit, and you must ensure that all content is correct and original. If you have any questions about this policy, please reach out to the instructor.

Note to Students

Take care of yourself! As a student, you may experience a range of challenges that can interfere with learning, such as strained relationships, increased anxiety, substance use, global pandemics, feeling down, difficulty concentrating and/or lack of motivation. All of us benefit from support during times of struggle. There are many helpful resources available on campus and an important part of having a healthy life is learning how to ask for help. Asking for support sooner rather than later is almost always helpful. GMU services are available, and treatment does work. You can learn more about confidential mental health services available on campus at: https://caps.gmu.edu/. Support is always available (24/7) from Counseling and Psychological Services: 703-527-4077.

Disabilities

If you have a documented learning disability or other condition which may affect academic performance, make sure this documentation is on file with the Office of Disability Services and come talk to me about accommodations. I will work with you to ensure that accommodations are provided as appropriate. If you suspect that you may have a disability and would benefit from accommodations but are not yet registered with the Office of Disability Services, I encourage you to contact them at ods@gmu.edu.