CS 678 Course Project
Advanced Natural Language Processing
Students will form groups (each with 2-3 people) for the course project. There will be in total two submissions toward completing the project: Project Kickstart and Final Project Submission.
Project Topics
We expect student projects to fall into roughly the following two categories:- NLP Application: Students are expected to develop an application or a tool using techniques learned from this class. Example applications include:
- Building a mental health therapy chatbot;
- Building a chatbot for K-12 education or help illiterate people learn to read and write;
- Building a system that finds related literature for scientists;
- Building a domain-specific agent that help people find information and perform tasks automatically on the specified domain context;
- …
- NLP Research: Alternatively, students are also encouraged to pick a research-oriented project, identifying a research problem and proposing a novel approach to address the problem. Example research topics include:
- Interpreting the mechanisms of LMs in reasoning and planning (e.g., math reasoning, web navigation);
- Benchmarking how well an LM can reason or plan in scenarios that were not covered by prior work;
- Developing approaches to improve state-of-the-art LM performance on an established benchmark (e.g., math reasoning, event extraction, etc.);
- Developing approaches to dynamically control the reasoning length of a large reasonin model (LRM);
- …
In comparison, an Application may involve more novelty in applying existing techniques to an interested scenario, whereas a Research project involves more novelty in discovering new scientific insights and/or advancing state-of-the-art methodologies.
Sometimes, a project may lie in the intersection of both categories. For example, when developing a domain-specific agent to help people with information seeking and task completion (Application), you may need find that existing techniques all fail to enable a reasonable agent performance, so you eventually devise new methodologies (Research). We encourage such projects as they often lead to both scientific merits and practical tools.
No matter which kind of project your team decides, students need to submit project progress under the following two checkpoints.
Checkpoint 1: Project Kickstart
By the first checkpoint, you are expected to:
- Form a team. Use Piazza if you have an idea and are looking for teammates or for feedback/iterating on your idea.
- Decide your project topic and do the preliminary work towards your implementation. Your preliminary work should be detailed in a PDF report submission.
- Prepare a short presentation of your project in class and submit a PDF of your slides.
- Peer-review other teams’ project presentations.
Report Specifics Your report should include the following components using the provided template (TBD):
- Project Title: Something descriptive but also memorable. Make sure to add names and emails of team members under the title.
- Project Description and Motivation: What application or research problem are you targeting in this project? What's your motivation (i.e., why does it matter to build this application or work on this research problem)?
- Related Work: Have people worked on a similar application or a relevant research problem? Include at least 3 papers directly relevant to the proposed project. For each paper, use 3-4 sentences to describe 1) what prior work has done and 2) how your work is different from their.
- Project Plan, which must contain the following subsections:
- Dataset(s) - Please list which dataset(s) you plan to use and whether or not you have access to them.
- Implementation - Describe how to plan to implement this application or research exploration. Be specific about any software, libraries, etc., you plan to use.
- Resources - How much compute will you need? Have you confirmed the resource availability (from ORC Hopper or other resources)? If not, where do you plan to obtain the resources?
- Evaluation - How do you plan to evaluate your application or research exploration? Please be specific about the dataset(s) you plan to use and the evaluation metrics. Consider both qualitative (e.g., human evaluation, case studies) and quantitative (e.g., evaluation through automatic metrics) evaluations.
- Team Work and Timeline Clarification: (1) Describe each team member's contribution to this checkpoint. (2) Describe how each team member expects to collaborate towards the final project and make a timeline (as concrete as possible). Note that each team member should contribute roughly the same amount of work to the project.
Presentation Specifics The slides of your project pitch should include a summary of what you have in the PDF report:
- Title slide (Project Name, Team Members Names, Team Members Emails)
- Project description and motivation (1 page)
- Related work (1 page)
- Project plan and timeline (2-3 pages)
- What data will you use?
- How do you plan to implement the project and what resources will you need?
- How do you plan to evaluate your project?
- In what timeline do you plan to finish the project?
Checkpoint 1 Grading This component is worth 20% of your overall grade, consisting of 15% for the report or checkpoint content and 5% for presentation. The project peer review is worth another 10% under “Class Participation”.
- The grading of the checkpoint 1 report or content will be based on (1) completeness (e.g., do you have all required elements in the report and presentation?), (2) clarity and quality (e.g., do you describe each element clearly, carefully, and thoughfully?), (3) novelty (e.g., is your application or research topic novel?), and (4) others, such as whether the team has a reasonable timeline, plan for compute resources, and plan for team work.
- The grading of the checkpoint 1 presentation will be based on whether the team presents the project clearly and carefully under the given time limit. Note that all team members need to present the project in person; failing to do it will result in significant point deduction for the entire Checkpoint 1 (beyond the 5%).
Checkpoint 1 Peer Review You will be assigned to review three randomly assigned projects from your classmates. For each project, write a peer-review summary covering the following points:
- Project description and motivation: Does the team describe their project scope clearly? Is the project well motivated?
- Related work: Does the team show to research for related work carefully? Do they describe the related work and show the distinction of their project clearly? Do they omit any related work or do you suggest further related work to the team?
- Project plan and timeline: Are the project plan and timeline adequate? Does the team describe clearly what dataset(s) and compute they will use? Is their implementation plan clear and reasonably devised? Similarly, do you think they have a proper evaluation plan?
- Write 1-3 things where you think the team does well;
- Write 1-3 things where you think the team should have done better, and provide your suggestion.
We require the peer review to be specific and constructive. Imagine yourself as a critic, examining the project outline with a critical mind and providing helpful suggestions to your classmates. When you do so, you will also receive helpful feedback from your classmates. The peer review is worth 10% of the total grade as part of the class participation.
Submission Guideline TBD
Checkpoint 2: Final Report Submission
By the second checkpoint, you are expected to complete the following activities:
- Finish the entire project;
- Prepare a short in-class presentation of your project outcome in the last class (Dec 7th);
- Submit a PDF report including a link to your source code on GitHub one week after the last class (Dec 14th).
Report Specifics The final project report should includ the following components using the provided template (TBD):
- Project Title: It is okay if your title is slightly different from the title in your checkpoint 1 report. Make sure to add names and emails of team members under the title.
- Detailed System/Methodology Description: Describe in full detail what you have implemented, how you implemented it, including what datasets and resources you used in the implementation. Tip: A suggestion for system/methodology description is to draw a figure showing your system/methodology workflow; exmaple see Fig 2 of Srivastava and Yao (2025). Sometimes, you also need a figure or a table demonstrating your data processing; example see Fig 3 of Srivastava and Yao (2025) where the authors demonstrated how they process an event extraction task as a python code generation problem with a figure, and Table 1 of Gentopia AI (2023), where the authors illustrated the tasks and datasets used in their experiments.
- Evaluation:
- Experimental Setup - How did you evaluate your application or research outcomes? Be specific about the evaluation procedure and the used metrics. If your project involves proposing a new approach, you may also need to describe the baseline approaches in this section.
- Experimental Results - What's your evaluation result? Does your evaluation tell you where your application or research exploration succeeds and/or fails? Tip: You often need a table to summarize the evaluation results; example see Table 1 of Srivastava and Yao (2025).
- Analysis - Complement your evaluation with a further analysis. The goal is to dive deep and discover the reasons why the application or research exploration succeeds and/or fails, such that future researcher (or yourself!) knows how to further improve it.
- Limitations and Future Work:Describe any limitations in your current project design, including the system/approach design, the experimental setup, the implementation procedure, problems revealed from the evaluation, etc., and discuss how these limitations can be addressed in the future.
- Team Work Clarification: Describe each team member's contribution to this checkpoint. Note that each team member should contribute roughly the same amount of work to the project.
Presentation Specifics The slides of your final project presentation should include a summary of what you have in the PDF report:
- Title slide (Project Name, Team Members Names, Team Members Emails)
- System/Methodology description (1 page)
- You must include a figure (better a real-time demo) showing how your system or approach works.
- Evaluation (2-3 pages)
- In what setup did you evaluate your project?
- What’s your evaluation result?
- What’s the key takeaways from the evaluation?
- Future work: How can future researchers further improve this project? (1 page)
Grading This checkpoint is worth 30% of your overall grade, consisting of 20% for the report, 5% for the source code, and 5% for project presentation.
- The grading of the checkpoint 2 report or content will be based on (1) completeness (e.g., do you have all required elements in the report and presentation?), (2) clarity and quality (e.g., do you describe each element clearly, carefully, and thoughfully? do you use figures or tables properly to demonstrate the system/methodology design and evaluation results? are the analysis and future work discussion thoughtful? etc.), (3) novelty (e.g., is your final application or research exploration novel?), and (4) others, such as whether the team shows reasonable team work.
- The grading of the source code will be based on whether the team has a clear documentation for the grader to reproduce the project (see https://github.com/Ziyu-Yao-NLP-Lab/PyCode-TextEE for example; the key is to include all necessary commands for the grader to train/test your system/methodology), and whether the source code is fully executable (including loading and processing any required datasets and running the evaluation). Incomplete or problematic source code could threat the validity of the project report, and the grader may accordingly apply deductions to the report grade.
- The grading of the final project presentation will be based on whether the team presents the project clearly and carefully under the given time limit. Note that all team members need to present the final project in person; failing to do it will result in significant point deduction for the entire Checkpoint 2 (beyond the 2%).
Submission Guideline TBD