CS 678 Course Project

Advanced Natural Language Processing


Students will form groups (each with 2-3 people) for the course project. There will be in total two submissions toward completing the project: Baseline Model Reproduction, and Model Analysis and Improvement. Requirements are stated in this page. We will also provide a LaTex template file for the report writing (which can also be downloaded from Blackboard).

Acknowledgement: Instructions and the template are adapted from UMass COMPSCI692A by Andrew McCallum, CMU CS711 by Graham Neubig, and from the ML Reproducibility Challenge 2022.

Checkpoint 1: Baseline Model Reproduction

In this project, you will have to choose a paper published in one of the following venues and attempt to reproduce their main result.

The objective is to assess if the experiments are reproducible, and to determine if the conclusions of the paper are supported by your findings. Your results can be either positive (i.e. confirm reproducibility), or negative (i.e. explain what you were unable to reproduce, and potentially explain why).

Essentially, think of your role as an inspector verifying the validity of the experimental results and conclusions of the paper.

What papers should I pick from the allowed venues? You can pick any paper published in the allowed venues as long as it introduced a new NLP model for an existing or a new NLP task/problem. Two common types of papers you should NOT pick: (a) papers that only introduced an opinion or position piece, without a technical or modeling contribution; (2) papers that are mostly about crowdsourcing (as for this type of papers you cannot really reproduce them, unless you similarly recruit crowdworkers, but this is not feasible given the cost).

A suggestion is to skim through the paper titles and first identify a broader topic of your interest. Then for a shortlist of papers you have the strongest interest, read their abstracts to understand their rough ideas, also making sure that the papers are not within the scope of this project checkpoint (i.e., it introduced a new NLP model for an existing or a new NLP task/problem).

Can I pick a paper outside the allowed venues? No. Exceptions are only given to students who have their own research agenda and have a clear plan on exactly what to do. For example, a PhD student researching in “NLP for Software Engineering” may be allowed to pick an NLP-relevant paper from a top venue in Software Engineering. However, if any student has this plan, they should reach out to the instructor for permission at an early stage.

Can I use the authors’ released source code? Using any published code, is allowed, as long as this is clear in your report, although we suggest that you first attempt to reimplement the experiments of the papers from scratch. In ACL Anthology, a lot of papers simply provide a github link in the first (or last) page. We suggest you first pick a paper that you find interesting e.g., because of the task or language(s) involved – do not just pick a paper because it has code available!

Scope We recommend you focus on the central claim of the paper. For example, if a paper introduces a new RL learning algorithm that performs better in sparse-reward environments, verify that you can re-implement the algorithm, run it on the same benchmarks and get results that are close to those in the original paper (exact reproducibility is in most cases very difficult due to minor implementation details). You do not need to reproduce all experiments in your selected paper, but only those that you feel are sufficient for you to verify the validity of the central claim.

Just re-running code is not a reproducibility study, and you need to approach any code with critical thinking and verify it does what is described in the paper and that these are sufficient to support the conclusions of the papers. Consider designing and running unit tests on the code to verify it works well and as described. Alternately, the methods presented can also be fully re-implemented according to the description in the paper. This is a higher bar for reproducibility that can take much more time, but may be helpful in detecting anomalies in the code, or shedding light on aspects of the implementation that affect results.

Some questions to consider:

  • Were you able to reproduce the exact experiments described in the paper?
  • If not, why not?
  • How sensitive is the implementation/model to hyperparameter choices, or the choice of random seed?
  • What about hardware limitations or other issues?

We suggest you take a look at the ML Reproducibility Challange website, which provides guidelines and resources for good reproducility research.

Report Specifics Generally, a report should include any information future researchers or practitioners would find useful for reproducing or building upon the chosen paper. The results of any experiments should be included; a “negative result” which doesn’t support the main claims of the original paper is still valuable.
For checkpoint 1, your report is expected to have about 4 pages.
Make sure to include the following in your project proposal:

  • Title for your project (this can also change later on). Something descriptive but also memorable.
  • Introduction section, which should explain the context of the paper. It should contain the following subsections:
    • Task / Research Question Description: What is the task the paper is trying to solve or what is the research question they are trying to answer?
    • Motivation & Limitations of existing work: Have others tried to solve the same task or answer a similar research question? What are they trying to do differently and why? What were the limitations or shortcomings of prior work?
    • Proposed Approach: Briefly describe the core contribution of the paper's proposed approach.
    • Likely challenges and mitigations: What is hard about this task / research question? What are your contingency plans if the reproduction turns out to be harder than expected or experiments do not go as planned?
  • Related Work Section: Include 3-4 sentence descriptions of no less than 3 papers directly relevant to the proposed research. Also mention how the paper you are working on differs from these.
  • Experiments section, which must contain the following subsections:
    • Datasets - Please list which datasets you are using, whether or not you have access to them, and whether or not they are publicly available with the same preprocessing and train / dev / tests as the work you will be reproducing.
    • Implementation - Please provide a link to a repo of your reimplementation (if applicable) and appropriately cite any resources you have used
    • Results - Provide a table comparing your results to the published results.
    • Discussion - Discuss any issues you faced. For example, if you use the authors' code, did you you find the code well documented and easy to understand, and did you identify any inconsistency between the code and how the method is described in the paper? If you implement the model from scratch, did you meet any difficulty caused by misleading language in the paper or the complexity of the model? Do your results differ from the published ones? If yes, why do you think that is?
    • Resources - Discuss the cost of your reproduction in terms of resources: computation, time, people, development effort, communication with the authors (if applicable, but note that you should not depend your progress fully on the authors' communication).
    • Error Analysis - Perform an error analysis on the model. Include 2-3 instances where the model fails. Discuss the error analysis in the paper -- what other analyses could the authors have ran? If you were able to perform additional error analyses, report it here.
  • Plan for Checkpoint 2 Describe your plan for Checkpoint 2 in a few paragraphs. Instructor will then follow up with feedback. This is important especially if you are unsure about the checkpoint requirement or if you plan to propose your own analysis/improvement.
  • Workload Clarification Describe how the team divides the workload in this checkpoint. Note that each team member should contribute roughly the same amount of work to the project.
  • Conclusion - Is the paper reproducible?

Submission: A single PDF should be turned in through Gradescope by the due date (only one person from each team needs to do this). The PDF should follow the proposed template and it should include a link to a Github repository with your reproduction code.

Grading: This component is worth 12% of your overall grade. The grading rubric for the report and reproduction component is as follows.

Grade
Component (portion) <=5 (fail) 8 (satisfactory) 10 (excellent)
Overall (2) Poor choice of paper. No effort at reproduction. Good choice of paper. Adequate effort in reproduction. Good choice of paper. Excellent effort in reproduction. Obviously more effort than peers.
Code (5) Insufficient effort. Simple copy of existing code. Adequate effort. Use existing code but with substantial effort in annotating and studying the code. Exceptional: Code is well-annotated. Include README files and bash scripts to aid reproduction, and comments on best practices and findings (e.g., if using authors' code, identifying potential inconsistency between the existing code and the paper description).
Report (5) No coherence, content missing, or serious grammar/spelling errors. Report fulfils the minimum requirements. Structure is acceptable. Literature review is OK. Some arguments and discussion. Excellent report in terms of content (good literature review, in-depth arguments and discussion, etc.). Well structured and formatted (e.g., using tables and figures smartly). No obvious grammar/spelling errors.

Checkpoint 2: Model Analysis and Improvement

In this checkpoint, you will perform a deeper analysis and improvement upon the baseline model you have reproduced in Checkpoint 1. Your checkpoint 2 project should explore at least one of the following dimensions:

  • Robustness In the previous checkpoint you performed a basic error analysis, and hopefully also sensitivity analysis with regards to its hyperparameter choices, about the reproduced model. Now, we turn our attention to robustness to data perturbation. Imagine you deployed your model, and suddenly your model had to deal with real-world data: ones with noise, spelling errors, typos, grammar mistakes, ambiguity, etc.
    Study the (award-winning) "Beyond Accuracy: Behavioral Testing of NLP Models with CheckList" and its accompanying code and python package. In this checkpoint, you are expected to: (1) Evaluate the robustness of your model in checkpoint 1: How does the model perform with regards to the checklist? What type of robustness exploration can/should you perform on your model? (2) Improve your model to be more robust: How can you improve the robustness of your model? In other words, can you make some improvement, so that the improved model can perform better on the same robustness checklist?
  • Multilinguality Most likely, the model/paper you reproduced has only been tested on one language (or just a hanful of them). What if we attempted to perform the same task on other languages or domains? Make the appropriate modifications to the code (e.g. if you have an English-only BERT-based model, substitute the English BERT with i.e. CamemBERT or FlauBERT to perform experiments in French, or substitute with the multilingual mBERT or XLM-R models to perform experiments in many languages). You can try to find datasets in other languages (beyond those the paper explores) or you can simply translate the existing datasets with the methods you used in HW3 (or both!). You may also explore the usage of Multilingual Checklist to perform behavioral testing on all languages you work with.
  • Your own analysis and improvement We also encourage students to get creative, taking inspiration from your life experiences and proposing your own model analysis and improvement! However, it is important to clarfiy your plan in your checkpoint 1 report, making sure that your deliverables satisfy the checkpoint requirement.

Report Specifics The final project report should have at least 6 pages. Your report should extend the checkpoint 1 report with the following new sections:

  • Checkpoint 2 Approaches. This section should include your effort in checkpoint 2.
    (1) For Robustness:
    • Approach for Evaluating the Model Robustness Describe what robustness analysis you have performed in this checkpoint. Provide sufficient details about your perturbation data, how you created it, how you used it as a robustness benchmark to evaluate the model, in what metrics, etc.
    • Approach for Improving the Model Robustness Describe your approach to improve the robustness of the model. Clearly specify how your created training data and validation data and how you trained the model (e.g., hyperparameters, training time, etc.).
    (2) For Multilinguality:
    • Approach for Training a Multilingual Model Describe your approach to make the model a multilingual one (or one that can understand languages different from the original one). Clearly specify how your created training data and validation data and how you trained the model (e.g., hyperparameters, training time, etc.).
    • Approach for Analyzing a Multilingual Model How are you going to evaluate the model on different languages (e.g., French), given that the original test set in checkpoint 1 is in the original language (e.g., English)? Describe your effort to evaluate a multilingual model, such as how you created a multilingual test set, which metrics you have used, etc.
    (3) For your own analysis/improvement: You should similarly describe how you perform the analysis and what approaches have you implemented for its improvement.
  • Checkpoint 2 Experimental Results In this section, describe your experimental results of checkpoint 2.
    (1) For Robustness:
    • Results of Robustness Evaluation Describe the evaluation results of your checkpoint 1 model on the robustness benchmark that you created. Include at least 2 examples where the model performs well and 2 examples where it fails (i.e., being not robust). Provide sufficient analysis and your thoughts on the observations.
    • Results of Improved Model Describe the evaluation results of your improved model on both the original test set (in checkpoint 1) and your robustness benchmark. Ideally, your improved model should perform comparably on the original test set but show to be much more robust on the robustness benchmark.
    (2) For Multilinguality:
    • Multilingual Model Performance Describe how your implemented multilingual model performs on the multilingual benchmark you created.
    • Analysis of Model Multilinguality Perform an in-depth analysis on your multilingual model. Does the model perform equally well in different languages? Why or Why not? For each language, include at least 2 examples where the model performs well and 2 examples where it fails. Provide sufficient analysis and your thoughts on the observations.
    (3) For your own analysis/improvement: You should similarly describe the analysis and experimental results of the checkpoint 1 or the improved model.
  • Discussions Provide any further discussions here, e.g., what challenges did you face when performing the analysis or trying to improve the model, and what could have been done if you will have more time on this project? Imagine you are writing this report to future researchers; be sure to include "generalizable insights" (e.g., broadly speaking, any tips or advice you'd like to share for researchers trying to analyze the robustness of an NLP model or trying to make a model multilingual?)
  • Workload Clarification Describe how the team divides the workload in both checkpoint 1 and checkpoint 2. Note that each team member should contribute roughly the same amount of work to the project.
  • Conclusion Give a short summary of what you have done in both checkpoint 1 and checkpoint 2. For checkpoint 1: is the paper reproducible? For checkpoint 2: do you find the model robust or is it feasible to make it a multilingual one? Also briefly summarize the potential future work.

Submission: A single PDF should be turned in through Gradescope by the due date (only one person from each team needs to submit this). Make sure the PDF includes links to a Github repository with your code that would allow us to reproduce your experiments. The PDF should follow the proposed template.

Grading: This component is worth 18% of your overall grade. The grading rubric for the project proposal component is as follows.

Grade
Component (portion) <=5 (fail) 8 (satisfactory) 10 (excellent)
Robustness or Multilinguality or Your Analysis/Improvement (5) No effort along the robustness/multilinguality dimension Fulfils the minimum requirements. For robustness: uses at least one dimension of the NLP Checklist or performs similar analysis. For multilinguality: experiments with the model in at least one language beyond the original paper Excellent analysis and improvement -- for robustness, studies multiple robustness dimensions and goes beyond the NLP Checklist, e.g. explores new robustness dimensions; for multilinguality, studies multiple languages/domains and perform error analysis (e.g. by introducing new multilingual Checklist templates).
Code (5) Insufficient effort. Simple copy of checkpoint 1's code. Adequate effort in writing code for checkpoint 2 progress. Code can be rerun by the TA with small effort. Exceptional: Substantial coding effort than checkpoint 1. Code is well documented with README files and bash scripts to aid reproduction. Code can be rerun by the TA without effort.
Report and Presentation (8) No coherence, content missing, or serious grammar/spelling errors. Presentation obviously not carefully prepared. Report fulfils the minimum requirements. Structure is acceptable. Literature review is OK. Some arguments and discussion. Presentation is OK. Excellent report in terms of content (good literature review, in-depth arguments and discussions, etc.). Well structured and formatted (e.g., using tables and figures smartly). No obvious grammar/spelling errors. Presentation is carefully prepared.
Previous