Repeatability Evaluation

ADHS'21 will feature a Repeatability Evaluation (RE). All accepted papers with computational artifacts will be invited to submit a Repeatability Evaluation Package (REP) for evaluation against ease of reproducibility and usability. The evaluation will be conducted by the RE committee who will try to reproduce the results and examine the usability. The aim of the RE is not only to find issues with reproducibility, but also to identify the effort required to reproduce the computational results.

Papers with RE packages that pass the repeatability evaluation will be listed online and in the final proceedings. Furthermore, such papers will also receive a repeatability badge that will be included on the first page of the published version. A paper with a RE package that is not able to pass the repeatability evaluation criteria, will be treated the same as a paper which did not submit a RE package.

It is optional to submit an REP, and if submitted, the evaluation result will not affect the review process of its corresponding paper. For the submission date, please see the Call for Papers. The repeatability evaluation effort at ADHS is inspired by the HSCC repeatability evaluation and borrows significantly from it.

Benefits to the Authors and Goals

The RE committee will review and provide feedback on the ease and quality of reproducibility of the results, which will further strengthen the quality of published papers with computational elements. The authors can use the feedback to improve the reproducibility and usability of their software. Another goal of the ADHS repeatability evaluation process is to provide authors feedback on the quality of the software.

Submission Guidelines

The website for submission of the Repeatability Evaluation Package (REP) is EasyChair.

The REP should have:

• A pdf of the submitted paper that will be used by the RE committee to compare against the results reproduced by the REP.
• A document (either a webpage, a pdf, or a plain text file) that includes information on:
  • Reproducible elements of the paper included in the REP (e.g., specific figures, tables, etc.).
  • Instructions for installing, running and extracting the reproducible elements. Scripts are preferred over complex instructions. Please include instructions on the expected output for each script included in the REP and how it relates to the computational elements in the paper.
  • System requirements for running the REP software (e.g., OS, compilers, environments, hardware specs, etc.).
  • Required licenses if any (such as Matlab and its toolboxes, optimization software), computing resources used and estimated time required to reproduce the results.
• The software and any accompanying data should be made available with a link that remains accessible throughout the review process. Please package the REP using one of the below options (in order of decreasing preference):
  • Virtual Machine using VirtualBox or VMware (most preferred).
  • Docker Image (list the host OS used in the instructions).
  • If the previous options are not viable, provide the source code as a zip file.

We strongly recommend that the authors do a dry run on the final version of the REP. This is best done by letting someone else try installing the REP on another machine and check the accuracy of the included installation and running instructions.

Repeatability Evaluation Criteria

The submissions will be evaluated based on the below criterion:

Reproducibility is evaluated in order to answer the question: can non-developers with the REP software and the provided instructions reproduce the claims and results in the paper (figures, tables, etc)? Specifically, this criteria checks for:

• Ease of reproducibility: Quality of instructions for installing the REP, and for reproducing the results.
• Quality and quantity of reproducible elements: How many elements can be reproduced, and to what accuracy (using the provided instructions)?

Usability addresses the ease of use and extension, and evaluates the REP on:

• Usage instructions: instructions for using, experimenting with the software, modifying parameters, adding new examples. This is exemplified by including a user guide.
• Software practices used: understandable, formatted, modular code that includes necessary comments, tests, etc.
• Instructions to extend and modify: this is exemplified by including a developer’s guide or a design document.

Additional Information

Like initial paper submissions, REPs are also considered confidential material. The committee members agree not to share REP contents, and to delete them after evaluation. REPs remain the property of the authors, and there is no requirement to post them publicly (although encouraged).

The repeatability evaluation process uses anonymous reviews so as to solicit honest feedback. Authors of REPs should make a genuine effort to avoid learning the identity of the reviewers. This effort may require turning off analytics, or only using systems with high enough traffic that REC accesses will not be apparent. In all cases where tracing is unavoidable the authors should provide warnings in the documentation so that reviewers can take necessary precautions to maintain anonymity.