A reproducible experiment checklist for research teams

“It worked on my machine” is not reproducibility. Whether another person can obtain a comparable result depends on the environment, data, code, parameters, and records. A reproducible experiment does not require identical numbers every time; it makes differences explainable.

Before the run

• Pin code, dependency, and runtime versions.
• Record the dataset name, download date, version, license, and preprocessing.
• Record hardware, drivers, random seeds, and key parameters.
• Define primary metrics, baselines, controls, and stopping conditions.

For external data or model APIs, also record the endpoint, model version, request parameters, quota, and price. When a service can change server-side, save the run time and returned configuration.

During the run

Give every run a unique identifier and keep its inputs, configuration, metrics, and logs. Keep failed runs too: record the failure, whether it was retried, and whether the retry changed parameters. Deleting failures makes stability look better than it is.

For stochastic experiments, do not report only the best run. State the number of runs, aggregation method, and variation. For small-sample or exploratory work, state exactly how much the result can support.

After the run

Keep the README, configuration, data description, command, and result summary together. Test a minimal end-to-end reproduction in a fresh environment before publishing a figure or writing a conclusion.

Write a separate “what we did not establish” section: unavailable data, unfixed service versions, missing ablations, and known measurement error. A transparent limitations section is usually more valuable than an absolute claim.

Reproducibility is part of research quality, but it does not make a conclusion correct. It shows whether a result is stable under given conditions; it does not replace theory or independent validation.