Reproducibility is an infrastructure problem
When a result cannot be reproduced, the instinct is to ask researchers to be more careful. Keep better notes, record the seed, write down the command. Careful people still lose to an environment that changes underneath them: a base image updated overnight, a library that resolves to a new minor version, a dataset refreshed in place.
It is common, and people say so
In 2016, Nature surveyed its readers on reproducibility and received 1,576 responses. The results, reported in 1,500 scientists lift the lid on reproducibility, are striking. The sample was self-selected, so treat the percentages as a picture of these respondents rather than of all scientists, but the direction is hard to ignore.
Researchers who tried and failed to reproduce results
Share of respondents. Both bars are lower bounds: the survey reports 'more than'.
The same survey asked what contributes most. More than 60% said each of two factors, pressure to publish and selective reporting, always or often contributed. Those are cultural causes, and they are hard to fix with tooling alone. But a large share of the failures people describe are mundane: missing code, missing data, an environment nobody recorded. Those are infrastructure problems, and infrastructure problems have infrastructure fixes.
Code sharing improved once it was easy to ask for
There is evidence that structure helps. For NeurIPS 2019, the organisers ran a reproducibility programme with a code submission policy and a checklist, described in Improving Reproducibility in Machine Learning Research. Roughly 40% of submissions included code at submission time. By the camera-ready stage that had risen to about 74%. Nobody was asked to become more virtuous. The conference made the right behaviour an explicit, expected step.
Papers that shared code at NeurIPS 2019
Share of papers, by stage
- ≈40%At submission
- ≈74%At camera-ready
Four things worth pinning
- Code: every experiment ties to a commit, and uncommitted changes are visible rather than silently included.
- Environment: dependencies resolve from a lockfile, and the base image is referenced by digest, not by a moving tag.
- Data: inputs are versioned and addressed immutably. Overwriting a dataset in place is the most common way to lose a result.
- Configuration: parameters live in a file stored with the run, not in a shell history.
Make the record automatic
None of this should rely on someone remembering. A run should capture its own commit, environment digest, data versions and configuration, and store them next to the output. When a colleague asks how a number was produced a year later, the answer is a lookup, not an archaeology project.
Can a new engineer, on a clean machine, rerun last quarter's headline experiment from a single command and get the same answer within tolerance?
The test we use at 75bit. If the answer is no, the gap is infrastructure.
Close that gap before the next result depends on it.
Sources