Blog

Reproducibility is an infrastructure problem

· 7 min read · 75bit

When a result cannot be reproduced, the instinct is to ask researchers to be more careful. Keep better notes, record the seed, write down the command. Careful people still lose to an environment that changes underneath them: a base image updated overnight, a library that resolves to a new minor version, a dataset refreshed in place.

It is common, and people say so

In 2016, Nature surveyed its readers on reproducibility and received 1,576 responses. The results, reported in 1,500 scientists lift the lid on reproducibility, are striking. The sample was self-selected, so treat the percentages as a picture of these respondents rather than of all scientists, but the direction is hard to ignore.

Researchers who tried and failed to reproduce results

Share of respondents. Both bars are lower bounds: the survey reports 'more than'.

  • Another scientist's experimentsover 70%
  • Their own experimentsover 50%
Baker, Nature 533, 452–454 (2016); 1,576 respondents, self-selected.

The same survey asked what contributes most. More than 60% said each of two factors, pressure to publish and selective reporting, always or often contributed. Those are cultural causes, and they are hard to fix with tooling alone. But a large share of the failures people describe are mundane: missing code, missing data, an environment nobody recorded. Those are infrastructure problems, and infrastructure problems have infrastructure fixes.

Code sharing improved once it was easy to ask for

There is evidence that structure helps. For NeurIPS 2019, the organisers ran a reproducibility programme with a code submission policy and a checklist, described in Improving Reproducibility in Machine Learning Research. Roughly 40% of submissions included code at submission time. By the camera-ready stage that had risen to about 74%. Nobody was asked to become more virtuous. The conference made the right behaviour an explicit, expected step.

Papers that shared code at NeurIPS 2019

Share of papers, by stage

  • ≈40%At submission
  • ≈74%At camera-ready
Pineau et al., arXiv:2003.12206, Table 1. Values rounded.

Four things worth pinning

  • Code: every experiment ties to a commit, and uncommitted changes are visible rather than silently included.
  • Environment: dependencies resolve from a lockfile, and the base image is referenced by digest, not by a moving tag.
  • Data: inputs are versioned and addressed immutably. Overwriting a dataset in place is the most common way to lose a result.
  • Configuration: parameters live in a file stored with the run, not in a shell history.

Make the record automatic

None of this should rely on someone remembering. A run should capture its own commit, environment digest, data versions and configuration, and store them next to the output. When a colleague asks how a number was produced a year later, the answer is a lookup, not an archaeology project.

Can a new engineer, on a clean machine, rerun last quarter's headline experiment from a single command and get the same answer within tolerance?

The test we use at 75bit. If the answer is no, the gap is infrastructure.

Close that gap before the next result depends on it.

Sources