Blog

Research code is not production code, and that is fine

· 6 min read · 75bit

A research prototype has one job: answer a question as quickly as possible. It is written by people who are rewarded for insight, not for uptime, and it is shaped by that job. Hard-coded paths, a notebook that only runs top to bottom, a dependency pinned to whatever was installed on one laptop. None of this is carelessness. It is the right trade for the stage the work is in.

Trouble starts when the same artefact is asked to do a second job. A system that other people depend on has to start from a clean machine, fail in ways you can see, recover without a person watching, and change safely when requirements move. Those are properties of engineering, and they do not appear by accident when a prototype succeeds.

The model is the small part

In 2015, a group of Google engineers published Hidden Technical Debt in Machine Learning Systems, a paper that has shaped how the field talks about this gap. Their central observation: in real machine learning systems, only a small fraction of the code is the learning itself. The rest is configuration, data collection and verification, feature extraction, serving, monitoring and the machinery that keeps all of it running.

What a real ML system is made of

Conceptual diagram, not to scale. Box sizes show relative emphasis, not measured code share.

Concept after Sculley et al., NeurIPS 2015. Redrawn as a schematic; not measured data.

The practical consequence is uncomfortable. When a research team says a model is 'done', most of the system around it does not exist yet. The surrounding components are not decoration. They decide whether the model's output can be trusted, reproduced, explained and paid for.

What actually changes at promotion

When a result moves from prototype to product, five things shift at once:

  • Inputs stop being whatever the researcher had to hand and become contracts with owners and schedules.
  • Failure stops being an annoyance and becomes an event someone has to be told about.
  • Change gives way from speed of iteration to safety: tests, reviews and staged releases.
  • Audience grows from people who understand the method to people who only see the output.
  • Cost becomes a number someone reports every month.

The same artefact, two jobs

Research prototypeProduction system
GoalAnswer a question quicklyKeep answering it reliably
InputsWhatever is to handVersioned, owned, monitored
When it breaksRerun the cellAlert, diagnose, recover
ChangeEdit in placeReview, test, staged release
Measure of successInsightUptime, accuracy over time, cost

Plan for the second job early

The expensive mistake is treating productionisation as a rewrite at the end. By then the prototype has accumulated assumptions that nobody wrote down, and the people who knew them have moved on to the next question. The cheaper path is to put an engineer beside the research from the start. Not to slow it down, but to notice which decisions will be hard to undo, and to keep those few decisions deliberate while everything else stays fast.

Research code does not need to be production code. It does need a route to becoming it, and that route is easiest to build while the work is still small.

Sources