Research code is not production code, and that is fine
A research prototype has one job: answer a question as quickly as possible. It is written by people who are rewarded for insight, not for uptime, and it is shaped by that job. Hard-coded paths, a notebook that only runs top to bottom, a dependency pinned to whatever was installed on one laptop. None of this is carelessness. It is the right trade for the stage the work is in.
Trouble starts when the same artefact is asked to do a second job. A system that other people depend on has to start from a clean machine, fail in ways you can see, recover without a person watching, and change safely when requirements move. Those are properties of engineering, and they do not appear by accident when a prototype succeeds.
The model is the small part
In 2015, a group of Google engineers published Hidden Technical Debt in Machine Learning Systems, a paper that has shaped how the field talks about this gap. Their central observation: in real machine learning systems, only a small fraction of the code is the learning itself. The rest is configuration, data collection and verification, feature extraction, serving, monitoring and the machinery that keeps all of it running.
What a real ML system is made of
Conceptual diagram, not to scale. Box sizes show relative emphasis, not measured code share.
The practical consequence is uncomfortable. When a research team says a model is 'done', most of the system around it does not exist yet. The surrounding components are not decoration. They decide whether the model's output can be trusted, reproduced, explained and paid for.
What actually changes at promotion
When a result moves from prototype to product, five things shift at once:
- Inputs stop being whatever the researcher had to hand and become contracts with owners and schedules.
- Failure stops being an annoyance and becomes an event someone has to be told about.
- Change gives way from speed of iteration to safety: tests, reviews and staged releases.
- Audience grows from people who understand the method to people who only see the output.
- Cost becomes a number someone reports every month.
The same artefact, two jobs
| Research prototype | Production system | |
|---|---|---|
| Goal | Answer a question quickly | Keep answering it reliably |
| Inputs | Whatever is to hand | Versioned, owned, monitored |
| When it breaks | Rerun the cell | Alert, diagnose, recover |
| Change | Edit in place | Review, test, staged release |
| Measure of success | Insight | Uptime, accuracy over time, cost |
Plan for the second job early
The expensive mistake is treating productionisation as a rewrite at the end. By then the prototype has accumulated assumptions that nobody wrote down, and the people who knew them have moved on to the next question. The cheaper path is to put an engineer beside the research from the start. Not to slow it down, but to notice which decisions will be hard to undo, and to keep those few decisions deliberate while everything else stays fast.
Research code does not need to be production code. It does need a route to becoming it, and that route is easiest to build while the work is still small.
Sources