Embedding engineers in research teams: what works
Putting engineers next to researchers sounds obvious, and it often goes badly. Either the engineers become a service desk for tickets, or they are treated as a later stage that receives work over a wall. The version that works looks different from both.
Your org chart is your architecture
In 1968, Melvin Conway argued that organisations that design systems are constrained to produce designs that copy their own communication structures. The idea became known as Conway's law. Applied here, it says that if research and engineering only talk at a hand-off, the system will have a seam exactly at that hand-off, and that seam is where assumptions get lost.
Two ways to move research toward production
Over the wall
- Research validates a result
- Hand-off meeting and a repository link
- Engineers reverse-engineer assumptions
- Rebuild, then discover what was lost
Embedded from the start
- Engineer joins when the question is chosen
- Hard-to-undo decisions kept deliberate
- Shared ownership of what finally runs
- Result arrives intact, already deployable
Join the question, not the backlog
An engineer who only sees finished requests will optimise requests. One who sits in the discussions where hypotheses are chosen will notice early which choices are about to become expensive: a data source that will not scale, a method that cannot be monitored, a dependency nobody can license. Those are cheap to avoid at the whiteboard and costly to reverse after a result exists.
Agree on the few things that must be solid
Researchers need room to be messy. The arrangement works when both sides agree on a short list of things that are never messy, and leave everything else to the researcher's judgement.
| Always solid | Left to the researcher |
|---|---|
| Versioned data | Choice of method and model |
| Tracked experiments | Notebook structure while exploring |
| Reviewed changes to shared code | Local scratch work and throwaway scripts |
| Recorded environment | Order and pace of experiments |
Fewer rules, held firmly, beat many rules that everyone quietly skips.
Share ownership of the output
If the engineer owns the pipeline and the researcher owns the model, nobody owns the result. Shared ownership of the thing that finally runs in production keeps both sides interested in its quality, and gives an obvious person to call when it misbehaves.
- Review each other's work, in both directions.
- Decide together what counts as done: a number in a paper is not a deployed system.
- Keep the pairing visible to the wider organisation, so the platform learns from each project.
What to measure
| Measure | Healthy when |
|---|---|
| Time from validated result to running system | It falls project over project |
| Results rebuilt after hand-off | It trends to zero |
| Questions engineers ask after the fact | Most are asked at design time instead |
When those numbers move the right way, the arrangement is working. When they do not, the team is probably still passing work over a wall, however close the desks are.
Sources