Blog

Embedding engineers in research teams: what works

· 7 min read · 75bit

Putting engineers next to researchers sounds obvious, and it often goes badly. Either the engineers become a service desk for tickets, or they are treated as a later stage that receives work over a wall. The version that works looks different from both.

Your org chart is your architecture

In 1968, Melvin Conway argued that organisations that design systems are constrained to produce designs that copy their own communication structures. The idea became known as Conway's law. Applied here, it says that if research and engineering only talk at a hand-off, the system will have a seam exactly at that hand-off, and that seam is where assumptions get lost.

Two ways to move research toward production

Over the wall

  1. Research validates a result
  2. Hand-off meeting and a repository link
  3. Engineers reverse-engineer assumptions
  4. Rebuild, then discover what was lost

Embedded from the start

  1. Engineer joins when the question is chosen
  2. Hard-to-undo decisions kept deliberate
  3. Shared ownership of what finally runs
  4. Result arrives intact, already deployable
Illustration of the pattern described by Conway (1968). Conceptual, not measured.

Join the question, not the backlog

An engineer who only sees finished requests will optimise requests. One who sits in the discussions where hypotheses are chosen will notice early which choices are about to become expensive: a data source that will not scale, a method that cannot be monitored, a dependency nobody can license. Those are cheap to avoid at the whiteboard and costly to reverse after a result exists.

Agree on the few things that must be solid

Researchers need room to be messy. The arrangement works when both sides agree on a short list of things that are never messy, and leave everything else to the researcher's judgement.

Always solidLeft to the researcher
Versioned dataChoice of method and model
Tracked experimentsNotebook structure while exploring
Reviewed changes to shared codeLocal scratch work and throwaway scripts
Recorded environmentOrder and pace of experiments

Fewer rules, held firmly, beat many rules that everyone quietly skips.

Share ownership of the output

If the engineer owns the pipeline and the researcher owns the model, nobody owns the result. Shared ownership of the thing that finally runs in production keeps both sides interested in its quality, and gives an obvious person to call when it misbehaves.

  • Review each other's work, in both directions.
  • Decide together what counts as done: a number in a paper is not a deployed system.
  • Keep the pairing visible to the wider organisation, so the platform learns from each project.

What to measure

MeasureHealthy when
Time from validated result to running systemIt falls project over project
Results rebuilt after hand-offIt trends to zero
Questions engineers ask after the factMost are asked at design time instead

When those numbers move the right way, the arrangement is working. When they do not, the team is probably still passing work over a wall, however close the desks are.

Sources