Blog

The compute stack your experiments quietly depend on

· 7 min read · 75bit

Most research teams describe their stack by the framework they use. The more important part is underneath: where jobs queue, where data lives, how long a stuck run holds a GPU, and who finds out when the bill doubles. These are unglamorous questions, and they set the ceiling on how many experiments a team can run in a week.

Waste is the default, and it is measurable

Flexera's annual State of the Cloud survey asks technology leaders to estimate how much of their IaaS and PaaS spend is wasted. The 2025 report, based on 759 respondents, put it at 27%. The 2026 report, published in March, put it at 29%, reversing a five-year downward trend, which Flexera attributes in part to the added cost complexity of AI workloads. These are self-reported estimates, not audited figures. Even so, roughly a quarter or more of cloud spend being seen as wasted is a useful prior for any research platform that has never been looked at closely.

29%

of IaaS and PaaS spend estimated as wasted in 2026, up from 27% in 2025, and the first rise after five years of decline.

Flexera, State of the Cloud 2025 and 2026. Self-reported by survey respondents.

Queueing is a research decision

A shared pool with no scheduling policy rewards whoever submits first and punishes everyone with a short job. Even a simple policy helps: priorities by purpose, a fair share per team, and preemption rules that people have agreed on in advance. The aim is for a researcher to know roughly when a job will start, because uncertainty about the queue changes how people design experiments.

Data movement is the hidden cost

Compute is easy to price. Data movement is not. Datasets copied between buckets, regions or clusters for each run add latency and cost that never appear on one line item. Keep data close to compute, cache what is reused, and make the location of a dataset a decision rather than an accident.

Visibility beats restriction

Teams usually reach for quotas first. We find visibility works better and creates less friction. Show each team what its jobs cost, tag every run with who started it and why, and surface idle resources before they turn into a surprise. People adjust quickly once the numbers are in front of them.

Two ways to control spend

RestrictionVisibility
How it worksHard quotas per teamCost per run, tagged to owner and purpose
Researcher experienceBlocked, then negotiatingInformed, adjusts without asking
Failure modeWork stops at the limitNeeds someone to read the numbers
Best usedAs a backstopAs the default
  • Tag every job with owner and purpose.
  • Report cost per experiment, not just per month.
  • Alert on idle accelerators and runs that have stopped making progress.

A good compute stack is one that researchers stop thinking about. Getting there is mostly careful, unglamorous work, and it pays back every week afterwards.

Sources