The price of a token has collapsed. The size of the bill has not. That contradiction is where most AI programs quietly die — not in a failed proof of concept, but six months later, when finance asks what the line item is for and nobody can answer.
We have written before about how much cheaper AI has become. Both things are true at once, and the tension between them is the whole story: unit costs fell dramatically while total spend rose anyway, because cheap inference invites far more inference.
The cost nobody forecasts is the one that recurs
Most AI budgets are built around the wrong number. Teams estimate the cost of building — licences, model access, a few weeks of engineering — and treat the rest as noise. The bills that actually arrive are operational.
Industry research puts inference as the dominant recurring cost and finds it is routinely under-forecast by three to five times. And inference is only the visible part. The categories that surprise teams are the ones nobody budgets: observability, evaluation runs, retries against a flaky endpoint, and human review of AI output — which is a payroll cost dressed as a technology one.
Meanwhile the visibility to catch any of this is largely absent. In Flexera’s 2026 research, 59% of organizations reported that wasted AI software spend had increased over the past year, and only 31% said they had accurate visibility into what AI was costing them. You cannot optimise what you cannot see, and most companies cannot see it.
Discipline, not ambition, separates the winners
The widely quoted MIT finding — that roughly 95% of generative AI pilots produce no measurable return — gets read as an indictment of the technology. The more useful reading is that it is an average concealing an enormous spread.
A 2026 survey of 240 enterprises broke positive-ROI rates down by how disciplined the operating approach was. The gradient is steep:
| Maturity stage | Positive ROI | What defines it |
|---|---|---|
| Ad-hoc pilots | 8% | Experiments with no owner, no budget ceiling, no success metric |
| Governed pilots | 34% | Named owner, defined scope, agreed measure of success |
| Productionized | 62% | Monitored, cost-tracked, integrated into a real workflow |
| Operating model | 88% | Standard practice with accountability, review and cost ownership |
An eleven-fold difference between the top and bottom tiers, and not one point of it is about model choice. The same research found top-quartile teams running at 60–75% lower cost per token than bottom-quartile teams — a three-to-fourfold swing — on the same commercially available models. And roughly one pilot in three reaches production in top-quartile programs, against fewer than one in ten at the bottom.
The organizations getting returns are not using better AI. They are running it like a system rather than a series of experiments.
On the 8% to 88% gap
Where the money actually goes
If you are trying to find your own overrun, these are the usual culprits, roughly in order of how often they surprise people.
- Retrieval bloat. Stuffing more context into every call because it improves answers. It does — and it multiplies your per-request cost by the size of what you pasted in.
- Retries and failures. Every timeout, rate limit and malformed response that gets retried is billed. Teams rarely instrument this, so it never appears in an estimate.
- Evaluation runs. Testing a prompt change across a few thousand cases is a real inference bill, repeated every iteration.
- Human review. Someone checking AI output is the largest hidden cost in most regulated workflows, and it sits in payroll rather than the AI budget.
- Abandoned pilots still running. The proof of concept nobody switched off, quietly calling an API for eight months.
- Shadow AI. Individual subscriptions on expense reports, invisible to procurement and impossible to negotiate as a volume.
Six controls that change the number
None of this requires a FinOps team. It requires the same rigour you would apply to any other recurring operational expense.
- Instrument before you scale. Log tokens, cost and latency per feature from day one. Retrofitting cost attribution onto a running system is painful and usually never happens.
- Give every AI workload an owner and a budget ceiling. Not a team — a person, with a number they are accountable for. This single change moves you from the 8% tier to the 34% tier.
- Set a kill date on every pilot. Decide up front what success looks like and when you will judge it. Pilots without an end date become permanent line items regardless of whether they worked.
- Right-size the model per task. Most production calls do not need a frontier model. Routing simple work to a smaller model is where much of that 60–75% cost gap comes from.
- Cache and trim aggressively. Cache repeated calls, and check what your cached-input rate actually is — some vendors have raised it while holding headline pricing flat.
- Consolidate shadow subscriptions. Find the individual seats on expense reports and negotiate them as one contract. This is usually the fastest cash saving available.
The uncomfortable part
The pattern in this data is not that AI does not work. It is that the discipline gap is far more expensive than the technology gap. Two companies buying identical model access can differ four-fold on unit cost and eleven-fold on the odds of getting a return, purely on the basis of whether anyone owns the outcome.
For a small or mid-sized business this is genuinely good news. You cannot outspend an enterprise on compute, but ownership, kill dates and cost tracking cost nothing, and they are exactly what most large organisations are worst at. This is the rare case where being small is an operating advantage.
Frequently asked questions
Our AI bill jumped and nobody can explain why. Where do we start?
Start with attribution, not reduction. Pull the last three months of usage from each provider and map spend to specific features or teams. In most cases a small number of workloads account for the majority of cost, and at least one of them is something nobody remembers launching.
Is it cheaper to run open-weight models ourselves?
Sometimes, and rarely for the reason people expect. Self-hosting trades a variable per-token bill for fixed infrastructure and engineering time. It wins at sustained high volume or where data cannot leave your environment. At low or spiky volume, hosted APIs are usually cheaper once you count the staff time honestly.
How do we forecast inference cost before we build?
Measure a representative sample rather than estimating. Run a few hundred real requests, record actual token counts including retries and system prompts, then multiply by realistic volume — and apply a buffer. Given that under-forecasting by three to five times is the documented norm, budgeting three times your naive estimate is a reasonable starting position.
What is a realistic ROI timeline?
For a scoped workflow automation with a named owner and a defined metric, weeks. For an open-ended “let’s explore AI” initiative, often never — that is precisely the 8% tier. The variable that predicts the timeline is whether you defined the measure of success before you started.
Getting the number under control
Delana Technologies builds AI systems that reach production and does cloud and cost optimisation for companies whose bills have outrun their forecasts. If your AI spend is rising faster than the value you can point to, call 239.414.5126 or contact us — the first conversation is usually about attribution, and it is free.
Sources: Flexera, 2026 State of ITAM Report (wasted AI spend and cost visibility); RapidData, State of Enterprise AI 2026, survey of 240 enterprises (ROI by maturity, cost-per-token variance, inference under-forecasting); MIT research on generative AI pilot outcomes, as cited in that report.