The AI Strategy Guide logo The AI Strategy Guide at the cusp of AI Subscribe →
Scalable AI Adoption: The Twenty-First Workload Problem — Capabilities illustration

Scalable AI Adoption: The Twenty-First Workload Problem

Tom Prommer · CIO/CTOUpdated 2026-05-2912 min read

A platform team I advised in early 2025 had shipped seven AI workloads in nine months and was on track to ship five more by year-end. In the November planning review, the CFO asked for the per-workload cost trajectory. The team had not calculated it. They produced the number on a Tuesday and presented it on the Friday. The number was bad. Each workload was costing roughly €180,000 per year to run, compounded by the fact that the first three workloads’ costs were rising as usage grew. Twelve workloads at €180,000 was €2.16M annual run-rate, against a programme budget of €1.4M. The programme had not failed; it had succeeded into insolvency. The Friday meeting cut the planned five new workloads to one and reallocated the engineering team to a six-month cost-reduction effort. The programme survived. Six other AI programmes I have audited from a comparable starting position did not, because they did not surface the unit economics until workload twelve was already in production and the marginal cost had compounded across the portfolio.

This is the scalable-adoption problem. The first three workloads tell you nothing useful about the twenty-first. The economics, the operations, the governance burden, the engineering attention — all of these scale non-linearly with workload count, and almost none of the literature on AI strategy treats this with the seriousness it warrants because the literature is mostly written by people who have shipped a pilot, not by people who have run a portfolio of twenty production workloads.

This page is about the work that distinguishes the programme that scales from the programme that succeeds-into-insolvency. The strategy-level frame on the same question lives in the roadmap hub; this page is the capability-layer version, and it focuses on the four moves that buy you the twenty-first workload at a marginal cost lower than the first.

The platform-versus-application sequencing trap

The first decision is sequencing. You can build a shared AI platform first and then build applications on top of it, or you can build applications first and then extract a shared platform from the patterns the applications reveal. Both decisions have advocates. Only one of them works at the twenty-workload scale, and the one that works is the second one, with one important caveat.

The teams that build the platform first ship a platform in month six that is internally coherent and externally wrong. The platform was designed against assumed workload patterns. The actual workload patterns, when applications start arriving in month eight, differ in three ways that nobody anticipated: the dominant pattern is retrieval-augmented generation rather than agentic orchestration; the latency requirements are bimodal (sub-second for some, batch-acceptable for others) rather than uniform; the model-selection axis matters more than the orchestration axis. The platform has to be refactored. The refactor takes two quarters. The applications that should have shipped in month nine ship in month fifteen. The programme has not failed but it has burned six months of opportunity cost, which is usually enough to lose the leader posture the strategy committed to.

The teams that build two or three applications first, deliberately and with the explicit intent of extracting platform patterns from them, ship the platform in month nine against an empirical understanding of what the workloads actually need. The platform is right on first design because it is built against observed reality rather than assumed reality. The remaining seventeen workloads ship onto it without major refactor.

The important caveat: the team has to decide, in advance, that the first two or three applications are platform-extraction exercises, not standalone products. The temptation, after the third application ships and works, is to ship application four and five before extracting the platform — because the applications are visibly delivering value and the platform extraction looks like overhead. This is the failure mode that produces the late-stage refactor at workload twelve, which is more expensive than the early-stage one at workload three. The discipline is to extract the platform on schedule even when the applications are succeeding.

Conway’s Law applies, predictably and recursively. The shape of your platform will mirror the shape of the team that builds it. If the team is a single platform engineering function, the platform will be coherent and slow. If the team is one engineer per application with informal coordination, the platform will be a fiction. The shape that works in my experience is a small platform team (three to five engineers) with explicit charter to consume the empirical patterns from the application teams, plus application engineers embedded into business-unit teams. The platform team’s success metric is the marginal cost of the Nth workload — twenty-first, fiftieth, whichever lies one beyond your current portfolio. The application teams’ success metric is the business value of their individual workload. These metrics are aligned if the structure is right and misaligned if the platform team is asked to ship applications, which is the predictable failure mode of cost-pressured organisations.

Shared services versus distributed ownership

The second decision is ownership. Does the shared platform team own model inference, evaluation, monitoring, governance integration, and on-call for AI-specific failures across all twenty workloads, or does each workload own its own end-to-end stack with the platform providing only common primitives.

The answer that scales is the first one for the shared primitives and the second one for the workload-specific behaviour, with a clear line between the two. The line itself is the work. The platform team owns: the model-routing layer, the inference caching layer, the evaluation harness, the cost-tracking instrumentation, the governance-artefact emission (model cards, deployment-gate evidence, red-team logs). The application teams own: the prompt design, the tool-calling surface, the workload-specific evaluations, the business-logic integration, the user-facing UX. The application team is on-call for the workload’s business behaviour. The platform team is on-call for the shared infrastructure’s behaviour.

This split is the structurally correct one because it puts the people who can fix a problem closest to the page that fires for that problem. A platform team on-call for prompt failures across twenty workloads will burn out within six months because they do not know the business logic of any of the twenty. An application team on-call for inference-layer failures will burn out because they do not own the inference layer. Both failure modes are common and both can be predicted from the org chart at design time.

The mistake to avoid: a “centre of excellence” model where the shared team owns everything and the application teams call into them for each new workload. The COE model is the failure mode the root hub flagged explicitly. It produces a queue, the queue grows, the application teams route around the COE, shadow AI replaces governed AI, and the programme either re-organises within eighteen months or fails. There is no version of the COE structure that scales to twenty workloads in my engagement experience. The shared-platform-plus-embedded-application-engineers structure scales reliably.

The unit-economics work

The third move is the one almost no team prioritises until it is late. Run the unit-economics math at workload two, not at workload twelve.

The math is concrete. For each workload, compute the per-call marginal cost — model inference, retrieval-system query, observability emission, governance overhead. Project the call volume at one-year steady state. Multiply. Add the engineering maintenance overhead — typically 0.2 to 0.5 FTE per production workload at steady state, depending on workload complexity. The number is the realistic per-workload annual run cost. Compare it against the workload’s business-case benefit. The ratio is what determines whether the workload survives at scale.

The pattern that kills programmes: the first three workloads were funded as discretionary innovation and the per-workload cost was never calculated because the spend was below CFO scrutiny threshold. Around workload five, the cumulative spend crosses the threshold. The CFO asks for the per-workload number. The team produces it for the first time. It is three to five times what they assumed because they had been amortising shared costs incorrectly, ignoring engineering maintenance, and using promotional vendor pricing that does not survive contract renewal. The programme then either freezes or invests in a cost-reduction effort that should have happened at workload two.

The cost-reduction levers, in the order they typically pay back:

Caching. Response and embedding caching for queries that repeat — and in any production workload, queries repeat more than the team expects. The first time I instrumented a customer-facing AI assistant for cache opportunity, the realised hit rate was 43%, against a team assumption of “almost no cache opportunity”. The cache reduced inference spend by 38%. This is consistently the highest-leverage lever and consistently the most under-used because it is unglamorous and because nobody publishes blog posts about it.

Routing to smaller models. Most queries do not need the largest available model. A router that classifies queries by complexity and routes them to the cheapest model that can answer them adequately reduces inference spend substantially. The 2024-2025 model-cost curves visible across commercial providers (for example, the comparison data OpenRouter publishes) show roughly an order-of-magnitude spread between the cheapest capable models and the largest frontier models for many enterprise tasks. The routing layer is the second-most-leveraged lever in my engagement data.

Batch inference. Any workflow whose latency requirement allows for minutes rather than seconds should be running on batched inference, which is meaningfully cheaper per token than synchronous inference. Most teams default everything to synchronous because the first few workloads were latency-sensitive. The shift to batched-by-default for batch-acceptable workloads typically reduces inference spend on those workloads by 30-50%.

Targeted fine-tuning. Replacing a large-model call with a smaller fine-tuned model for a high-volume narrow task. This is the lever most teams reach for first because it sounds sophisticated; in practice it should be applied last because it adds operational complexity (a model to host, a re-training cadence, a drift-monitoring concern) that the first three levers do not. Fine-tuning is genuinely valuable at the twenty-workload scale where the operational complexity is amortised across volume. At the five-workload scale it is usually premature.

If a programme applies the first three levers between workload two and workload five, the marginal cost of workload twenty-one is meaningfully below the marginal cost of workload one. If a programme does not apply them, the marginal cost is unchanged and the programme freezes at the workload count where the cumulative spend hits the CFO threshold.

Use-case stacking and the workload-funds-the-next-workload pattern

The fourth move is structural rather than economic. The teams that scale to twenty workloads are not running twenty independent business cases. They are running a portfolio in which earlier workloads fund the platform investment that makes later workloads cheaper.

The pattern: workloads one through three are the highest-value, lowest-risk applications the strategy identified. They are funded individually, they generate clear business benefit, and they pay for themselves on standalone economics. The platform investment that comes out of those three workloads (the caching layer, the routing layer, the evaluation harness, the governance instrumentation) is funded as a tax on those three workloads’ business cases — typically 15-25% of the workload’s first-year value.

Workloads four through ten benefit from the platform investment and are correspondingly cheaper to ship. Their business cases require less standalone return because the marginal cost is lower. They can therefore be marginal-value applications that would not have justified standalone investment at the original cost structure but do justify it at the reduced one.

Workloads eleven through twenty are the long tail. They ship at marginal cost low enough that the business case threshold becomes low. This is where the programme realises the optionality value that the strategy document promised — the ability to ship AI capability at twenty places in the organisation, not just three. This optionality is the actual ROI of the platform investment, and it is invisible at workload three.

The failure mode of teams that do not run this stacking pattern: each workload has to clear the full original cost bar on its own merits. Workloads four through ten are individually harder to justify than workloads one through three because the obvious high-value applications have been done. The programme either ships diminishing-return workloads at the original cost (which the CFO eventually stops) or stops shipping new workloads (which kills the scaling thesis). Either failure mode is visible from workload six or seven.

What this means for the strategy document

The strategy that approved the AI programme is upstream of all of this. The strategies that hold up at workload twenty are the ones that committed, in writing, to platform investment after the first two applications, to unit-economics tracking from workload two, and to the shared-platform-plus-embedded-engineers organisational structure. The strategies that do not include those commitments produce programmes that ship the first three workloads beautifully and stall at five.

If you are writing a strategy now, fold the platform-investment commitment into the operating-budget transition path described in the readiness assessment. The platform investment is the single largest line item in the operating-budget transition for most enterprise programmes, and the strategy document that does not name it is the strategy document that runs the team off the funding cliff.

If you are running a programme that has already shipped its first three workloads, the highest-leverage move I can recommend is to spend the next two months on the unit-economics work and the platform extraction, and to defer workloads four and five by a quarter. The two-month deferral feels expensive at the time. It is the cheapest insurance against the workload-twelve crisis that I know how to buy.

The honest signal of a scalable AI programme is that the marginal cost of the latest workload is below the average cost of the first three. The signal of a programme that will not scale is that each new workload costs about what the first one did. The math is dull, but it is the math that decides which programmes get to twenty and which ones freeze at five. As the parent hub put it: strategies are written by people who never touch a terminal; capabilities are operated by people who were not in the room when the strategy was signed. The bridge between them is the unit-economics work, and the work has to start at workload two, not at workload twelve when the CFO finally asks.

Sources

The unit-economics worksheet and the platform-extraction sequencing template are CC-BY-4.0 and linked from the capabilities hub. Methodology: pattern drawn from fractional CTO engagements (2023-2026) on enterprise AI programmes that either reached twenty production workloads or froze before reaching ten. The structural difference between the two outcomes is consistent enough to publish; the individual workload details vary by sector and are abstracted.

Frequently asked questions

What is the most common reason AI adoption stalls at five to ten workloads?
Unit economics. The first three workloads are funded as discretionary investment and the per-workload cost does not get scrutinised. Around workload five the cumulative spend crosses a CFO threshold, the per-workload cost gets calculated for the first time, and it turns out to be roughly three times what the team assumed because nobody had run the math at scale. The programme then either invests in cost reduction (caching, smaller models, batch inference) or it freezes. The teams that invested in the unit-economics work in workload two scale to twenty. The teams that did not, do not.
Should we build a shared AI platform or let each team adopt independently?
Both, in sequence, with the sequence mattering more than most teams realise. The first two to three workloads should be application-led — build the workloads, learn what the shared platform actually needs to do, then build the platform against the empirical pattern. The teams that build the platform first ship a platform optimised for the wrong workloads and spend the next year refactoring. The teams that build five applications first ship a coherent platform in month nine that all five can migrate to. Conway's Law applies — the platform shape will mirror your organisational shape, so design the organisational shape deliberately.
How do you keep model costs down as workload count grows?
Four levers, in roughly the order you should apply them. Caching of common queries and responses (the cheapest and most under-used). Routing to the smallest capable model for each query, not the largest. Batch inference for any workflow that is not latency-critical. Targeted fine-tuning of smaller models to replace large-model calls on high-volume narrow tasks. Most teams reach for the fourth lever first because it sounds sophisticated; the first lever is where the cost reduction actually lives at the twenty-workload scale.
Why does the twenty-first workload become a budget-killer specifically?
Because the marginal cost of the twenty-first looks identical to the marginal cost of the first if the team has not invested in scaling economics. Each new workload pays full retail for model inference, full retail for engineering integration work, and full retail for governance overhead. The teams that scale have brought the marginal cost of the twenty-first workload below the average cost of the first ten through platform investment. The teams that do not, hit a wall where workload twenty-one's business case cannot justify the unchanged marginal cost, and the programme freezes at twenty.