Cost architecture study · production LLM economics

What a Production Agent Fleet Actually Costs

A 53-section brief asked for a router that sends each call to the cheapest model that is quality-indistinguishable from the frontier one. Before designing it I decomposed seven days of production traces and thirty days of warehouse logs. Cache writes were 62.9% of spend; model price was a multiplier on a bill whose shape was set by context handling. Perfectly routing every easy turn would have saved about 1%. So I reframed the unit of routing from which model to what execution budget, and recommended roughly 150 lines of middleware instead of the router service I was asked for.

2026economicsevaluationplatformagenticleadership

What I owned

Sole author of the analysis, the architecture recommendation, and the executive evidence pack behind it.

open the diagram →

Cache writes as a share of spend: 62.9%observedCache writes as a share of spend
Ceiling on the briefed approach: ~1%calculatedCeiling on the briefed approach
Cache defect found by diffing two requests: $27,163/yrcalculatedCache defect found by diffing two requests

10components
2request paths
3load stages
6decisions, argued
4failures, owned

Built onLangSmith tracesSAP HANA logsVertex AI

01Problem 02My role 03Architecture 04Decisions 05What broke 06Impact

The problem

What was actually wrong

A production multi-agent platform was running a frontier model across every agent in the fleet. Spend was growing with adoption and nobody could say which part of it was avoidable. The proposed fix, written up as a detailed brief with 26 numbered questions, was a per-request router: classify each call's difficulty, send the easy ones to a cheaper model, keep frontier quality on the hard ones.

It is a reasonable-sounding proposal, and it is the one almost every organisation reaches for. The brief invited disagreement in one of its sections. I took that seriously enough to check the premise before building anything.

Before this existed

Cost was tracked as a single monthly number with no decomposition. The warehouse recorded token counts but no cache fields, so the largest component of the bill was invisible to every internal dashboard.

Constraints

The conditions the design had to hold under

Constraints are the interesting part of an architecture. Without them any diagram looks reasonable.

The obvious comparison was impossible: The mid-tier model everyone assumed was 'nearly as good' was blocked by an organisation-level allow-list and had never once run against this workload. There was no data, anywhere in the org, on how it performed here.
No trustworthy quality baseline: Only 0.78% of production traces carried any user feedback. The brief's constraint, do not degrade current quality, presumed a quality level that had never been measured.
62.9% of the bill was not in the warehouse: Token extraction summed root-graph messages only and recorded no cache-read or cache-write fields. The whole analysis had to be reconstructed from tracing data instead.
Prompt caches are per-model: Any mid-thread model switch discards the warm cache and forces a cold write of the full prompt, which averaged 45,659 tokens. This makes model switching an architectural constraint, not a config choice.

My role

Sole author of the analysis, the architecture recommendation, and the executive evidence pack behind it.

Designed

The execution-budget model that replaced per-request model routing
The middleware placement, argued from ordering constraints against the existing stack
The three-axis task taxonomy: capability required × execution shape × blast radius
The evidence-label taxonomy governing which claims could appear on an executive slide

Built

The cost decomposition over 25,452 production LLM runs and 9,553 warehouse turns
The cache-state classifier that separated unavoidable cold writes from cache busts
The switching-cost model that priced a mid-turn model change in calls
The audit of the platform's existing evaluation suite

Led

Four written disagreements with the brief owner, delivered on request and in full
A recommendation for less engineering than was asked for, with the saving quantified
An executive deliverable governed by a rule that no slide may carry a claim absent from the evidence table
Escalation of an organisational policy ticket as the programme's highest-value action

Architecture

How it is put together

The recommendation was one new component. A single fail-open middleware resolves an execution budget (model, reasoning effort, tool ceiling, context policy) once per cache-cold boundary, from free deterministic signals, subject to a hard blast-radius floor, logging every decision whether or not it is enforced.

The same architecture, in prose

A turn arrives at the agent runtime and passes through an existing middleware chain before reaching the model.

The chain runs in a fixed order: history summarization, then access resolution, then thinking-repair, then (the only new component) the budget router, then prompt-cache breakpointing, then cache metrics, then the model call.

The router's placement is argued from four ordering constraints. It sits after access resolution so routing can never see or influence pre-access state. It sits after thinking-repair so it operates on a well-formed request. It sits before prompt-cache so cache breakpoints land on the request the router actually produced. It sits before cache-metrics so telemetry attributes cost to the chosen model.

The router reads deterministic signals (turn boundary, delegation depth, domain, declared blast radius), and emits an execution budget. On every error path it emits nothing, which produces a request byte-identical to today's.

Every resolution is written to a decision log regardless of whether it was enforced. That log is what makes shadow evaluation free.

Every path, written out (2)· the walkthroughs above, as text

Resolving a budget

A planner asks a broad question that starts a new turn.

01
A cache-cold seam
The turn begins. No warm cache exists for this thread yet, so a model choice here costs nothing extra.
Routing is only free where the cache is already cold. Everywhere else it costs 2.26 calls of cold write.
Fails by: If this were mid-loop, the switch would need the cheaper model to survive 3.4 to 9.1 consecutive calls to break even, against an average of 11.4 calls per turn.
02
Access resolves first
The caller's role-scoped data view is resolved and pinned into the request.
The router is deliberately downstream of this, so no cost decision can ever observe pre-access state.
03
Budget resolution
Free deterministic signals (turn boundary, delegation depth, domain, blast radius) resolve to an execution budget: model, reasoning effort, tool ceiling, context policy.
Model is one field of four. The effort dial and the tool ceiling attack output length and loop length, which is where the money actually is.
Fails by: Any error, timeout or unrecognised task class returns no budget at all.
04
Blast-radius floor applied
The domain's declared floor is applied. It can raise the budget; it can never lower it.
A classifier being right 95% of the time is not good enough when the other 5% is a committed delivery date.
05
Cache breakpoints placed
Cache-control markers are placed on the request the router produced.
Ordering matters: breakpoints computed before routing would cache a prefix that no longer exists.
Fails by: A breakpoint placed on a moving tail message caches a prefix the next turn will not end at, the defect that cost $27k a year.
06
The decision is logged either way
The resolution is written whether or not enforcement is switched on.
This is what makes shadow evaluation free. You can measure what the router would have done for weeks before it does anything.
07
The call goes out
The provider request is sent with whatever budget survived.
Price is a multiplier on a bill whose shape was already set upstream.

Failure path: the router breaks

The budget table is unreachable, or the task class is unrecognised, or the resolver throws. A router that can make the system worse than not having it is not shippable.

01
Resolution fails
The resolver raises, times out, or encounters a task class it has no entry for.
All three are treated identically. There is one error path, not three.
02
Return no budget
The router returns nothing. No effort setting, no model override, no tool ceiling.
Returning nothing means sending no output_config, which produces a request byte-identical to today's. This is the value returned on every error path.
03
Log the failure as a decision
The failed resolution is logged like any other, with its reason.
A router that fails silently is a router nobody can debug. The failure rate is itself a shadow-mode metric.
04
Today's behaviour, exactly
The call proceeds with the platform's existing defaults.
Fail open, always. The worst outcome of a total router outage is that we pay what we pay today.

What changes as load grows· the scale stages, as text

One turn

code

1 model call · cache warm

The middleware stack runs in order on a single turn: access resolution, thinking repair, summarization if the history warrants it, then the budget router picks a tier and the call goes out against a warm prompt cache. The decision log records which tier was chosen and why.

Gives first: Nothing. The stack adds bookkeeping, not latency; the model call is the whole cost.

The whole agent fleet

observed

~3,600 model calls/day across every agent

Cost stops being a per-call question and becomes a distribution. The tier mix is the only lever that moves the total, so the router's defaults matter more than any single agent's prompt. Cache breakpoints have to be placed where the shared prefix actually is, because a breakpoint in the wrong place makes every call a miss and nothing warns you. Summarization runs on the long-lived agents constantly, and its own model call is now a visible line item.

Gives first: Prompt cache hit rate. It is the difference between the fleet's bill and several times the fleet's bill, and it degrades silently, which is why cache metrics are a component in the diagram rather than a dashboard someone remembers to check.

A spike with a cold cache

modelled

burst traffic · shared prefix churning

A prompt change, a deploy or a new agent invalidates the shared prefix and the fleet drops to cold-cache pricing at the worst possible moment. The blast-radius floor is what stops this from becoming an incident: it caps how far the router may downgrade, so a cost spike degrades quality by a bounded amount instead of routing everything to the cheapest model and quietly wrecking output.

Gives first: The floor itself is the tradeoff: it guarantees the spike costs money rather than correctness. Raising it protects quality and raises the bill; lowering it does the reverse. There is no setting that avoids both.

Decisions

The calls I would defend

Each one with the alternatives I rejected, what the choice cost, and how it turned out.

Decision

The unit of routing is not which model answers this question. It is what execution budget this task gets: a named tuple of model, reasoning effort, tool ceiling and context policy.

Context

The brief assumed a single difficulty axis feeding a model choice. Decomposing the bill showed model price was the smallest of the four levers that actually move spend.

Alternatives, and why not

Per-request model routing, as briefedTurns under 50k tokens are 25.6% of traffic and 1.04% of spend. Perfectly routing every one of them to a free model saves about 1%.
A cascade: try cheap, escalate on failureIn an 11.4-step loop you pay the wasted cheap trajectory, plus a $0.2854 cold cache write, plus the full frontier run.
A learned router trained on paired comparisonsNeeds at least three usable models and thousands of paired quality comparisons. We had roughly one usable model and zero comparisons. Revisit in 9–18 months, possibly never.

Rationale

Ranked by measured value, the levers are: an effort dial per task class (14–36%), tool-result admission control (10–25%), an orchestrator context diet (10–20%), budget routing at seams (8–15%), and per-call model routing (~1%). Three of the top four have nothing to do with which model is called.

What it cost

Budgets are coarser than per-request decisions and will occasionally give an easy task an expensive budget. That waste is bounded and measurable; the alternative's downside, degrading a high-stakes answer, is neither.

Outcome

Modelled at $110–125k/yr from levers 1–3 alone, against a $167k/yr baseline, without changing a single model. Most of the achievable saving does not require changing a single model.

What I would do today

I would still separate the axes, but I would build the effort dial first and ship it alone. It is the single highest-value change and it does not need the rest of the design to exist.

What it moved in the diagram

Budget router

What broke

Failures, and what they changed

Every one of these is a thing that went wrong in a system I own. They are here because the architecture is largely a record of them.

A moving cache breakpoint, worth $27k a year

What happened: 21.3% of frontier spend was cold writes on calls where a warm cache demonstrably existed. 1,999 of the 2,248 busted calls happened within ten seconds of the previous call in the same scope. Genuine TTL expiry explained 14 of them.
Root cause: I diffed two consecutive calls five seconds apart. System message byte-identical, tool definitions the same md5, message prefix identical, yet cache_read was 0 and cache_creation was 88,579. Only one cache-control marker was present, on the tail message. The static system breakpoint was silently returning nothing, because the system prompt arrives as the first message rather than as a system message. A single moving breakpoint caches a prefix that the next turn no longer ends at.
What I did: Repriced the busted tokens at read rate rather than write rate to size the defect: $522/week, $27,163/year, with 93% of busts in the orchestrator. Reported the anomaly as observed, the saving as calculated, and the mechanism as a leading hypothesis pending a one-line fix and an A/B.
What changed in the architecture: A static breakpoint that anchors to the true prefix, plus cache-read and cache-write fields recorded in the warehouse so this class of defect is visible without reconstructing it from traces.
What it taught me: This one defect is worth more than the entire realistic model-routing lever. It was found by looking at two adjacent requests rather than at an aggregate. And it must not be presented as already fixed: observed, calculated, hypothesised are three different words.

The evaluation suite returned green on everything

What happened: The platform had over 400 evaluation scenarios. Before trusting them to gate a cost programme, I audited the assertions. 265 of 271 asserted nothing.
Root cause: Scenarios had accumulated as descriptions of intent rather than as checks. A suite that cannot fail looks identical, on a dashboard, to a suite that always passes.
What I did: Marked the legacy suite as unusable for gating, and made building a real golden set the first deliverable of the cost programme rather than a later phase.
What changed in the architecture: Never gate on the legacy suite. Quality gating moved to a purpose-built golden set with assertions that can fail.
What it taught me: Scenario count is a vanity metric. Auditing your own safety net before you rely on it is the actual work, and doing it before a cost programme, rather than after a regression, is the difference between a finding and an incident.

A dead model in the library boots cleanly, then 400s on every request

What happened: Key validation checked library membership but not availability. Two of nine library entries were dead. A configuration change pointing at one would start the service successfully and then fail every request.
Root cause: Validation confirmed the name existed, not that the endpoint answered.
What I did: Raised as a P0 established by measurement rather than by design proposal: it was already broken in production, independent of the routing programme.
What changed in the architecture: A liveness gate at startup, so a model that cannot be called cannot be selected. This matters more once a router can move traffic between models.
What it taught me: Any component that widens the set of models in the request path inherits every unvalidated entry in the library.

The internal pricing table overstated the frontier model by 3×

What happened: The pricing table carried the frontier model at $15/$75 against a real $5/$25. Every cost figure it had ever produced was wrong in the same direction.
Root cause: A price table updated once at integration and never since, with no test tying it to the published rates.
What I did: Corrected the table and re-derived every figure in the study from it, which is why the analysis reports the frontier model as cheaper than the organisation believed.
What changed in the architecture: Prices moved to a checked, dated artifact rather than a constant.
What it taught me: The direction of this error is the interesting part. It had been making frontier models look expensive for months, which is precisely the belief the programme was built on.

Evaluation

How I knew whether it worked

The study is itself an evaluation artifact. Its methodology is the deliverable, and every claim in the executive pack is bound to an evidence label and a source row.

Four evidence labels, applied per claim

Observed: counted from traces, no modelling. Calculated: arithmetic on observed values and published prices. Projected: requires an assumption about future behaviour, always given as a range, never stated as a result. External: cited. The rule governing the deck was that it must not contain a claim that is not in the evidence table.

The programme's own success criterion was unmeasurable

Only 33 of 4,240 root traces carried user feedback, so cost per successful outcome could not be computed. That was presented as the programme's principal risk and converted into a leadership ask, rather than filled in with an estimate.

A tempting number kept off the savings slide

192 repeated question texts accounted for $688, or 22.5% of the week, an obvious deduplication win. It went in the appendix with a caveat instead, because supply-chain data changes hourly and a correct answer at 09:00 may be a wrong answer at 11:00. It is a signal about short-window result reuse, not a bankable saving.

Shadow mode before enforcement

Because every budget resolution is logged whether or not it is enforced, the router's decisions can be evaluated against production for weeks before a single request changes. Gating on measurement is built into the component rather than added as a rollout phase.

Five open questions, published

The document ends by listing what it could not establish: approximate vendor pricing on one provider, an assumed but unverified list-price parity, an unmeasured effort-to-trajectory effect (so the 26% planning figure should not be booked until it is), unknown mid-tier model capability, and an untested setting at quarter-end peak.

Security and safety

What the system refuses to do

Cost optimisation touches the request path, so two of its most attractive options were rejected on security and integrity grounds rather than on cost.

Semantic caching is a cross-tenant leak

Beyond the ≤3% ceiling on savings, a semantic cache keyed on question similarity is access-control-blind. Two users with different data entitlements asking a similar question would share a cached answer. Rejected on that basis alone.

Routing sits downstream of access resolution

The middleware ordering is a security constraint, not an implementation convenience. Placing the router after access resolution means a cost decision structurally cannot observe or influence pre-access state.

Third-party routing services were ruled out

A commercial router cannot see our cache state or our access policy, and using one would mean sending supply-chain prompts to another party.

Blast radius is a floor

High-stakes domains are never routed down, whatever any classifier says. The floor is applied after resolution and can only raise the budget.

Impact

What changed, and how it is known

Every figure carries its basis. Nothing here is rounded up, and nothing modelled is presented as a result.

Where the money actually went

observed

Decomposed from production traces. The brief assumed the lever was which model served the request; almost two thirds of the bill was the cost of writing prompt-cache entries, which no model substitution touches.

Cache writes: 62.9%
Output tokens: 20.8%
Cache reads: 12.7%
Fresh input: 3.6%

The spend is tail-heavy

observed

Three quarters of the cost sits in roughly a quarter of the turns. A fleet-wide model policy is the wrong instrument for a distribution shaped like this; a budget that binds on the tail is the right one.

Share of spend76.5%

Share of turns27.8%

62.9%observed

Cache writes as a share of spendAgainst 20.8% output, 12.7% cache read, 3.6% fresh input

~1%calculated

Ceiling on the briefed approachPerfect routing of every sub-50k-token turn to a free model

$27,163/yrcalculated

Cache defect found by diffing two requests2,248 busted calls repriced at read rate rather than write rate

76.5% of cost in 27.8% of turnsobserved

Spend concentrated in the tail

25–34%modelled

Modelled reduction with no model change$167k/yr baseline to $110–125k/yr from the top three levers

265 of 271code

Eval assertions that asserted nothing

~150 lines, one tablecode

Recommended implementationAgainst a router service, a classifier and a new subsystem

Business: A cost programme was redirected from a lever worth about 1% to levers modelled at 25–52%, and a $27k/yr defect unrelated to the original question was found and sized along the way.
Engineering: The recommendation was materially less engineering than was asked for: one fail-open middleware and one table, no router service, no classifier, no cascade, no new persistence. A costed do-not-do list keeps the rejected options rejected.
People using it: No user-visible change was proposed until a quality measurement existed that could detect a regression. That sequencing was the recommendation's main constraint.

Leadership and hindsight

What I influenced, and what I would change

Technical leadership

Four written disagreements with the person who set the brief, delivered because the brief asked for them and answered in full rather than diplomatically.
Recommended less work than was asked for, and quantified what the saving from the simpler design would be, so the recommendation could be checked rather than trusted.
Named a one-day organisational policy ticket as the highest-value action in an engineering programme, the correct answer even though it was not an engineering answer.
Produced an executive evidence pack governed by a rule that the deck may not contain a claim absent from the evidence table.
Converted the two questions I could not answer into leadership asks rather than estimates.
Published a do-not-do list with costed reasons, so that rejected options stay rejected after I am no longer in the room.

What I would do differently

I would ship the effort dial on its own, first. It is the largest single lever, it needs about 60 lines, and bundling it into a wider architecture recommendation slowed it down.
I would instrument cache fields in the warehouse before doing any of the analysis. Reconstructing 62.9% of a bill from tracing data was several days of work that a schema change would have made unnecessary.
I would run the eval-suite audit at the start rather than at the point where I needed the suite. It changed what the first deliverable had to be, and finding that out earlier would have changed the plan.
The one thing I would not change is refusing to estimate the quality baseline. The pressure to put a number there was real, and the document is more useful for not having one.

Go deeper

The detail, for people who want it

Collapsed by default. The case study stands without any of it.

Why per-call routing is negative expected value

Prompt caches are per-model. Switching model mid-thread invalidates the warm cache and forces a cold write of the entire prompt, which averaged 45,659 tokens on this workload.

Priced out, that switch costs the equivalent of 2.26 calls. For the switch to pay for itself, the cheaper model has to survive 3.4 consecutive calls at the cheapest tier, 5.3 at the current mid-tier, or 9.1 at the previous mid-tier.

The measured average is 11.4 calls per turn, but those calls are not independent decisions. A per-call router re-evaluates at every step, so the expected run length between switches is far below the break-even threshold at every tier.

The synthesis that follows: each agentic step admits roughly 12,700 new tokens into a cached context at 1.25× input price, and does so 11.4 times per turn. The cost driver is context admission, not price per token.

The three-axis task taxonomy

The brief assumed one axis: difficulty. The data shows the axes do not correlate, so collapsing them loses the information that matters.

Capability required: how much reasoning the task genuinely needs.

Execution shape: how long the loop runs and how much context it admits. A bulk extract is low-capability but huge-shape.

Blast radius: what happens if the answer is wrong. This is not a scoring factor; it is a floor.

A single difficulty score would route the bulk extract to a cheap model because it is easy, and then pay for its enormous context on that model anyway. Separating shape from capability is what makes the effort dial and the tool ceiling independently useful.

How the cache-state classification was built

Every large-prompt call in the seven-day window was classified by whether a warm cache should have existed within its own trace and checkpoint scope.

Warm, cache read: 15,093 calls, $1,136.74, 37.3%. Cold and first in scope, therefore unavoidable: 4,770 calls, $1,205.85, 39.5%. Cold bust, a warm cache existed and the prefix changed anyway: 2,248 calls, $648.82, 21.3%. Cold from genuine TTL expiry: 14 calls, $10.22, 0.3%.

That last row is the finding. If TTL expiry explains 14 calls and 2,248 are busting, something is changing the prefix, not the clock. Nineteen hundred and ninety-nine of them happened within ten seconds of the previous call in the same scope.

The classification is what turned an aggregate, 'cache writes are 62.9% of spend', into an actionable split between the part that is physics and the part that is a bug.

Stack

What it is built on, and why that

A technology list without reasons is a list of things I have heard of.

LangSmith traces: The primary cost dataset: 25,452 LLM runs across seven days.The warehouse recorded no cache fields, so tracing data was the only place the largest component of the bill was visible at all. That gap became a finding in its own right.
SAP HANA logs: Thirty days of turn-level data (9,553 turns, 4.496 billion tokens) for distribution and tail analysis.Long enough to characterise the tail, which is where three-quarters of the spend lives, and independent of the tracing sample.
Vertex AI: Verifying live that the middleware seam the design depends on actually behaves as documented.The recommendation rests on a specific ordering property of the framework. I read the framework source and then confirmed it against the live provider rather than trusting either alone.

6 more sections are written and hidden: constraints, evaluation, safety, leadership, deeper, stack.