Dynamic Model Routing: Cutting LLM Spend Without Cutting Capability

Most enterprise LLM spend is not a model-selection problem. It is a routing problem.

Organisations typically discover this in the same order. A team ships something useful on a frontier model. Adoption spreads. Six months later the monthly invoice is an agenda item, and the first instinct is to swap the model for a cheaper one across the board. Quality drops, the team swaps back, and the conclusion is that the cost is simply the price of the capability.

That conclusion is wrong, because it treats a heterogeneous workload as a single decision. A governed gateway that routes each request to the cheapest model capable of completing that specific task changes the economics without touching the ceiling on quality.

This is a reference architecture for that gateway. It describes what the components do, what the non-obvious failure modes are, and what to measure. It is deliberately not a benchmark: routing outcomes depend entirely on your traffic mix, and any figure quoted without your prompt distribution behind it is decoration.

The shape of enterprise traffic

The reason routing works is that production LLM traffic is not evenly demanding.

Once a system is embedded in real workflows, a large share of requests are deterministic transformations wearing a natural-language interface: extract five fields from this document, classify this ticket into one of nine queues, rewrite this paragraph in a fixed tone, convert this response into valid JSON. These have short contexts, no multi-step reasoning, and a verifiable correct answer.

A much smaller share are the requests the system was actually bought for: open-ended analysis, multi-step planning, tool orchestration, anything where the model has to hold several constraints at once and decide what to do.

Sending both classes to the same frontier endpoint means paying reasoning prices for formatting work. The distribution is the opportunity, and it is why the first thing to build is not a router but instrumentation. You cannot design tiers before you know what your traffic actually looks like. Log prompt length, context size, whether tool calls were requested, whether the output was schema-constrained, and whether a human accepted the result. A fortnight of that data tells you whether routing is worth building at all.

What the gateway does

The gateway is a single ingress point that every model call passes through. Nothing in the application talks to a model provider directly. That constraint is what makes everything else possible.

[ App / Agent ]
       │
       ▼
┌──────────────────────────────────────────┐
│  AI GATEWAY                              │
│                                          │
│  1. Authenticate + tag (team, workload)  │
│  2. Classify task complexity             │
│  3. Route to tier                        │
│  4. Validate output                      │
│  5. Escalate on failure                  │
│  6. Meter + log                          │
└──────────────────────────────────────────┘
       │                    │
       ▼                    ▼
[ Local / small model ]  [ Frontier model ]

Five of those six steps are unglamorous plumbing. They are also the reason the sixth one is safe to switch on.

Critically, the gateway is the natural home for controls that have nothing to do with cost: egress inspection, PII masking, prompt-injection screening, per-team budget enforcement. Teams that build a routing layer and a security layer separately end up with two chokepoints, two sets of credentials, and two incomplete logs. Build one.

Classifying the task

Three approaches, in ascending order of cost and accuracy.

Heuristics. Context length, presence of a tool-call schema, output token limit, whether the prompt matches a known template. Nearly free, surprisingly effective, and brittle in exactly the way you would expect. Good enough to start, and good enough permanently for high-volume templated traffic where the caller already knows the shape of the job.

Embedding similarity. Embed the incoming prompt, compare against labelled clusters of past requests with known outcomes. Handles novel phrasings that heuristics miss. Requires you to have been logging outcomes, which is the argument for instrumenting first.

A small classifier model. A fine-tuned encoder or small LLM that scores complexity directly. Most accurate, most expensive, and it introduces a dependency that itself needs monitoring and retraining.

The pragmatic architecture is layered: heuristics resolve the obvious majority, and only the ambiguous remainder reaches a more expensive classifier. Routing the router is not a joke; it is the same principle applied one level down.

A note on a tempting shortcut: do not classify on caller identity. Routing by team or user is easy to implement and structurally wrong, because the same team submits both trivial and hard requests. It optimises the wrong variable and produces exactly the quality complaints that get routing projects cancelled.

The latency budget

The router runs on every request, including the ones it decides not to reroute. Its cost is therefore paid unconditionally, and it must be small enough to disappear.

Two budgets matter:

Latency. The classifier sits in the critical path. If classification adds more delay than the tier change saves, the user experience degrades while the invoice improves, and the feature gets switched off. Heuristic classification should be effectively free. A model-based classifier needs to be genuinely small and genuinely local; a network hop to another provider to decide which provider to use is a design that will not survive review.

Cost. Router inference is not free either. If a classifier costs a meaningful fraction of the cheap tier it routes to, the arithmetic collapses on high-volume, low-complexity traffic — precisely the traffic where routing should pay off most.

Both budgets argue for the same thing: resolve as much as possible with the cheapest possible signal.

Where it breaks

This is the part that determines whether a routing layer survives its first quarter in production.

Escalation cascades

Routing without an escalation path is not routing, it is downgrading. Every tier needs a defined failure condition — schema validation failure, low self-reported confidence, refusal, timeout — that promotes the request to a higher tier.

The consequence is that some requests are paid for twice. If your escalation rate is 10%, your effective cost is the cheap tier on everything plus the expensive tier on a tenth of it, plus the latency of both in sequence. Escalation rate is therefore not a quality metric to watch casually; it is a term in the cost model. A router that misclassifies 40% of traffic downward can cost more than no router at all, while also being slower.

Prompt caching changes the arithmetic

This is the failure mode most cost models miss.

Frontier providers offer substantial discounts on repeated prompt prefixes. Workloads with large stable system prompts or shared document context can see most of their input billed at a reduced cache-read rate. A router that diverts these requests to a local model loses those cache hits — and the comparison it made was against the uncached headline price, not the price actually being paid.

The result is a router that reports savings while total spend is flat or higher. Any honest cost model has to compare against your realised blended rate, cache behaviour included, not the rate card. Where a workload has a high cache-hit ratio, the correct routing decision is frequently to leave it alone.

Validation failures multiply tokens

Structured output is where naive token optimisation backfires. A cheaper model that returns malformed JSON triggers a retry. Two or three retries, and you have spent more tokens and more wall-clock time than a single call to a capable model.

This is why the unit of measurement has to be cost per successfully completed task, not cost per token. Optimising token count while ignoring retry rate is how teams produce a 30% reduction in tokens and no reduction in spend.

Deterministic validation at the gateway — schema conformance, required fields, type checks — is cheap, catches this class of failure immediately, and drives the escalation decision. Validate before you return, always.

Thresholds go stale

Your prompt mix is not stationary. New features ship, users learn what the system is good at, and a distribution that was 70% simple at launch drifts. Routing thresholds calibrated once are wrong within months, usually in the direction of routing too aggressively downward.

Treat thresholds as configuration with an owner and a review cadence, not as constants. Alert on distribution shift — a sudden change in escalation rate is the signal, and it usually arrives before the quality complaints do.

What to measure

Six numbers. Anything less and you cannot defend the architecture; anything more and nobody reads the dashboard.

Cost per successfully completed task, by workload. The headline metric. Tokens are an input, not an outcome.
Escalation rate, by tier. Rising means the classifier is drifting or the traffic changed.
p95 end-to-end latency, router included. Measured from the caller's perspective, not the model's.
Quality regression rate. Sampled human review or automated evaluation against a fixed held-out set. Without this you have no evidence that savings were not simply purchased with quality.
Realised blended input rate, cache hits included. Guards against the caching trap above.
Spend by team and workload. Requires request tagging at the gateway, which is why tagging is step one and not an afterthought.

Anti-patterns

Routing by identity rather than task. Discussed above. The single most common design error.
No escalation path. Turns a cost optimisation into a silent quality cut.
Optimising tokens instead of task cost. Produces impressive charts and flat invoices.
Comparing against rate-card pricing. Ignores caching and overstates savings, sometimes entirely.
A separate security chokepoint. Two gateways means two incomplete audit trails.
Shipping the router before the instrumentation. You cannot tier a distribution you have never measured.

What this is worth

The honest answer is that it depends on your traffic, and the range across real workloads is wide enough that a single number would be misleading. A workload dominated by extraction and classification has substantial headroom. A workload that is genuinely reasoning-heavy, or that already benefits from high cache-hit ratios, may have very little — and the correct engineering answer in that case is to leave it alone and say so.

What is dependable is the shape of the finding: measure first, tier what is measurably tierable, validate every output, keep an escalation path, and price against what you actually pay rather than the rate card. A gateway built that way makes cost a governable parameter instead of a quarterly surprise, and it puts the security controls you will eventually need in the one place every request already passes through.