Once a candidate model clears the required quality threshold, the choice between session pinning and per-call routing becomes a stateful break-even calculation. A first visit to another model may require that model to process the accumulated prefix at its cache-write or uncached-input rate. But the source model’s cache is not necessarily evicted, and a later return may reuse its longest matching live prefix. The router should therefore compare total expected cost over the remaining horizon, using separate cache state for every model. That comparison must include cache writes, cache reads, uncached input, output tokens, expected retries, and the probability that an earlier cache entry will still be alive when the router returns.
That makes prompt caching part of the routing policy, not a billing detail discovered afterward. Route once when context is long and repeatedly reused. Route per step when context is short, cold anyway, or when the difficulty gap between steps is large enough to pay for the switch.
I use context to mean all input sent to the model on the current call: system instructions, tool definitions, history, and the new request. The prefix is the unchanged opening part of that input. The provider processes a matching prefix once, stores the KV state for that model, then reads it from cache on the next call and processes only the changed tail.
Two routing strategies optimize different units
Pin-to-session routing selects a model when a task begins and keeps all gathering, action, and verification calls on that model. The unit of optimization is the whole task. The router can be simple because it makes one consequential choice, with a guardrail for obviously risky assignments and a fallback when the chosen model cannot complete the work.
Per-call routing selects a model for each step. The unit is a single invocation. A cheap small language model can gather routine facts, a stronger model can perform a difficult action, and another cheap model can verify a structured result. The harness stays in place while model selection moves inside its loop. Capability requirements vary across steps, so the routing decision changes with the work.
Both designs are internally sensible. They simply put the boundary in a different place. Pinning protects continuity of the provider’s cache. Per-call routing exploits variation in step difficulty. The conflict begins when a per-step saving is calculated as if the accumulated context moved between models for free, even though the destination model must process that history again.
A destination model without a live matching cache entry starts cold
An agent usually sends more than the latest user message. Each call can include system instructions, tool definitions, task state, previous tool results, and the conversation transcript. Prompt caching lets a provider reuse work for a matching prefix instead of processing those tokens again at the normal input rate.
The useful object here is the prefix, not the conversation as an abstract thread. A cache hit requires the provider to recognize eligible leading tokens for the same model. The cache is provider-managed, model-specific, and only partly observable to the application. Brick’s current routing documentation makes the operational assumption explicit: if the destination model has no live matching cache entry, it must process the forwarded context at its ordinary input or cache-write rate. The switch does not evict the source model’s cache, though. If the router returns before the entry expires, that model can still reuse its longest matching prefix — in the LiteLLM traces the returning cache stayed warm in 97.4 percent of cases at a five-minute TTL and 99.3 percent at a one-hour TTL. A switch is a cold start for the new model, not an eviction for the old one.
Suppose model A has already consumed a 40,000-token prefix. The next call appends 2,000 tokens. Staying on A bills the eligible 40,000-token prefix at the cache-read rate and charges only the appended 2,000 tokens as fresh input, subject to the provider’s cache rules. Moving to model B does not transfer A’s KV state. On its first visit to that prefix, model B sees a 42,000-token cold prompt and bills the whole thing at its cold-input or cache-write rate. The router did not merely choose a new price for 2,000 tokens. It chose a new price for the history too. The true cost of the switch is the difference: B’s cold prefix against A’s cache-read prefix, not against nothing.
Cache pricing makes the first fill more expensive in some systems. Under Anthropic’s prompt caching prices, a five-minute cache write costs 1.25 times base input, a one-hour write costs 2 times base input, and a cache read costs 0.1 times base input. Reuse behaves like an investment: you first fund a write, then earn that cost back through reads. A first visit to a new model can restart that cycle there, but it does not necessarily discard the earlier model’s cache.
Eligibility matters as well. OpenAI’s current prompt-caching rules are model-specific. GPT-5.6 and later use a minimum cacheable prefix of 1,024 visible input tokens and report the exact eligible boundary. Earlier models generally use a 2,048-token minimum and report cached tokens rounded down to a multiple of 128. The routing policy should therefore use the rules and usage fields of the actual model rather than assume one universal threshold. A routing policy that treats every token in every call as warm invents savings that the billing system never offered.
The practical implication is sharp. Track cached-prefix tokens by model, not merely conversation tokens by task. A session with 60,000 tokens in its transcript can still be cold after a changed prefix. A call whose total eligible prefix is below the model’s caching threshold receives no cache benefit. Cache state is part of the model choice.
Per-call routing matches model capability to each step
Per-call routing earns its keep when the agent loop contains work of uneven difficulty. Gathering, acting, and verifying are labels, but labels alone are too coarse. A verification step might be a trivial schema check or the hardest reasoning step in the task. The router needs a representation of capability and a separate estimate of what the current query demands.
Brick’s mechanism is concrete. The SR1 paper scores each candidate model on six capability dimensions, estimates difficulty for the query, then dispatches with a cost-penalized geometric rule. Capability is not compressed into one league table. The query occupies a location in a multi-dimensional capability space, and cost pushes the selection toward the cheapest model that still clears the estimated requirement.
That geometric step matters. A single global score would assume that a model stronger at one kind of work is proportionally stronger at all others. Six dimensions let the router distinguish differently shaped capability profiles. The difficulty estimate then makes the decision local to this call. Cost is a penalty, not the only objective, so the cheapest model should win only when its capability remains adequate.
The failure modes follow directly from that mechanism. Underestimate difficulty and the selected small model may fail, triggering a retry or fallback that erases the original saving. Overestimate difficulty and calls drift toward expensive models. Score the capability dimensions poorly and the geometric rule becomes precise arithmetic over bad coordinates. Add a model switch without cache state in the cost term and the router optimizes the visible step while billing the invisible prefix.
Per-call routing is therefore strongest when steps are separable. Short prompts, little prefix reuse, or already-cold requests keep the switching penalty small. A wide difficulty spread gives the policy something valuable to exploit. Sending ten routine calls to an inexpensive model can fund one hard call on a stronger model, especially when each call carries little history.
The strategy weakens when the harness replays a growing transcript on every turn. Then model identity becomes part of state. You can still route per call, but the cost function must include the cold rebuild, expected retries, and any later opportunity to reuse the newly created cache. Ignoring those terms is not semantic routing. It is a price-table lookup.
A switch is worthwhile when model-price savings exceed the incremental cache cost
The right comparison is not one rebuild against one saving. It is the total expected cost of staying against the total expected cost of switching, over the horizon you actually expect to run. Write that difference as:
ΔC(H) = E[C_switch(H)] − E[C_stay(H)]
Switch only when ΔC(H) < 0. Each side sums the provider’s billing categories over the remaining calls — cache_write × write_rate + cache_read × read_rate + uncached_input × input_rate + output × output_rate — plus expected retry cost, evaluated separately for the stay path and the switch path, each with its own model’s cache state.
For a single step the difference has a readable shape. If both models use equivalent token accounting and router overhead is negligible, the expression simplifies to:
ΔC ≈ P × (W_B − R_A) + U × (I_B − I_A) + (Y_B × O_B − Y_A × O_A) + ΔE_retry
Here P is the matching prefix that is warm on model A but cold on B, so a switch pays B’s write-or-cold rate W_B on it instead of A’s cache-read rate R_A; the source model’s read price is part of the term, not a free baseline. U is the fresh uncached input for the step, priced at I_B on the new model versus I_A on the old. Y_A and Y_B are the expected output tokens each model produces, priced at O_A and O_B. ΔE_retry is the difference in expected failure-and-retry cost between the two models. The switch pays off only when cheaper input and output on B outrun the extra P × (W_B − R_A) it costs to reprocess the prefix.
This approximation assumes that the cache boundary ends at P and that U is billed as ordinary input. If the provider writes through the new tail, classify those tokens under the cache-write term instead. In production, use the billing categories reported by each provider rather than infer them from prompt length alone.
If the destination model will receive several later calls with the same prefix, its write cost is amortized across those future reads, which lowers the effective W_B. If the router is likely to switch back, price only the portion that the source model can no longer read from a live matching cache. Add a full rebuild only if that cache has expired or the prefix no longer matches. The comparison needs a horizon because a one-call view can misprice switching in either direction: it misses both future cache reuse on B and possible return costs on A.
Even this simplified model makes the point. The prefix term P × (W_B − R_A) grows with task history. The savings terms grow with the work assigned to the cheaper model after the switch. A late switch for one small verification call puts a large number on the prefix side and little on the savings side, so ΔC stays positive and session pinning should win. An early switch before a batch of short, routine calls keeps P small and lets cheaper input and output accumulate across several steps, so ΔC can turn negative and per-call routing can win.
The model-price gap must be large enough to repay the incremental cache cost. A cheaper model with a modest discount needs more assigned tokens to repay it; a large price gap reaches break-even sooner. Anthropic’s 0.1-times cache-read rate makes a heavily reused warm prefix particularly expensive to abandon, while its 1.25-times or 2-times write rate raises the entry cost of the new cache. Those ratios do not prove that pinning always wins. They show why the route cannot be chosen from uncached list prices alone.
There are cases where no calculation is needed. When the eligible visible prefix is shorter than the selected model’s cache threshold, there is no cache hit to forfeit. For GPT-5.6 and later that threshold is 1,024 tokens; earlier models generally use 2,048. With a long, stable prefix read repeatedly on one model, switching for a tiny tail is a bad default. Between those poles, log actual cached tokens, cache writes, uncached input, retries, and model prices. Estimates are useful for policy. Bills are useful for correcting it.
The 80% figure belongs to caching, not routing
The headline number needs careful ownership. A cross-provider prompt-caching study found that caching alone can reduce API cost by roughly 41 to 80 percent and, in its results table, reduce time to first token by roughly 6 to 31 percent on long multi-turn agent sessions, depending on the provider and cache strategy; the study’s abstract summarizes that range as 13 to 31 percent. The 41 to 80 percent range illustrates the scale of caching savings that a poor routing policy can erode. It does not directly measure the loss caused by model switching.
Routing and caching can still compound. Across 95 real agent sessions comprising 8,174 API calls, a LiteLLM benchmark using provider cache accounting measured auto-routing plus caching at 37.4 percent cheaper than caching on one frontier model. The benchmark accounts for both mechanisms and keeps their combined saving separate from the caching-only range.
I would reject both slogans. “Always pin” discards step-level price differences that exist between models. “Route every call for up to 80 percent savings” borrows a caching result to market routing. The more precise claim is this: cache-aware routing can beat a pinned frontier model when savings from cheaper models exceed incremental cache, failure, and retry costs.
A minimal router needs memory of the bill
Start with a simple router, a capability guardrail, and a fallback. Then give it the state required for the decision: the current and candidate models; a per-model prefix fingerprint or breakpoint; eligible cached-token counts; last cache use and TTL or expiry estimate; expected remaining calls, input tokens, and output tokens; input, output, cache-read, and cache-write rates; router overhead; and observed failure and retry rates.
The operational rule is simple: compare the expected cost of staying with the expected cost of switching over the relevant horizon. On the switch path, price the destination model’s actual cache state, including any cold or cache-write tokens, cheaper step tokens, router overhead, and expected retries. On the stay path, price the source model’s remaining cache reads, fresh input, output, and expected failures. Keep the session pinned when the switch path is more expensive; switch only when the expected saving clears that difference by a sufficient margin.
Pin-to-session is the sensible default for long accumulated context, stable prefixes, repeated reuse, and small differences in step difficulty. Per-call routing fits short or naturally isolated steps, cold prompts, large capability variation, and a price gap big enough to repay the incremental cache cost. A hybrid can pin within a phase and reconsider at a clean boundary where the prefix is small or was going cold anyway.
That is the design I would ship: not allegiance to one camp, but a router that understands each model’s capabilities and remembers what switching costs.