Not every request deserves an agent

For a while, I sent every user message through the full agent loop: planning, tool calls, a large context, and the top model. It was simpler that way: one path, one architecture, fewer decisions in the code.

The problem showed up where I did not expect it. Cost and latency grew not only on hard tasks, but on the most ordinary messages: “thanks”, “can you make it shorter”, “what can you do”, “hi”. Running a multi-step agent on a request like that is like calling in a crane to lift a spoon.

The fix is boring and cheap: put a small classifier in front of the agent. It looks at the request and decides where to send it: to a template, to a small model, to a direct tool call, or to the full agent loop.

The expensive path should kick in only where it is actually needed.

The naive architecture: one path for everything

The simplest architecture looks like this:

request → agent_loop → response

One route. Always the top model. Always tool-calling. Always a big system prompt and a growing context.

But real traffic is not like that. It contains everything at once:

• greetings;
• thank-yous;
• smalltalk;
• short clarifications like “yes” or “make it shorter”;
• FAQ questions;
• off-topic messages;
• requests that need exactly one known tool.

For all of this, the full agent loop is often overkill.

The cost of the naive path on a simple request comes from three things. First, tokens: the agent’s system prompt, the tool descriptions, and the conversation history — on a “thanks”, that can mean hundreds or thousands of wasted tokens. Then latency: the agent loop is several model turns and round-trips to tools, so even on a simple task the user waits as long as on a hard one. And quality: the more light, junk requests pass through the expensive model, the more its context gets polluted, and on the genuinely hard tasks that starts to get in the way.

The main shift in your head is this: the model tier is not an application constant, it is a routing decision.

Not “I have an agent on the top model”, but “I have several roads of different cost, and I decide which one to send a given request down”.

The cost funnel

The pattern is simple: keep the top of the funnel cheap and wide, and make the expensive path narrow. We handle most of the traffic fast and cheap. Only the requests that truly need it reach the full agent loop.

There are three minimally useful routes:

1. Trivial — a greeting, a thank-you, smalltalk, a simple rephrase. You can answer with a template or a small model, no tools.
2. Single tool — the request needs one clear action. The tool can be called directly, without agent planning.
3. Agent — the task is ambiguous, multi-step, or needs several tools. Here you need the full loop.

You can even start with two routes: cheap and agent. The point is to stop sending all traffic down the most expensive path.

Why it works

The request distribution is almost always heavy-tailed. Most of the traffic turns out to be simple, a smaller part genuinely hard.

You cannot guess the exact percentages up front. You have to measure them on your own traffic: add route logging and look at how many requests really need the agent. When I did this, the cheap requests outnumbered the expensive ones by a wide margin. For an architectural decision that number was enough.

The cost difference between routes is usually an order of magnitude or two. The classifier is one short call to a small model with a small answer: an enum and a confidence. The agent pass is several model calls, a growing context, and tool round-trips.

Caching helps too, but you should not describe it as “almost free”. The classifier’s system prompt is stable, so most of the input tokens can be covered by the prompt cache. But it is not zero: you still pay for output, for the cache write, and you lose the discount once the TTL expires.

A short classifier with a cached prefix is genuinely cheap. Just not free.

You can estimate the average cost like this:

avg = P(trivial)·c_cheap + P(single_tool)·c_tool + P(agent)·c_agent

c_agent dominates, because it is orders of magnitude larger than the rest. If you move even half of the traffic out of agent and into cheap, the average cost drops not by percent, but by multiples.

Latency improves for the same reason. Cheap requests get a fast answer, and the expensive model stops spending time and context on trivia.

The classifier itself

The classifier should not answer the user. Its job is to make a routing decision.

It is useful to classify not just “simple / hard”, but several signals:

• intent category;
• whether a tool is needed;
• which tool is needed;
• difficulty level;
• language;
• a safety flag;
• whether to answer at all.

Often this is not one label, but a small schema.

The classifier can be a small model, an embedding classifier, or a hybrid of rules and a model. Either way, temperature = 0: here you want determinism, not pretty text.

It is better to force the answer to be structured right away. Parsing prose is brittle, while a schema is easy to test.

from enum import Enum
from pydantic import BaseModel

class Route(str, Enum):
    TRIVIAL = "trivial"          # greeting, thanks, smalltalk
    SINGLE_TOOL = "single_tool"  # one known action, no planning
    AGENT = "agent"              # ambiguous or multi-step task

class Triage(BaseModel):
    route: Route
    tool: str | None = None
    confidence: float            # 0..1

async def classify(message: str, recent: list[str]) -> Triage:
    return await small_model.structured(
        system=CLASSIFIER_PROMPT,
        input={
            "message": message,
            "recent": recent[-2:],
        },
        schema=Triage,
        temperature=0,
    )

Then comes the important part: the classifier’s mistakes do not cost the same.

If a hard request is wrongly sent to the cheap path, the user gets a bad answer. If a simple request is wrongly sent to the agent, you just overpay. The first breaks trust in the product. The second breaks only the economics.

So on low confidence you should escalate upward — toward capability, not downward toward cheapness.

triage = await classify(msg, recent)

# When in doubt, choose capability over savings.
if triage.confidence < 0.6:
    return await run_agent(msg)

match triage.route:
    case Route.TRIVIAL:
        return template_reply(msg)

    case Route.SINGLE_TOOL if triage.tool is not None:
        return await call_tool(triage.tool, msg)

    case Route.AGENT:
        return await run_agent(msg)

    case _:
        # single_tool with no named tool — same logic: escalate up
        return await run_agent(msg)

The 0.6 threshold here is just an example. You have to calibrate it on your own data. A model’s raw confidence is usually too optimistic, and a sane boundary only shows up on a golden set.

One caveat: single_tool is a cost decision, not a safety one. A single tool call can still delete, pay, or send — cheap to route is not cheap to get wrong. The routing above is deliberately minimal; before the direct call_tool, gate on a safety signal (one of the classifier signals listed earlier) — run unattended only when it is clearly safe, otherwise confirm or escalate to the agent. Skipping the agent does not mean skipping the guardrails.

Inside a conversation the route depends on context. A “yes” after a hard question is not a trivial request. But the classifier does not need the whole history. The last one or two turns are usually enough. If you give it the entire conversation, it starts to cost as much as the system it is supposed to make cheaper.

And one more important thing: the classifier drifts. As the number of intents grows, the prompt bloats and accuracy starts to wobble. Three practices fix this:

• version the prompt;
• keep a golden set and a confusion matrix;
• do not inflate the taxonomy.

Three or four routes are usually more reliable than fifteen pretty intents.

The classifier is code, not just config. Treat it like code: test it, version it, and look regularly at where it gets things wrong.

What to measure

Without metrics you cannot tell whether the funnel works at all.

The minimal set:

• route distribution: trivial / single_tool / agent;
• cost per request by route;
• average cost per request;
• the share of misroute errors;
• separately, the share of cases where a hard request went down the cheap path;
• the share of escalations due to low confidence;
• p50/p95 latency by route;
• A/B: “everything to the agent” against “the funnel”.

The last point matters most. It shows whether you hurt quality in exchange for savings.

When you don’t need this

This pattern is not free. There are cases where the classifier only gets in the way.

If almost all requests are genuinely hard, the classifier becomes pure overhead: you added a call in front of every request but saved almost nothing.

On the hard path the classifier and the agent run in sequence. So you are adding latency exactly where it is already high.

There are a few ways to soften this:

• keep the classifier very fast;
• block on it only for the cheap path;
• if needed, start the agent speculatively in parallel and cancel the extra work once the verdict comes in.

The classifier also becomes a single point of failure. If it goes down, the whole input must not go down with it. The rule is the same as on low confidence: on any classifier error, fall through to the agent.

Capability beats savings.

Takeaway

The cheapest optimization in an LLM app is not calling the expensive model where it is not needed.

A triage classifier pays for itself quickly: it lowers average cost, speeds up common requests, and reserves the expensive agent loop for tasks that actually need it.

The model tier is a routing decision.

Err toward capability. Measure the funnel. Version the classifier like code.