The Router Pattern: How Teams Control LLM Cost and Reliability at Scale
•
•
2 min read
Most teams start by wiring one strong model into a workflow and iterating prompts until it feels good. That’s a perfectly reasonable place to begin.
At scale, though, it gets expensive and slow — because you end up using a large model for tasks that are mostly deterministic. The fix is simple:
Route requests. Don’t treat them all the same.
A router is just a decision layer that picks the cheapest path that still meets your quality bar: deterministic logic when it’s enough, a smaller model for routine interpretation, and a stronger model only when the input is genuinely messy.
This isn’t just an “architecture trend.” Projects like RouteLLM treat “which model should answer this?” as a routing problem: given a strong (expensive) model and a weak (cheap) model, it uses preference-style training signals to learn when the cheaper lane is good enough — and when it’s worth escalating — aiming to cut cost while keeping quality close to the strong model.
A helpful way to picture routing: airport security
Think of routing like airport security.
Most passengers go through the standard lane. It’s fast, predictable, and handles the majority of people just fine. Some go through a priority lane. And a small number get pulled aside for manual inspection.
You wouldn’t send everyone through manual inspection. That would be slow, expensive, and unnecessary.
LLM routing is the same idea: most requests don’t need the slowest, most expensive path — but you want it there when something doesn’t look right.
An example most teams run into
Imagine you’re extracting structured fields from a form: name, email, address, totals, IDs.
When the page is predictable, you don’t need an LLM. Straightforward extraction plus normalization and validation will usually get you a clean result quickly and cheaply.
Where things break down is when real-world variability shows up: the layout shifts, labels aren’t consistent (“Email” vs “Contact” vs “Billing Email”), values aren’t close to their labels, or the data is split across odd parts of the DOM.
This is where routing matters. You try the deterministic path first, validate what you extracted, and only escalate when something important is missing or fails validation. At that point, an LLM can help — but in a narrow way — focused only on the ambiguous pieces, returning structured output with clear constraints.
You’re not asking the model to do the whole job. You’re using it as a targeted fallback when the page stops being predictable.
What routers usually look at in practice
Good routers don’t need to be magical. They usually make an explicit call based on a few signals:
- How ambiguous is the input?
- Is there a single correct answer we can verify deterministically?
- What’s the blast radius if we’re wrong?
- Have we seen this pattern succeed cheaply before?
In practice, routers often start as simple rules plus verification, then get tuned over time using logs and thresholds.
The real point
Routing is about being intentional — not throwing a big model at problems code already solves.
Code handles what’s provable. Models handle what’s fuzzy.
And the win is simple: you stop paying the expensive path for easy requests.
Cheap by default. Expensive on purpose.