How it works
Model routing sits in front of a model call and decides which model handles it. The router reads a signal — the task type, an estimated difficulty, a token or cost budget, a latency ceiling, or an explicit tag from the calling stage — and maps that signal to a model. The mapping can be a static table (planning goes to a strong model, extraction goes to a cheap one), a small classifier that scores difficulty, or a cascade that starts cheap and escalates on failure.
A typical routing loop:
- Classify the unit of work — a plan step, a tool call, a summary, a judgment.
- Look up or predict the model that clears the quality bar at the lowest cost.
- Dispatch, capturing the model ID, cost, and latency alongside the output.
- Optionally verify the result, and re-route to a stronger model on failure.
The router is a control point, not a model itself. Keep it deterministic wherever you can: a lookup table you can read and diff beats a learned policy you cannot inspect. Route on properties you can observe before the call, and record every decision so a trajectory can be replayed and audited later.
Why it matters in an agent harness
An agent loop that pins every call to one frontier model is easy to reason about and expensive to run. Most steps in a real loop — parsing a tool result, formatting output, classifying intent — do not need the strongest model. Routing turns model choice from a hardcoded default into a governed decision, and that has direct engineering consequences.
Cost and latency become tunable. You spend capability where it moves the outcome and save it everywhere else, without rewriting the loop. The budget lives in the router, not scattered across call sites.
Failure containment improves. A cascade that starts cheap and escalates gives you a natural place to bound retries: cheap attempt, verify, escalate once, then stop. That keeps a stuck step from silently burning the strongest model in a loop.
Observability sharpens. When each decision records why a model was chosen and what it cost, you can answer questions operators actually ask — which stage drove spend, whether a downgrade caused a regression, where latency accumulated. A route you cannot explain after the fact is a route you cannot trust.
Evaluation gets a target. Routing is only safe if the cheaper model is genuinely good enough for the work you sent it. That claim has to be measured, not assumed. The router and the eval suite are two halves of the same control: one makes the choice, the other proves the choice held.
Model routing vs model fallback
Both send work to more than one model, but they decide at different times, and the difference changes how you wire the loop.
| Model routing | Model fallback | |
|---|---|---|
| When it decides | Before the call, from observable signal | After a call fails or is rejected |
| Primary goal | Match capability to cost per step | Recover from an error or refusal |
| Cost profile | Predictable, budgeted upfront | Extra calls only on failure |
| Main risk | Misclassifying a hard task as easy | Masking a systemic failure with retries |
Routing is a prediction: you commit to a model based on what you know before running. Fallback is a reaction: you escalate because something already went wrong. A robust harness usually uses both — route upfront to control the common case, fall back to contain the exceptions — but they need separate budgets and separate logging, or a fallback storm will hide inside your routing metrics.
The Rifty take
We optimize for routing decisions you can read, replay, and defend, not for the cleverest policy. A deterministic table plus an eval that proves the cheap path holds beats an opaque learned router that saves a few cents and costs you an explanation. We accept slightly higher spend to keep the decision legible, and we enforce one boundary hard: a downgrade is only allowed on work an eval has shown the smaller model can carry.
Common failure modes
- Silent capability cliff. A task drifts harder over time; the router keeps sending it to the cheap model because the routing signal never updated. Re-check routes against evals, not against the day you set them.
- Unbounded escalation. Fallback with no retry cap turns one hard step into a loop that quietly runs the most expensive model repeatedly. Cap the escalation and record it.
- Untracked decisions. If the model ID and cost are not stored with the output, you cannot attribute a regression to a route, and the whole system becomes unauditable.
- Routing on invisible signal. Deciding on something you only know after the call forces a second call and defeats the point. Route on what you can observe first.
- Nondeterministic classifier at the front door. A learned router that answers differently on identical input makes the whole loop harder to reproduce. Prefer a stable mapping, and treat any model-based router as a component that itself needs evaluation.
Frequently asked questions
When should I route to a smaller model instead of the frontier one?
Route down for work an eval has shown a smaller model handles reliably — classification, extraction, formatting, and routine tool-result parsing. Keep the strong model for planning, judgment, and multi-step reasoning. The rule is evidence, not intuition: downgrade only where measurement proves the cheaper model clears the bar.
How is model routing different from model fallback?
Routing decides before the call, from signals you can observe upfront, to match capability to cost. Fallback decides after a failure, escalating to recover. Routing controls the common case predictably; fallback contains exceptions. Most harnesses use both, but give them separate budgets and logs so fallback retries do not hide inside routing metrics.
Should the router itself be a model?
Prefer a deterministic router — a lookup table or simple rules on observable signals — because you can read, diff, and replay it. A model-based classifier can route when difficulty is hard to predict, but it becomes another nondeterministic component that needs its own evaluation and adds latency at the front door of every call.
What breaks most often in a routing setup?
Silent capability cliffs, where a task drifts harder but the route never updates, and unbounded escalation, where uncapped fallback runs the expensive model in a loop. Both stem from the same gap: decisions and their costs are not recorded, so no eval catches the drift and no cap stops the spend.
How do I keep routing from hurting output quality?
Tie every route to an eval that proves the chosen model is good enough for that work, and re-run those evals as tasks change. Record the model ID and cost with each output so you can attribute any regression to a specific route. Routing without measurement is just a cost cut you cannot defend.