LLM router: model routing strategies, evaluation and fallbacks
The main ways to route LLM requests by task, cost and latency, how fallback chains and learned routers work, how to check quality, and how to handle rate limits and timeouts.
Last updated October 22, 2026
What is an LLM router?
An LLM router decides which language model handles each request. It might send simple tasks to a small, cheap model and hard ones to a stronger model, pick the fastest available provider, or switch to a backup when a provider fails. The goal is the right answer at the lowest cost and latency.
Get started with the LLM Gateway
Sign up and our team will walk you through Perimattic LLM Gateway for your models, providers and budgets.
Sign upWhy route LLM requests to different models?
Models differ widely in price, speed and ability, and most applications send a mix of easy and hard requests. Sending everything to the strongest model overpays for the easy ones; sending everything to the cheapest model fails the hard ones. A router matches each request to a model that is good enough for it.
The FinOps Foundation’s paper Optimizing GenAI Usage lists prompt routing, sending requests to the best-fit model through gateways, semantic routing or threshold rules, among its main levers for AI cost. Routing also improves reliability: if one provider is down or rate-limiting you, a router can send traffic elsewhere.
What are the main LLM routing strategies?
| Strategy | How it decides | Strengths | Weaknesses |
|---|---|---|---|
| Rule-based by task | Fixed rules on route, feature, prompt template or metadata (for example, classification goes to a small model) | Predictable, easy to audit, no extra model call | Needs someone to maintain the rules; misses variation within a task |
| Cost-based | Cheapest model that meets a minimum quality tier for the request type | Directly lowers spend | Only as good as your quality tiers; prices change often |
| Latency-based | Fastest model or provider, using recent measured latency | Better responsiveness for interactive features | Fast is not always good enough; latency fluctuates |
| Fallback chain | Primary model first; on error, rate limit or timeout, the next in the list | Keeps requests succeeding through provider incidents | Backup may cost more or answer differently; retries add delay |
| Escalation (cascade) | Try a cheaper model first; if the answer fails a check, retry on a stronger one | Pays for the strong model only when needed | Failed attempts cost time and tokens; needs a reliable check |
| Classifier or learned router | A small model scores each prompt and predicts which model will answer well enough | Adapts to each request, not just each route | Needs training data and evaluation; harder to explain; adds its own call |
In practice most teams combine them: rules for known tasks, a fallback chain on every route, and a learned router only where traffic is varied enough to justify it.
How do you check that routing does not hurt quality?
- Build an evaluation set of real requests for each route, with expected answers or grading criteria, and score every candidate model on it before routing traffic.
- Set quality guardrails: a minimum score each model must meet on a route, and automatic checks on outputs (valid JSON, required fields, refusals) that trigger escalation.
- A/B test a routing change on a share of live traffic, comparing quality signals such as user ratings, task completion or human review, as well as cost and latency.
- Re-evaluate on model changes. When a provider releases or retires a model, rerun the evaluation before switching.
Tracing and evaluation tools such as Perimattic AI Suite help compare models on the same requests and watch quality after a routing change.
How should a router handle rate limits, timeouts and provider errors?
Providers document how they signal limits and failures. What the two largest say:
| Situation | Anthropic (Claude API) | OpenAI API |
|---|---|---|
| Rate limit exceeded | HTTP 429 rate_limit_error, describing which limit (requests, input or output tokens per minute) was exceeded; wait for the retry-after seconds when the header is present (rate limits) | HTTP 429; limits can be hit on requests or tokens per minute or per day, whichever comes first; Retry-After may be sent and should be treated as a minimum wait (rate limits) |
| Spend or quota limit reached | A tier spend-cap 429 has no retry-after header and keeps failing until access resumes, so retrying does not help | Hard spend limits also return 429; do not retry quota or billing errors |
| Overload or server error | 529 overloaded_error when the API is temporarily overloaded; 500 api_error, retry with exponential backoff (errors) | Retry-After may also appear on 503 overload responses |
| Timeouts | 504 timeout_error; Anthropic suggests streaming or the Message Batches API for long requests | Not covered separately in the rate limits guide; its retry advice is to cap both the number of attempts and the total retry time |
| Do failed requests count? | Not stated on the rate limits page; it does note that a per-minute limit may be enforced over shorter intervals, so short bursts can trigger 429s | Yes: unsuccessful requests count toward per-minute limits, so constant resending does not work |
A sensible router policy, based on that guidance:
- On 429 with a retry-after value, wait at least that long, or send the request to a fallback model straight away if latency matters more.
- On overload, server errors and timeouts, retry with exponential backoff and jitter, then fall back.
- Cap retries and total time per request, so a slow failure does not become a slower one.
- Never retry spend-limit, billing or invalid-request errors; alert someone instead.
- Temporarily stop sending traffic to a provider that keeps failing, and test it again later.
How does Perimattic LLM Gateway route requests?
Perimattic LLM Gateway includes:
- Routing rules that send each request to a model by cost, latency or task.
- Automatic fallback when a provider errors or rate-limits.
- One OpenAI-compatible API in front of OpenAI, Anthropic, Google, Mistral, DeepSeek and open models.
- Cost tracking by team, feature and customer, so you can see the effect of each routing change.
Get started: Sign up and our team will get in touch, or estimate routing savings with the LLM gateway savings calculator.
Related guides and tools
LLM routing, answered
What is the difference between model routing and load balancing?
Load balancing spreads identical requests across copies of the same model or deployment to share capacity. Model routing chooses between different models, based on the task, cost, latency or quality needed. A gateway often does both: it routes to a model, then balances across the providers or keys that serve it.
Does routing to cheaper models hurt answer quality?
It can if you do not measure it. Cheaper models are weaker on some tasks, so route only the request types where an evaluation shows the cheaper model is good enough, and keep monitoring quality after the change. Escalating to a stronger model when the first answer fails a check is a common safeguard.
Should every error trigger a fallback to another model?
No. Rate limits, timeouts, overload and server errors are good candidates for retry or fallback. Errors caused by the request itself, such as invalid parameters, will fail on the backup too, and billing or quota errors need someone to act; OpenAI’s guidance is not to retry those.
Want one API for every model, with costs under control?
Sign up to tell us which models and providers you use, and our team will get in touch to set up the LLM Gateway for your stack.
Sign up