LLM gateway: one API for every model
Call OpenAI, Anthropic, Google, Mistral, DeepSeek and open models through one OpenAI-compatible API, with routing, fallback, caching, budgets and cost tracking in one place.
Last updated October 22, 2026
What is an LLM gateway?
An LLM gateway is a service that sits between your application and large language model providers. Your code calls one API; the gateway forwards each request to the right model and adds shared controls: routing and fallback, caching, budgets and rate limits, cost tracking and request logs. It is also called an AI gateway.
Get started with the LLM Gateway
Sign up and our team will walk you through Perimattic LLM Gateway for your models, providers and budgets.
Sign upWhat does an LLM gateway actually do?
Without a gateway, every application talks to every model provider directly: its own SDK, its own API key, its own error handling and its own idea of what was spent. A gateway moves those concerns into one shared layer.
- One API. Applications send requests in one format, often the OpenAI chat format, and the gateway translates them for each provider.
- Routing and fallback. Rules decide which model handles each request, and what happens when a provider is down or rate-limits you.
- Caching. Repeated or near-identical requests can be answered from a cache instead of a paid model call.
- Budgets and limits. Spend and request limits per team, key or customer stop one workload from running up the whole bill.
- Cost tracking and logs. Every request is recorded with its model, tokens and cost, so spend can be attributed and problems debugged.
What does Perimattic LLM Gateway include?
Perimattic LLM Gateway brings these controls together behind one API.
| Feature | What it does |
|---|---|
| One API | One OpenAI-compatible API in front of OpenAI, Anthropic, Google, Mistral, DeepSeek and open models |
| Routing rules | Send each request to a model by cost, latency or task |
| Automatic fallback | Retry on another model or provider when one errors or rate-limits |
| Caching | Exact-match and semantic caching of responses |
| Budgets and rate limits | Limits per team, API key or customer |
| Cost tracking | Attribution by team, feature and customer, with alerts |
| Request logging | Logs of every request with PII redaction |
| Deployment | Runs in your own cloud (VPC) or managed by Perimattic |
How is an LLM gateway different from an API gateway or calling providers directly?
A general API gateway and an LLM gateway both sit in front of services, but an LLM gateway understands model requests: tokens, models, prompts and per-token prices. Calling providers directly is the simplest option and is often fine for a single app on a single model.
| Calling providers directly | General API gateway | LLM gateway | |
|---|---|---|---|
| Setup | None beyond each provider SDK | Usually already in place for your APIs | One more service to run or buy |
| Multiple providers | Separate SDK, key and error handling per provider | Can route by path, but does not translate model request formats | One request format, translated per provider |
| Cost tracking | Read each provider’s billing separately | Counts requests, not tokens or model prices | Tokens and cost per request, team or customer |
| Limits | Each provider’s own limits; some let you set them per workspace or project | Request-rate limits | Token and spend budgets as well as request limits |
| Caching | Provider prompt caching only, where offered | HTTP caching, which rarely fits POST model calls | Exact and semantic response caching |
| Fallback | Write it yourself in each app | Generic retries to another upstream | Model-aware fallback to another model or provider |
| Extra hop and failure point | No | Yes | Yes, so it needs to be run for high availability |
| Best for | One app, one provider, early prototypes | Non-LLM APIs, auth and traffic at the edge | Several apps, teams or providers sharing LLM spend |
How do routing, fallback, caching and budgets work in a gateway?
Routing
A routing rule picks the model for each request: a small, cheap model for simple tasks, a stronger one for hard ones, or the fastest available for latency-sensitive calls. See the LLM router guide for the main strategies.
Fallback
When a provider returns a rate-limit or server error, the gateway can retry with backoff or send the request to a backup model. Providers document these errors: both OpenAI and Anthropic return HTTP 429 when a rate limit is exceeded.
Caching
An exact cache returns a stored answer for a byte-identical request; a semantic cache also matches requests that mean the same thing. Both are different from the prompt caching providers run themselves. See semantic caching.
Budgets and cost tracking
Because every request passes through it, the gateway can count tokens and cost per team, key or customer and stop or alert when a budget is reached. The FinOps for AI guide covers the metrics that matter. For tracing and evaluating the quality of what models return, see Perimattic AI Suite.
Should you self-host an LLM gateway or use a managed one?
Self-hosted means the gateway runs in your own cloud account. Prompts, responses and logs stay inside your network, and you control where data is stored. In exchange, your team runs it: upgrades, scaling, monitoring and keeping it highly available, since every model call depends on it.
Managed means a vendor runs the gateway for you. You get started faster and skip the operations work, but prompts and responses pass through a third party, so check where it runs, what it logs and for how long.
If you want LiteLLM in your stack without operating it yourself, Perimattic also offers managed LiteLLM hosting through ManageStacks.
What do the major LLM APIs cost?
A gateway does not change what providers charge per token; it changes which model each request goes to and how many requests reach a model at all. Per-token prices, checked against each provider’s own pricing page, are in our pricing guides:
What are the alternatives to building on an LLM gateway?
You can call providers directly, use an open-source proxy you run yourself, or use a hosted model marketplace that resells many models behind one key. Each has real strengths. We compare them honestly, including where they are the better choice:
Which free tools help you estimate LLM costs?
- LLM gateway savings calculator: estimate what caching and model routing could save on your monthly bill.
- Token calculator: count the tokens in a prompt.
- LLM cost calculator: estimate monthly spend by model.
- LLM observability cost calculator: estimate what tracing and monitoring will cost.
All tools are listed on the LLM gateway tools page.
Where does Perimattic LLM Gateway fit?
Perimattic LLM Gateway is one OpenAI-compatible API in front of OpenAI, Anthropic, Google, Mistral, DeepSeek and open models, with routing by cost, latency or task, automatic fallback, exact and semantic caching, budgets and rate limits per team, key or customer, cost attribution with alerts, request logging with PII redaction, and a choice of running it in your own cloud or managed.
Get started: Sign up and our team will walk you through Perimattic LLM Gateway for your models and providers.
Go deeper
LLM gateways, answered
Is an LLM gateway the same as an AI gateway?
Mostly, yes. Both terms describe a layer between applications and AI model providers that adds routing, security and cost controls. Some vendors use "AI gateway" more broadly to include non-LLM models or agent traffic, but for calls to language models the two terms mean the same thing.
Does an LLM gateway add latency?
Every extra network hop adds some time, and how much depends on where the gateway runs and what it does to each request, such as logging, redaction or a cache lookup. Running it close to your application and to the providers keeps the overhead small, and a cache hit can return faster than a model call. Measure it in your own setup.
How do I get started with Perimattic LLM Gateway?
Sign up on this page and tell us which models and providers you use, roughly how many requests you send and where you want the gateway to run. Our team will contact you to understand your needs, talk through routing, caching and budget rules for your workloads, and set you up.
Want one API for every model, with costs under control?
Sign up to tell us which models and providers you use, and our team will get in touch to set up the LLM Gateway for your stack.
Sign up