LLM proxy: what it is and when you need one
How an LLM proxy works, how it compares with a gateway and a router, what to log and redact, the main ways to deploy one and the risks to plan for.
Last updated October 22, 2026
What is an LLM proxy?
An LLM proxy is a server that sits between your application and an LLM provider and forwards model requests on your behalf. Because every call passes through it, it can hold API keys centrally, log requests, redact sensitive data, enforce limits and switch providers without changing application code. Feature-rich proxies are usually called LLM gateways.
Get started with the LLM Gateway
Sign up and our team will walk you through Perimattic LLM Gateway for your models, providers and budgets.
Sign upHow does an LLM proxy work?
Your application sends its model request to the proxy instead of the provider. Often the only code change is the base URL, because many proxies accept the OpenAI request format. The proxy then:
- Authenticates the caller with its own key, so provider keys never sit in application code.
- Applies policies: rate limits, budgets, allowed models, redaction of sensitive fields.
- Forwards the request to the provider, translating the format if needed.
- Streams or returns the response, and records tokens, latency, cost and errors.
What is the difference between an LLM proxy, a gateway and a router?
These are overlapping names, not strict categories. The table describes how the terms are generally used; a specific product may do more or less than its label suggests.
| LLM proxy | LLM gateway | LLM router | |
|---|---|---|---|
| Main job | Forward requests and add central controls | Run all LLM traffic through one managed layer | Choose which model handles each request |
| Typical features | Key management, logging, basic limits, format translation | Proxy features plus multi-provider routing, fallback, caching, budgets, cost attribution | Rules, cost or latency logic, or a trained classifier that picks a model |
| Where it runs | As a service in front of providers | As a service in front of providers | Inside a gateway, as a library in your app, or as a hosted service |
| Changes which model answers? | Usually no; forwards as asked | Yes, if routing rules are set | Yes, that is its purpose |
| Best for | Central keys and logs with minimal change | Several teams, apps or providers sharing spend | Cutting cost or latency by matching model to task |
Read more in the LLM gateway overview and the LLM router guide.
What should an LLM proxy log, and what should it redact?
A proxy sees every prompt and response, which makes it the natural place for logs and also a place where sensitive data can pile up. Decide deliberately what to keep.
| Data | Log it? | Why |
|---|---|---|
| Timestamp, model, provider, status code | Yes | Needed for debugging, fallback analysis and uptime |
| Input, output and cached tokens | Yes | The basis of cost tracking |
| Latency (time to first token and total) | Yes | Shows slow models and providers |
| Team, feature, customer, API key ID | Yes | Lets spend be attributed and limits enforced |
| Errors and retries | Yes | Reveals rate limits and provider incidents |
| Full prompt and response text | Only if you need it, with retention limits | Useful for debugging and evaluation, but may contain personal or confidential data |
| Names, emails, phone numbers, account or card numbers | Redact before storing | Personal data in logs widens your compliance scope |
| Provider API keys and secrets | Never | Logs are read by more people than keys should be |
Redaction can happen before the request is sent to the provider (so the model never sees the data) or only before logs are written (so the model sees it but your logs do not). The first is stricter but can change the answer; choose per use case. For tracing and evaluating model behavior over time, see Perimattic AI Suite.
How can you deploy an LLM proxy?
- Central service in your cloud. One proxy cluster that all apps call. Data stays in your network; you run it and keep it available.
- Sidecar or per-app instance. A proxy deployed next to each application. No shared single point of failure, but limits and budgets are harder to enforce across apps.
- Managed or hosted. A vendor runs the proxy. Quickest to start, but prompts and responses pass through a third party, so check where it runs and what it stores.
- Managed on a dedicated instance. A provider runs and maintains the proxy software on an instance used only by you, and handles upgrades and backups. For example, Perimattic offers managed LiteLLM hosting through ManageStacks.
What are the risks of putting a proxy in front of your LLM calls?
- Single point of failure. If the proxy is down, every model call fails, even when all providers are healthy. Run more than one instance, health-check it and decide whether apps should fail open to the provider.
- Added latency. Every request makes an extra network hop, and logging, redaction or cache lookups add processing time. How much depends on your setup, so measure it under real load.
- A concentration of sensitive data. Central logs of prompts and responses are valuable to attackers; restrict access and set retention limits.
- Feature lag. When a provider ships a new API feature, a proxy that translates formats may not support it straight away.
- Another system to operate. Upgrades, scaling and monitoring become your job unless someone runs it for you.
Where does Perimattic LLM Gateway fit as an LLM proxy?
Perimattic LLM Gateway does a proxy’s job and more:
- One OpenAI-compatible API in front of OpenAI, Anthropic, Google, Mistral, DeepSeek and open models.
- Request logging with PII redaction.
- Budgets and rate limits per team, API key or customer.
- Automatic fallback when a provider errors or rate-limits.
- Runs in your own cloud (VPC) or managed.
Get started: Sign up and our team will get in touch, or estimate savings with the LLM gateway savings calculator.
Related guides
LLM proxies, answered
Is an LLM proxy the same as an LLM gateway?
The terms overlap and vendors use them loosely. A proxy is the basic pattern: forward requests and add a few controls. A gateway is usually a proxy with a wider set of features, such as routing across providers, caching, budgets and cost attribution. Many products describe themselves as both.
Do I need an LLM proxy if I only use one provider?
Not always. With one app and one provider, calling the provider directly is simpler. A proxy starts to pay off when several apps or teams share keys, when you need central logs or redaction for compliance, or when you want the option to switch or add providers later without code changes.
Does an LLM proxy work with streaming responses?
It can, but the proxy must pass server-sent events through as they arrive rather than buffering the whole response, or users lose the benefit of streaming. Logging and redaction of streamed output also have to work on partial chunks, so check how any proxy you choose handles streaming.
Want one API for every model, with costs under control?
Sign up to tell us which models and providers you use, and our team will get in touch to set up the LLM Gateway for your stack.
Sign up