Semantic caching for LLMs: how it works and when to use it
How exact and semantic caching differ, how similarity thresholds work, the false-hit risk, what you should never cache and how provider prompt caching from OpenAI and Anthropic fits alongside.
Last updated October 22, 2026
What is semantic caching?
Semantic caching stores LLM responses and reuses them for new requests that mean the same thing, even when the wording differs. Each prompt is turned into an embedding vector; if a stored prompt is similar enough, above a threshold you set, the cached answer is returned instead of calling the model, saving its cost and time.
Get started with the LLM Gateway
Sign up and our team will walk you through Perimattic LLM Gateway for your models, providers and budgets.
Sign upWhat is the difference between exact and semantic caching?
An exact cache stores each response under a key built from the full request (model, messages and parameters). Only a byte-identical request gets a hit. A semantic cache compares meaning, so “How do I reset my password?” and “I forgot my password, how do I change it?” can share one stored answer.
| Exact-match cache | Semantic cache | |
|---|---|---|
| Matches | Identical requests only | Requests whose embeddings are similar above a threshold |
| Lookup | Hash the request, look up the key | Embed the prompt, search a vector index for the nearest stored prompt |
| Extra cost per request | Negligible | An embedding call and a vector search on every request, hit or miss |
| Hit rate | Low unless requests repeat word for word | Higher on natural-language traffic with many rephrasings |
| Risk of a wrong answer | Very low; the request is identical | Real: similar wording can have a different meaning |
| Good for | Repeated system calls, fixed classification prompts, retries | FAQ-style questions, support bots, search-like queries |
How do similarity thresholds work in a semantic cache?
- The incoming prompt is converted into an embedding: a list of numbers that represents its meaning.
- The cache searches stored prompt embeddings for the nearest match, typically by cosine similarity.
- If the similarity score is above your threshold, the stored response is returned.
- Otherwise the request goes to the model, and the new prompt and response are stored for next time.
The threshold is the main dial. Set it higher and you get fewer hits but fewer wrong answers; set it lower and you get more hits and more risk. The right value depends on your embedding model and your traffic, so tune it on real requests rather than copying a number from elsewhere.
It usually helps to embed only the part of the request that carries the question (for example, the last user message) and to include the model, system prompt version and tenant in the cache key, so a change to any of them does not return an outdated answer.
What is a false cache hit, and how do you prevent one?
A false hit is when two prompts score as similar but need different answers. “Cancel my order” and “Don’t cancel my order” can look close to an embedding model. “What is the refund policy for the US?” and “…for the UK?” differ by one word that changes the answer.
- Start with a strict threshold and loosen it only with evidence.
- Log every hit with its similarity score so wrong answers can be traced.
- Review a sample of hits by hand, and track complaints or thumbs-down on cached answers.
- Set an expiry time on entries so answers refresh when facts change.
- Cache only routes where you have checked the behavior, not all traffic at once.
Tracing and evaluation tools such as Perimattic AI Suite can help you compare cached and fresh answers over time.
What should you never cache?
- Personalized answers. Anything built from one user’s account, history or documents, unless the cache is scoped to that user.
- Time-sensitive answers. Prices, stock levels, order status, news or anything “today”.
- Tool and agent steps with side effects. A cached answer must never stand in for an action such as a booking or a payment.
- Creative or varied output. If users expect a different answer each time, a cache defeats the purpose.
- Requests containing sensitive data. Storing them in a shared cache spreads that data further.
How is provider prompt caching different from gateway caching?
OpenAI and Anthropic both offer prompt caching. It is not response caching: the model still runs and writes a fresh answer, but the processing of a repeated prompt prefix, such as a long system prompt, tool definitions or a document, is reused, and those input tokens are billed at a lower rate. It works alongside a gateway cache, not instead of it.
| Anthropic prompt caching | OpenAI prompt caching | Gateway response cache | |
|---|---|---|---|
| How it is turned on | A cache_control field, either once at the top level of the request (automatic) or on up to 4 content blocks (explicit breakpoints) | On by default for supported models; no code change needed | Configured in the gateway, per route or key |
| What must match | The prompt prefix (tools, system, messages in that order) must be 100 percent identical up to the breakpoint | The entire rendered prefix must match | The whole request (exact) or its meaning (semantic) |
| Minimum length | 512 to 4,096 tokens, depending on the model; shorter prompts are processed without caching | 1,024 tokens for GPT-5.6 and later | None |
| How long it lasts | 5 minutes by default, refreshed each time it is used; 1 hour optional at extra cost | For GPT-5.6 and later, a 30-minute minimum by default, or 12 hours if requested in the Responses API | Whatever expiry you set |
| Price effect | Reads are 0.1× the base input price on most models (lower on some newer models); writes are 1.25× (5-minute) or 2× (1-hour) | For GPT-5.6 and later, reads are 0.1× the input rate (0.05× for GPT-6.1 Sol); writes are 1.25× (30-minute) or 2.5× (12-hour) | A hit costs no model tokens at all |
| Output | Freshly generated each time | Freshly generated each time | The stored response |
Sources: Anthropic prompt caching docs and OpenAI prompt caching guide. Both describe further details, including different minimums and retention for older models. Prices checked October 22, 2026; providers change prices often — confirm on the provider’s page. Per-token prices are in our Claude API and OpenAI API pricing guides.
To get the most from provider prompt caching, put stable content (instructions, tools, reference documents) first and anything that changes per request last, because both providers only reuse a matching prefix.
How does Perimattic LLM Gateway handle caching?
Perimattic LLM Gateway includes:
- Exact-match and semantic caching of responses.
- Cost tracking and attribution by team, feature and customer, so you can see what caching saves.
- Request logging with PII redaction.
- One OpenAI-compatible API in front of multiple providers.
Get started: Sign up and our team will get in touch, or estimate the effect of a cache with the LLM gateway savings calculator.
Related guides and tools
Semantic caching, answered
Is semantic caching the same as prompt caching?
No. Semantic caching, usually run by your gateway or app, skips the model call entirely and returns a stored answer. Provider prompt caching, offered by OpenAI and Anthropic, still calls the model and generates a fresh answer, but reuses the processing of a repeated prompt prefix and bills those input tokens at a lower rate.
What similarity threshold should I use for semantic caching?
There is no universal number: it depends on your embedding model, how it scores similarity and how costly a wrong answer is. Start strict, log near-misses and false hits on real traffic, have people review a sample and loosen the threshold only while answers stay correct.
Can semantic caching leak data between users?
Yes, if the cache is shared too widely. An answer generated for one user or tenant can be served to another who asks a similar question. Scope cache keys by tenant and, for anything personal, by user, or do not cache those requests at all.
Want one API for every model, with costs under control?
Sign up to tell us which models and providers you use, and our team will get in touch to set up the LLM Gateway for your stack.
Sign up