Prompt caching: cache keys, misses, and getting a useful hit rate

You send the same prompt twice but get a cache miss. The response is identical, yet the API charges you full price. The root cause is usually a subtle difference in how the provider constructs the cache key—and most developers don’t realise how specific that key is. Understanding the mechanics of cache key generation, where misses come from, and how to design for reuse is the difference between paying 50% of input cost and paying full price every time. For a deeper look at why your specific setup might be failing, see Prompt Caching: What Actually Gets Cached and Why Your Hit Rate Is Zero.

What the cache key actually contains

Every provider hashes a subset of your request to form a cache key. The exact composition differs, but the principle is the same: the key must be deterministic across requests that are semantically identical, while excluding any metadata that varies between calls.

Provider-specific hashing

OpenAI hashes the prompt text (the concatenation of all messages) plus a session-level identifier tied to your API key and organisation. The session identifier is stable for the same API key, so two requests from the same account with the same prompt text will produce the same key. Anthropic’s approach is more aggressive: it hashes the entire conversation history including tool results. If you send a previous message with a different tool output, the key changes. This makes Anthropic’s caching more powerful for long-running conversations but also more brittle—any change in the history, even a trailing space, flips the key.

Metadata excluded

Both providers deliberately strip fields that are not part of the core prompt. System fingerprints, user IDs, and request-level flags (like temperature or max_tokens) are excluded from the hash. This prevents false misses caused by orthogonal parameters. You can change the temperature between requests and still hit the cache—provided the prompt text is identical. However, if you embed a user ID inside the system prompt, that ID becomes part of the text and therefore part of the key.

Length matters

OpenAI only caches prompts that exceed 1024 tokens. Shorter prompts are never cached, so you cannot get a discount on tiny requests. Anthropic has no minimum length, but the cache key is computed over the entire history, so short prompts in a fresh conversation may still be cached if you explicitly mark them with cache_control. Google Gemini requires you to create a named cache object with a TTL; the cache key is based on the content hash and model version, and the object must be created before inference.

Multimodal exclusion

Image and audio inputs are not included in the cache key on any current API. If you send a prompt with an image, the text part may be cached, but the image is not hashed. This means mixing modalities always produces a miss—the system cannot reuse a cached result from a text-only request even if the text is identical. The practical implication: if your application alternates between text and multimodal inputs, you will never see a cache hit for the multimodal calls.

Why your second request often misses

The most common reason for a zero hit rate is that something in your prompt changes between requests. Here are the usual suspects.

Timestamps, request IDs, and user UUIDs

Your system prompt might contain {{current_date}} or {{session_id}}. Every request gets a unique value, so the hash changes every time. The fix is obvious: move dynamic values to user messages or exclude them from the cache key. But many templating engines inject these by default, and developers don’t notice until they check the x-cache-status header.

Whitespace and encoding differences

Trailing newlines, Unicode normalization (NFC vs NFD), and even the order of JSON keys in structured prompts alter the hash. Consider this:

{"role": "user", "content": "Hello"}
{"content": "Hello", "role": "user"}

If your JSON serialiser does not guarantee key ordering, the two strings are different. The same applies to lists of messages: adding a trailing space at the end of a system message changes the entire key.

Conversation history mismatch

If you append a system message before the user message in one call but not another, the key changes entirely. This is especially common in chat applications where the conversation history is built incrementally. A caching-friendly design keeps the static prefix (system prompt + fixed preamble) stable and only appends the new user message.

Session termination

Most providers expire cache entries after a few minutes of inactivity. OpenAI’s cache TTL is approximately 5–10 minutes after the last request that used the entry. If your pipeline processes requests slowly (e.g., batch processing with delays between calls), the cache may have already expired. For long-running agents, you may need to keep the cache warm by sending periodic “heartbeat” requests that reuse the same prefix.

Practical patterns to improve hit rates

Once you understand the mechanics, you can design your system to maximise cache hits. Here are patterns that work in production.

Stabilize your system prompt

Generate a static prefix once per session and cache it client-side. Store the system message and any fixed preamble in a variable, then reuse that exact string for every request in the session. Do not use templates that inject dynamic values into the system message. If you need per-user information, put it in the user message and keep the system prompt constant. This approach mirrors how you might structure Reliable Structured Outputs from LLMs Using JSON Schema—keeping the schema fixed while varying only the input data.

Use deterministic serialization

When constructing JSON messages, enforce a consistent key order. In JavaScript, you can sort object keys before serialising:

function deterministicStringify(obj) {
  return JSON.stringify(obj, Object.keys(obj).sort());
}

For Python, use json.dumps(obj, sort_keys=True). This single change eliminates one of the most common sources of false misses.

Batch similar requests

If your application receives many user queries that share a long common prefix (e.g., a system prompt describing a product catalog), group those requests so they reuse the same cached entry. OpenAI’s prefix caching works best when the first N tokens are identical across requests. By batching, you ensure that the shared prefix stays in the cache for multiple requests, amortising the cost of the first full-price call.

Monitor with headers

Both OpenAI and Anthropic return cache status headers. OpenAI includes x-cache-status with values hit or miss. Anthropic returns Cache-Control: cached in the response when a cache hit occurs. Log these headers and alert on a zero hit rate. This is the fastest way to detect that your cache key is unintentionally unique.

Provider comparison: OpenAI vs Anthropic vs Google

The key trade-off: prefix caching (OpenAI) is simpler—you do nothing and get automatic caching for long prompts—but it wastes space on unique prefixes. Explicit context caching (Anthropic, Google) gives you control over what is cached, enabling you to reuse a large conversation history across many requests, but requires careful lifecycle management. If you fail to mark messages with cache_control, nothing gets cached.

Failure modes when caching goes wrong

Even with a good hit rate, caching introduces failure modes that you must handle.

Stale cache serves outdated knowledge

If your system prompt includes breaking news, prices, or time-sensitive instructions, a cached response will repeat the old knowledge until the TTL expires. The fix is to set a short TTL or to invalidate the cache when the underlying data changes. For example, if you cache a prompt that includes today’s stock prices, the cache must be invalidated at the end of the trading day.

Cache poisoning via shared keys

Multi-tenant applications that reuse the same API key for different users can leak responses if the cache key does not include per-user metadata. If two users send the same system prompt but expect different user-specific results, the second user will get the first user’s cached response. The only safe pattern is to either include a user-specific identifier in the prompt (which breaks caching) or to use a separate API key per tenant (which also breaks caching because the session identifier changes). There is no clean solution here—you must choose between caching and isolation.

No fallback on cache miss

Some client libraries treat a cache miss as a soft error and retry the request, doubling latency instead of halving it. This happens when the library’s internal cache does not wait for the response and immediately retries on a miss. Always check your library’s behaviour. If you are writing your own caching layer, implement a fallback that sends the request normally on a miss, not a retry.

Overhead outweighs benefit

Cache key computation and lookup can add 5–10ms per request, which is noticeable on very short prompts. If your average prompt is under 300 tokens, the overhead may exceed the savings. Measure before you invest.

When to skip prompt caching entirely

Caching is not always the right tool. Consider skipping it in these scenarios.

• Short, unique prompts (under 300 tokens) see negligible benefit. The overhead of hashing and lookup wipes out any savings. Moreover, OpenAI does not cache prompts under 1024 tokens anyway.
• Dynamic agents that construct every prompt from scratch—for example, a RAG pipeline that retrieves different context for each query—will never hit a cache because the prefix changes every time. In such cases, focus on reducing input tokens via summarisation rather than caching.
• Compliance-driven apps that require fresh responses per user (e.g., medical diagnosis, legal advice) should avoid caching entirely. A cached response that serves a previous user’s context to a new user could violate data privacy regulations. Even if the cache key is perfect, the risk of a bug is too high.

Key takeaways

• Cache keys are built from the prompt text only—metadata like user IDs and temperature are excluded—so any variation in the text, including whitespace and JSON key order, causes a miss.
• The most common reason for a zero hit rate is a dynamic value (timestamp, request ID) embedded in the system prompt.
• Use deterministic serialisation (sort_keys=True in Python, sorted keys in JavaScript) and stabilise your system prompt to maximise hits.
• Monitor x-cache-status or Cache-Control headers to debug cache misses in production.
• Choose between prefix caching (OpenAI, automatic) and explicit context caching (Anthropic, Google, manual) based on your workload’s predictability and lifecycle requirements.