Prompt Caching: What Actually Gets Cached and Why Your Hit Rate Is Zero

You enabled prompt caching on your LLM provider, watched the first few requests, and saw a steady zero in the cache-hit column. The documentation promised reduced latency and lower cost, but every call is being processed as if it were the first. The problem is almost certainly in how your prompts are structured—specifically, what the cache key actually sees.

What prompt caching actually caches

Cache keys are built from the prefix of input tokens—the exact sequence of token IDs that includes the system message and leading user turns. There is no semantic similarity matching; the cache operates at the byte-perfect, token-level granularity of the prompt as it enters the model’s context window. If two requests differ by a single token anywhere in the prefix, the cache misses.

The scope of what is cached varies by provider. Anthropic caches at the request prefix level: you mark a portion of the conversation (system prompt or early user messages) with a cache_control header, and that prefix is stored for reuse across subsequent requests. OpenAI caches repeated prompt segments automatically—no explicit opt-in—but the cache key is derived from the exact text of system and user messages that appear in every request. Local inference, using frameworks like vLLM or TensorRT-LLM, caches the full KV cache for the entire sequence, meaning any prefix reuse (even partial) can accelerate generation.

It’s important to distinguish between provider-managed caches (transparent to your code) and application-level caches you build yourself (e.g., storing prompt-response pairs in Redis). Provider caches operate inside the inference API; you pay for the cached portion at a reduced rate but have no control over eviction. Application-level caches give you full control but require explicit logic to check and reuse responses.

I covered the mechanics of structuring predictable prompts in Reliable Structured Outputs from LLMs Using JSON Schema, where deterministic formatting is equally sensitive to subtle changes in the prompt prefix.

Zero hit rate: common causes

The most frequent culprit is a unique prefix injected into every request. Timestamps, user IDs, or randomly generated session tokens that appear in the system message or early user turn destroy cache key collision. Even if the core prompt is identical, the cache sees a different token sequence each time.

Dynamic system instructions that vary per call—such as a date string, user context, or a random instruction like “Today’s date is 2025-03-15”—prevent reuse even when the rest of the prompt is static. I’ve seen teams embed a request ID in the system message for logging purposes, inadvertently guaranteeing a 0% hit rate.

Model version upgrades or changes can invalidate cached entries. If the provider rolls out a new tokeniser or a different model architecture, the internal representation of your prompt changes, and the old cache entries become meaningless. This is often silent; you’ll just see your hit rate drop after a deployment.

Distributed caching across regions or instances may have inconsistent states. If your traffic is geo-distributed and the cache is per-region, the same prompt sent from two different regions will miss on both. Similarly, if you run multiple API keys or accounts, each may have its own cache pool.

Comparing cache designs: Anthropic vs OpenAI vs local

Anthropic’s design requires you to explicitly mark the cacheable prefix using a cache_control header. The cached portion is stored for 5 minutes after the last hit; if you don’t reuse it within that window, it expires. The cost reduction applies only to the tokens that were cached.

OpenAI caches automatically for prompts that repeat exactly. There’s no header to set, but you can inspect the prompt_tokens_details field in the response to see how many tokens were cached. The cache TTL is not documented publicly; it’s likely an LRU with a few minutes of idle time.

Local KV cache reuse gives you full control. In batch inference or multi-turn agents, you can keep the KV cache for the system prompt and replay it across requests. The trade-off is memory management: you need to decide when to evict and how to handle different prompt lengths. This approach works with any model, but it’s only practical when you control the inference infrastructure.

The choice between these depends on your environment. If you’re using a serverless API, Anthropic or OpenAI caches are the only option, and you must design your prompts accordingly. If you have dedicated GPUs, local KV caching can yield higher hit rates and lower latency, but you bear the operational cost.

Designing prompts for cache reuse

Place a static prefix before dynamic content. Separate the invariant instructions (system prompt) from the variable user input so that the cache key covers the largest possible shared segment. For example, put the system message and any fixed few-shot examples at the beginning, and append the user’s query at the end.

# Anthropic example: mark system prompt as cacheable
import anthropic
 
client = anthropic.Anthropic()
 
response = client.messages.create(
    model="claude-sonnet-4-20250514",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": "You are a helpful assistant specialized in Python code review.",
            "cache_control": {"type": "ephemeral"}
        }
    ],
    messages=[
        {"role": "user", "content": "Review this function:\n" + user_code}
    ]
)

Use cache-aware prompt templates: parameterise the dynamic parts and keep them at the end of the message. In OpenAI, the cache key includes the entire messages array, so you want the first N messages to be identical across calls.

# OpenAI example: static system message, dynamic content at the end
import openai
 
system_message = {
    "role": "system",
    "content": "You are a helpful assistant specialized in Python code review."
}
 
# user_code varies per request, but system message is always the same
response = openai.chat.completions.create(
    model="gpt-4o",
    messages=[system_message, {"role": "user", "content": f"Review this function:\n{user_code}"}]
)
# Check cached tokens in response
print(response.usage.prompt_tokens_details.cached_tokens)

Avoid injecting unique identifiers like timestamps or request IDs into the system message unless absolutely necessary. If you need them for logging, move them to the user message or a separate metadata channel that doesn’t affect the cache key.

Prepare for cold starts. If you know a particular prompt template will be used heavily, batch similar requests or pre-warm the cache with dummy calls during low-traffic windows. For example, send a single request with the exact prefix you intend to cache, then reuse it within the TTL window.

Limits and failure modes

Cache eviction policies vary. Most providers use LRU with a TTL (e.g., 5–60 minutes). Long idle periods or high throughput from other users can push your entries out. If you have a burst of unique prompts, your cached prefix may be evicted before it’s reused.

Context window limits: very long prompts may only cache the first part. Some providers cap the cacheable prefix length (e.g., Anthropic caches up to the first 4K tokens of the system prompt). If your prompt is longer, the remainder is not cached, reducing the benefit.

Inconsistency across regions: if your traffic is geo-distributed, caches are typically per-region. Expect variable hit rates unless you pin users to a specific region via API configuration.

Debugging is difficult. APIs rarely expose the exact cache key components or eviction reasons. You’re left guessing why a particular prefix didn’t hit. This is where logging and instrumentation become critical.

Measuring and debugging cache hits

Check API response headers. Anthropic returns x-cache-hit (boolean) and x-cache-prefix (the token range that was cached). OpenAI includes prompt_tokens_details with a cached_tokens count. Log these values on every request.

Build a cache-aware application layer. Instrument your LLM client to record cache status per prompt template. Track hit rate over time and set up alerts for sudden drops. For example, if your hit rate falls below 50%, it may indicate a model update or a change in your prompt template.

Use a middleware like LangChain’s caching to wrap provider caches with an in-memory layer for predictable hit rates. LangChain’s InMemoryCache or RedisCache can store responses keyed by the exact prompt string, giving you a fallback when the provider cache misses.

Practical workflows and tooling

Implement a caching proxy (e.g., Redis + hash of prompt prefix) for cross-application reuse. Hash the system message and the first N user messages, store the response, and serve it if the same hash appears within a configurable TTL. This gives you custom eviction policies and works across API keys or accounts.

Integrate cache metrics into CI. When you deploy a new prompt version, run a set of test requests and validate that the hit rate remains above a threshold. If it drops, the change likely introduced a new dynamic element. This prevents silent regressions in production.

For local inference, use frameworks like vLLM that support prefix caching out of the box. vLLM’s --enable-prefix-caching flag automatically reuses KV cache for common prefixes, and you can monitor hit rates via its metrics endpoint.

If you’re building multi-turn agents, consider using a local KV cache for the conversation history. The system prompt and earlier turns remain cached as the conversation progresses, reducing latency for each subsequent turn. This is particularly effective with models that have large context windows.

The same discipline of avoiding unnecessary dynamism applies to configuration systems. In Tailwind CSS v4 without a config file, separating static defaults from overrides prevents cache invalidation in your build pipeline.

Key takeaways

• Prompt caching matches exact token sequences, not semantic content; any variation in the prefix destroys the cache hit.
• The most common cause of zero hit rate is dynamic content (timestamps, IDs, user context) injected into the system message or early user turns.
• Anthropic requires explicit cache_control headers; OpenAI caches automatically; local inference gives full control but demands memory management.
• Design prompts with a static prefix and dynamic suffix to maximise reuse; avoid unique identifiers in the cacheable portion.
• Measure cache hits via response headers and log them; integrate cache metrics into your deployment pipeline to catch regressions early.