Prompt caching in practice

The short version

Prompt caching stores the model's processed prefix so repeated requests skip recomputation. If your prompts share a long system prompt, few-shot examples, or retrieved context — and most production prompts do — caching cuts both latency and cost dramatically. On some providers it's automatic; on others you opt in. Either way, it's the cheapest performance win in LLM engineering.

How it works

Transformer inference processes your prompt in two phases: the expensive prefill (computing key-value states for every input token) and the cheaper decode (generating one token at a time). When two requests share an identical prefix, the second can reuse the cached KV states and skip most of the prefill.

The catch is the word identical. Cache hits require an exact prefix match, byte for byte. One changed word at the start of your prompt invalidates everything after it.

Designing for cache hits

Structure prompts with the stable content first and the variable content last:

1. System prompt and persona — static, always first.
2. Few-shot examples — static, keep them frozen.
3. Retrieved context / tool definitions — semi-stable; order deterministically (sort tool definitions, sort retrieved chunks by ID, not by score).
4. The actual user query — last, where it belongs anyway.

Two details that quietly destroy hit rates:

• Timestamps and request IDs in the prefix. Anything dynamic — dates, user names, session tokens — must go after the cached region or in a separate field.
• Non-deterministic ordering. If your RAG retriever returns chunks in score order and scores tie-break randomly, your prefix changes every request. Sort by something stable.

What it saves

Typical numbers: 50–90% latency reduction on time-to-first-token for long prompts, and providers usually discount cached input tokens 50–90%. For agents that re-send growing conversation histories or tool schemas on every turn, caching is often the difference between a snappy product and a sluggish one.

Rule of thumb: if your p50 prompt is over 2k tokens and you haven't structured for caching, you're leaving the easiest money in LLM engineering on the table. Audit your prefix stability before reaching for a smaller model.