AI caching strategies

What was covered

• The problem. The system prompt is processed again on every request. That adds latency and cost, because the prompt can be a large share of the tokens.
• Under the hood. Each attention layer produces key and value (KV) vectors for every token. They stay in GPU memory during a request and are thrown away at the end, so the next request starts from scratch.
• Layer one, the KV cache. Inside a single request, handled by the model.
• Layer two, the prompt cache. You tell the provider to store and reuse the KV vectors for an identical system prompt, so the model only processes the new part of the query. Depending on the provider you need an explicit directive, usually called cache control.
• Layer three, the response cache. Keep the full answer to an identical query in a database such as Redis. Repeated queries then cost nothing. It works for quick questions and documentation, not for multi-turn conversations.
• A useful analogy. The layers resemble a CPU’s L1, L2 and L3 caches. Unlike the model’s internal cache, the prompt and response caches need active invalidation.

What came up

• Where to start. The prompt cache is the preferred option, and its effect depends on the provider and on including the directive correctly.
• Limits. A minimum number of tokens is needed to trigger the cache, and entries have a time to live. OpenAI’s is reportedly around 30 minutes.
• Semantic caching is risky in chat. Reusing answers by question similarity can trap a conversational bot in a loop, repeating the same reply to variations of one question.
• Model routers. These caches can also be configured through routers such as LiteLLM, though compatibility must be checked. Pedro took the action to find out whether the team’s LiteLLM setup supports cache control.

Takeaway

Start with prompt caching, add a response cache for repeat questions, and think twice before adding semantic caching to a conversational bot.