Prompt caching

Both providers bill a prompt's cached prefix at a fraction of the normal input price, and read it faster. A chat with tools sends the whole conversation again on every step, so whether that prefix is reused decides most of what a long conversation costs.

Kai is built around one rule:

The request for step n+1 starts, byte for byte, with the request for step n.

The only exceptions are the start of a new segment (below). Everything in this page is how Kai keeps that rule, and what it does when it can't.

Requests are built from what's storedCopy Anchor Link

Kai builds each request fresh from what it has stored about the conversation, and from nothing else: no clock, no current settings, no live tool lists. The same conversation always produces the same request.

Kai's test suite checks this rule for both providers, step by step, through tool loops with parallel calls, branches, notes, attachments, summaries and model switches.

SegmentsCopy Anchor Link

A segment is a stretch of conversation with one model, one system prompt and one toolset. It starts at a context message in the message tree, which holds the model, the system snapshot and the toolset snapshot. Inside a segment, every request is an append-only extension of the previous one.

A new segment starts when something changes that can't be appended:

• The model (or the key it's reached through).
• The tools, unless the change can be made in place (see Tool changes mid-conversation).
• Compaction: a summary replaces the older messages.

The first request of a segment re-reads the conversation once, with the new system prompt and tools, and pays for a cache write. The user message gets a note saying what changed ("The available tools changed. Use only the tools currently listed." or "Earlier responses in this conversation came from a different model."), so the model isn't surprised by the history.

Because the system prompt is rebuilt only at a segment start, that's also when edits to agents, workspace settings and server instructions take effect.

AnthropicCopy Anchor Link

Anthropic caches only up to explicit cache_control breakpoints, in the order tools → system → messages. Kai places two:

1. The last system block. Caches the tools and the system prompt together. It's the same for every request in a segment, so a new turn always has this much to read even if the conversation part missed.
2. The last content block of the last message (a moving tail). Each step writes the cache up to the end of the conversation so far; the next step, whose request starts with exactly those bytes, reads it.

{
  "tools": [ … ],
  "system": [
    { "type": "text", "text": "<Kai's base prompt>" },
    { "type": "text", "text": "<agent instructions>", "cache_control": { "type": "ephemeral" } }
  ],
  "messages": [
    { "role": "user", "content": [ ... ] },
    { "role": "assistant", "content": [ <thinking, as stored>, <tool_use, as stored> ] },
    { "role": "user", "content": [ { "type": "tool_result", ..., "cache_control": { "type": "ephemeral" } } ] }
  ]
}

Kai places the breakpoints itself rather than relying on automatic caching, so it works the same through gateways.

• Lifetime. 5 minutes by default; a key's Cache lifetime setting can make it one hour ("ttl": "1h"), which costs more to write but survives people pausing between messages.
• Thinking. Thinking blocks are replayed exactly as stored, signature included, for the current segment. Thinking from before the segment start (another model's, or before a summary) is left out, so a signature is never sent to a model that didn't produce it.
• Notes go at the start of the user message, in a <system-reminder> text block. User text can't open or close one: Kai escapes the tag.

Tool changes mid-conversationCopy Anchor Link

The tool list is the very first thing in the cached prefix. Changing it — someone switches a server on in the tools menu, or a server adds a tool — would normally invalidate the whole cache and re-read the conversation.

On Anthropic, Kai instead appends the change. The user message is followed by a system message that adds and removes tools in place, sent with the inline-tools-2026-09-15 beta:

// …the conversation so far, unchanged…
{ "role": "user", "content": [{ "type": "text", "text": "Now file it in Linear." }] },
{
  "role": "system",
  "content": [
    {
      "type": "tool_addition",
      "tool": {
        "type": "tool_definition",
        "definition": {
          "name": "mcp__linear__create_issue",
          "description": "…",
          "input_schema": { … }
        }
      }
    },
    {
      "type": "tool_removal",
      "tool": { "type": "tool_reference", "name": "mcp__files__read" }
    }
  ]
}

The request's tool list stays the segment's original one, so everything before the change stays cached. The change itself is stored as a tool_change note on the user message (the full definitions added, the names removed, and a tool whose definition changed counts as added again), and it appears in the same place on every later request. Tool calls are checked against the tools in effect after the last change.

A tool change is made in place when all of these hold:

• the provider is Anthropic and the model supports it;
• the key's Mid-conversation tool changes setting is on (the default);
• the model didn't change in the same turn;
• it isn't a regenerate, and the turn doesn't start with compaction;
• neither the old nor the new toolset uses deferred loading.

Otherwise the change starts a new segment. If a gateway rejects the beta, Kai turns the setting off for that key (see Gateways) and falls back to new segments.

OpenAICopy Anchor Link

The Responses API caches automatically: any request whose beginning matches a recent one reads that prefix, from 1,024 tokens up. There are no breakpoints to place, so the work is all in keeping the prefix identical and in landing on the same cache.

• prompt_cache_key is kai:<conversation id>. OpenAI routes requests with the same key to the same cache, so a conversation keeps hitting its own prefix instead of depending on which machine served it.
• store: false, no previous_response_id. Kai owns the transcript and sends it every time, which is what makes the request reproducible from stored rows.
• Reasoning. Kai asks for reasoning.encrypted_content and replays each reasoning item verbatim (minus output-only fields), so the model keeps its reasoning across steps and turns, and the replayed bytes match.
• The system prompt is the first developer item. Notes go in a developer item just before the user message they belong to.
• Tool changes always start a new segment: Kai has no in-place tool change for the Responses API, so a new tools list means a new prefix.

Deferred tools (tool search)Copy Anchor Link

When a chat has many tools (over 40, or about 10,000 tokens of definitions) and the model supports it, Kai sends them with defer_loading: true and adds the provider's tool search tool. The model loads the definitions it needs on demand, and the prefix stays small. The deferred list is part of the segment's toolset snapshot like any other, so it doesn't move between steps. A toolset change with deferred loading starts a new segment.

Switching providersCopy Anchor Link

Changing the model starts a segment, and the new request is rebuilt from the provider-neutral columns of earlier parts (text, tool name, input, results). The other provider's raw payloads are never sent: an OpenAI conversation moved to Claude carries no OpenAI items, and the reverse. Every tool call is still paired with its result.

CompactionCopy Anchor Link

When a conversation nears the model's context limit, Kai asks the model for a summary and inserts a compaction context message. From then on, requests start at the summary and include nothing older, which is a new prefix: one full read, then cached as usual.

What still costs a re-readCopy Anchor Link

• A new segment: model change, tool change without in-place support, compaction.
• The cache expiring: 5 minutes (or an hour) without a request.
• Changing a key's settings that shape the request (web search on or off), or Kai turning off a feature a gateway rejected.
• Changing the reasoning effort, on models where the provider keys the cache on it.
• Prompts below the provider's minimum cacheable length.

Measuring itCopy Anchor Link

• Usage shows cached reads, cache writes and the cache hit ratio (cached reads ÷ all input) by day, member and model. Steps after the first in a segment should read nearly all of their input from the cache.
• Kai logs a cache_miss warning (with the conversation id) when a step after the first reads nothing from the cache on a context over 4,096 tokens.