Context management

Every step of a chat sends the model a context: the system prompt, the tools, and the conversation so far. Kai keeps that context within the model's limits without losing what matters, and without breaking the prompt cache more than it has to.

What the model seesCopy Anchor Link

For each step, Kai builds the request from the window: the messages on the current branch, from the latest compaction (or the start of the chat) to the newest message.

Messages on other branches (earlier edits and regenerations) are never sent. Only the branch the person is looking at is the conversation.

Tool resultsCopy Anchor Link

Tool output is the fastest way to fill a context window: one list_issues call can return a megabyte. Kai shapes each result once, when the tool returns, and stores exactly what the model is given (model_content). The stored result is never reshaped later, so the same request reads the same way on every step (which the prompt cache needs).

• Per result: at most about 12,000 tokens.

• Per step: all results of one step (parallel calls) share about 60,000 tokens. When they'd exceed it, each result's budget shrinks in proportion, but never below 500 tokens.

• Truncation keeps the head and the tail, cut on line boundaries, with a marker in the middle:

  Copy Text

  …first part of the output…
  [… 48210 tokens truncated. The full output is saved; the user can open it. …]
  …last part of the output…

• The full output is saved to object storage. People can open it from the tool call in the chat; the model sees only the budgeted text.

• Images in a result are stored as attachments, at most four per result. Models without vision get [image omitted] instead.

Errors are results too: they're prefixed with Error: and go through the same budgets.

Deferred toolsCopy Anchor Link

Tool definitions take context on every step. When a chat has more than 40 tools, or more than about 10,000 tokens of definitions, and the model supports tool search, Kai sends the definitions with defer_loading: true and adds the provider's tool search tool. The model searches and loads only the tools it needs. Without tool search support, Kai sends them all and the tools menu warns that the chat has many tools; above 128 tools it refuses until some are turned off.

Output and stepsCopy Anchor Link

• Output per step: the model's output limit, capped at 64,000 tokens. When a reply hits it, Kai keeps what arrived and marks the message as cut off.
• Steps per reply: 50. A reply that's still calling tools after 50 steps stops with The response stopped after 50 steps.
• Context overflow mid-reply: if a step is rejected because the context is too long (a tool loop that grew past the limit within one reply), the reply stops with The response ran out of context. The next message will start with a summary.

CompactionCopy Anchor Link

Long conversations are compacted: Kai asks the model to summarize the earlier part, and the summary replaces it for the model. People still see the whole conversation, with a divider where the summary starts.

WhenCopy Anchor Link

Each model has a compaction threshold: 80% of its context window, and at most 160,000 tokens for Claude or 200,000 for GPT. When someone sends a message, Kai takes the context size of the previous step (as the provider reported it) plus an estimate of the new message and its text attachments. If that's over the threshold, the turn starts with a compaction.

People can also compact by hand: once a chat is past half its threshold, the composer shows a context meter and Summarize earlier messages now.

HowCopy Anchor Link

1. Kai inserts a compaction context message before the new user message.

2. The summary request is the conversation's own next request, plus one instruction at the end, so it reads the prompt cache like any other step. The instruction asks for five sections: the goal and standing instructions; what's done, keeping exact names, numbers, ids and links; decisions and why; open items; and facts from tools that will be needed again. It also says not to call tools.

3. The result is checked:

   • If the model called a tool instead of writing, Kai retries once with tools switched off (tool_choice: none).
   • An empty summary, or one cut off by the output limit, fails the compaction.
   • A summary longer than a quarter of the conversation it replaces fails too: it wouldn't save enough.
   • If even the summary request is too long for the model, Kai drops the oldest quarter of the turns (whole turns, so a tool call never loses its result) and tries again, up to three times.

4. Kai stores the summary with the person's recent messages verbatim (their most recent ones, up to about 8,000 tokens), so their exact wording survives even if the summary paraphrased it:

   Copy Text

   The earlier part of this conversation was replaced by the summary below,
   which you wrote. Continue from it. Do not mention the summary unless asked.
   
   <summary>
   …
   </summary>
   
   The person's most recent messages before this point, in full:
   
   <recent_messages>
   <message>…</message>
   </recent_messages>

5. The compaction also rebuilds the system prompt and toolset from current settings, like any new segment. Then the turn continues normally.

From then on, every request starts at the summary and includes nothing older. The summary is provider-neutral text, so a chat can switch models or providers after a compaction without carrying anything provider-specific across.

What a compaction costsCopy Anchor Link

The summary request reads the cache, so its input is mostly billed at the cached rate; its output is the summary. The next request is a new prefix (a full read and a cache write), and after that the chat is cached as usual. On Usage, compaction shows as its own kind (summaries).

Notes instead of context changesCopy Anchor Link

Facts that change during a conversation (the date, "the previous reply was stopped", "the model changed", "the tools changed") are added as short notes on the user message they belong to, not by rewriting the system prompt. They cost a few tokens once, stay in place, and keep the earlier context and its cache intact. See Prompt caching.

Reasoning across stepsCopy Anchor Link

• Claude: thinking blocks from the current segment are replayed, with their signatures, so the model keeps its reasoning through a tool loop and across turns. Thinking from before a model switch or a summary is dropped.
• GPT: reasoning items are replayed with their encrypted content, so the model keeps its reasoning without Kai storing it in readable form beyond the summary the provider returns.

Limits at a glanceCopy Anchor Link

Kai's own estimates count about four bytes per token; the context size that triggers a summary is the provider's own count from the last step.