Preserve live KV cache across normalized tool replay

[rratmansky]

Preserve live KV cache across normalized tool replay

Summary

Preserve the resident KV checkpoint when a Chat Completions client replays a successful tool-call turn with semantically equivalent but byte-normalized assistant text.

The server already uses a remembered visible transcript to continue from richer live state after tool-less thinking. This extends that existing mechanism to successful tool-call turns:

• build the syntax-aware client-visible assistant transcript after a tool call;
• remember it alongside the sampled live token frontier;
• validate that the next rendered prompt exactly extends that visible transcript;
• if it does, retain the sampled live tokens and tokenize only the new text suffix.

The visible continuation check now runs before GLM token rewind. Requests that do not match still use the existing token, rendered-text, disk-cache, or cold-prefill paths.

Motivation

A long GLM-5.3 agent session repeatedly lost its entire resident cache because parse-and-replay normalized one trailing newline immediately before a tool call:

live_tokens_before: 161095
prompt_tokens: 161124
live_prompt_common: 161018
memory_miss_reason: token-mismatch
cache_source: none
cached_tokens: 0

first mismatch:
  live:   "):\n"
  prompt: "):"

Although the visible conversation was the same and more than 99.9% of the token history agreed, the request prefetched all 161,124 tokens from zero. Cache misses at this context took roughly 420–525 seconds; a neighboring successful resident-cache hit completed in about 14 seconds.

This is particularly painful for tool-using agents because every tool result immediately triggers another request.

Implementation

• Generalize the existing visible-chat continuation description to cover normalized tool-call replay as well as omitted tool-less reasoning.
• Add a small pure visible-prefix predicate that requires:
  • a valid remembered state;
  • the same live token frontier;
  • an exact byte prefix of the newly rendered prompt;
  • at least one byte of new prompt suffix.
• After a successful non-Responses tool call, remember the syntax-aware visible transcript produced from the request prefix and parsed assistant/tool-call output.
• Try this validated visible continuation before GLM rewind.
• Report successful use as cache_source=chat-visible and log visible chat continuation match=visible-prefix.

Responses API behavior is unchanged; it retains its existing protocol-specific continuation mechanism.

Safety

This does not perform an arbitrary token rewind and does not modify GLM KDA/indexer state.

The sampled live KV state is reused only when the next request exactly extends the remembered client-visible transcript at the same live frontier. Edited histories, unrelated branches, stale slots, non-extending prompts, failed/length-truncated tool turns, and image-identity mismatches continue through the existing fallback paths.

The implementation uses the existing syntax-aware tool renderer, so it covers both DeepSeek DSML and GLM <tool_call> replay.

Related work

This is a narrowly scoped, current-main implementation of the visible-checkpoint approach discussed in several existing reports and PRs:

• Port visible-KV checkpoint fixes for tool turns (fixes token-mismatch reprocesses on agentic workloads) #789 contains the closest implementation and is a broader superset, including tool-context final-answer checkpoints, streaming-error preservation, and KV waypoint changes. This PR isolates only the successful tool-call frontier fix and adds GLM-5.3 production validation.
• server: remember a tools-aware visible KV checkpoint after tool-call turns #709 handles the complementary DeepSeek case where clients omit reasoning_content; this PR was validated with a client that replays parsed reasoning.
• Server: chat/completions live tool continuation via tool_call_id #611 / OpenAI /v1/chat/completions tool rounds never hit live continuation (match=tool-output-ids); live slot discarded despite full-prompt common #609 bind OpenAI tool-result continuation through tool_call_id, an alternative protocol-specific solution to the same repeated-prefill symptom.
• server: preserve GLM tool-call separators on replay #904 preserves GLM tool-call separators at rendering time and is complementary: it prevents one source of replay drift, while this PR safely retains live KV when semantically equivalent visible replay still differs from sampled tokens.

Related broader reports include #691, #816, #781, and #364.

Tests

Machine and test configuration:

• Apple M5 Max
• Metal backend
• ds4flash.gguf -> GLM-5.3-Flash-Q2.gguf

Commands run:

make clean
make -j4
make -j4 ds4_test
./ds4_test --server
./ds4_test --tool-call-quality

Results:

server: OK
tool-call-quality: OK
ds4 tests: ok

The server tests include:

• canonical visible transcript equality for DeepSeek tool syntax;
• canonical visible transcript equality for GLM tool syntax;
• acceptance of the observed normalized "):\n" -> "):" tool-turn replay;
• rejection when the live frontier differs;
• rejection when visible history changes;
• rejection when the request does not extend the remembered transcript.

A complete ./ds4_test --all run was also attempted with the local GLM-5.3 Q2 model. Its server, tool-call quality, long-context, GLM continued-prefill, and Metal kernel sections passed. It reported 20 model/golden-vector discrepancies in logprob vectors, SSD-streaming vectors, a local golden vector, and long-prompt Metal tensor equivalence. Those checks compare the locally configured GLM model against existing expected vectors and are outside this server-only change; they are reported here as notable failures per CONTRIBUTING.md.

[rratmansky]

Manual end-to-end validation on Apple M5 Max / Metal with GLM-5.3-Flash-Q2.gguf and a ~162K–170K-token tool-using agent session:

• 18 consecutive tool-result continuations selected cache_source: chat-visible.
• Reused live frontiers ranged from 162,940 to 170,109 tokens.
• Newly prefetched suffixes ranged from 23 to 714 tokens.
• Prompt-processing time for those continuations ranged from 0.304s to 2.803s.
• The initial cold prompt (161,995 tokens) took 446.468s to prefill.
• A later cold miss (166,208 tokens) took 468.552s to prefill.

Representative trace entry:

prompt_tokens: 162980
effective_prompt_tokens: 162981
cached_tokens: 162940
live_prompt_common: 162385
cache_source: chat-visible

The stale ordinary token common-prefix position while the full live frontier was reused confirms that the new visible-transcript path—not an incidental exact-token hit—provided the speedup.

One boundary remains: after a tool-enabled turn finished with a normal final answer rather than a tool call, a later user turn cold-missed. This patch records successful tool-call frontiers, so it fixes the repeated tool-call/tool-result failure mode but does not claim to cover every non-tool final-answer transition.