kvstore: stamp checkpoints with a behavioral tokenizer fingerprint - #983
kvstore: stamp checkpoints with a behavioral tokenizer fingerprint#983nazerim wants to merge 2 commits into
Conversation
Persisted KV checkpoints encode their conversation as a token history whose bytes, on future requests, are re-tokenized by whatever tokenizer the running engine has. When tokenizer code or data changes upstream (the JoyAI pre-tokenizer has seen several rule fixes), existing checkpoint token histories become unreproducible: every later request diverges from the restored session at the first changed merge boundary, reloads the same stale file by text key, and re-imports the stale tokenization. The visible symptom is a permanent per-turn live-miss + disk-reload + partial-re-prefill loop for every session resumed with old checkpoints (observed in production: 29 consecutive turns, ~38k tokens re-prefilled each turn, session permanently pinned to a stale frontier). A text-equality check at load cannot detect this class without also rejecting legitimate preserved-reasoning continuations, whose stored sampled tokens intentionally differ from a whole-text re-tokenization of their bytes - the bridge's premise. The correct discriminator is the engine identity that produced the tokens. This commit adds a behavioral fingerprint: an append-only set of probe strings (apostrophe-punctuation seams, digit runs, indentation, newline joins, UTF-8 letters, special-token edges) is run through the real tokenizer once, and the resulting ids hashed together with the token table. It covers tokenizer data AND code changes with no human-maintained version constant. Checkpoint trailers now begin with a TOKFP section carrying the fingerprint; the loader pre-checks it by a 16-byte seek to the trailer, before touching the payload (session untouched), and mismatches fall back to the normal cold path, which rewrites a fresh stamped file under the same key. Unstamped legacy files keep today's trusted behavior. Verified in a live agent session: stamped files restore at full depth (451k tokens, 38-token re-prefill); a synthetic probe change (simulated upstream tokenizer edit) makes every old stamped file reject in microseconds with a clear warning and the session self-heals on the next store.
|
Hey Naz — I like the aim here. Preventing a session from repeatedly restoring an incompatible checkpoint is useful hardening. I spotted a restart-stability issue in the fingerprint on In h = tokenizer_fp_hash_bytes(h, e->vocab.token,
(size_t)e->vocab.n_vocab * sizeof(ds4_str));
I think the fix is to hash a canonical serialization of the tokenizer data: token IDs/order, lengths and the actual bytes behind each pointer, plus the other data needed to identify its tokenization behavior. Avoid hashing raw structs/process representations. Two particularly useful regressions would be:
This is a static-code finding, not a claim that I've run your full model-backed test sequence. Thanks for working on this — the underlying problem is definitely worth addressing, and this looks like a fixable snag. 🙂 |
|
You're right, and thank you — this is the kind of bug my live validation was uniquely suited to hide. Same binary + same args + same load sequence on macOS reproduced the allocator layout across my restart test, so the pointer-hash passed by luck; on any different mapping (or a different host) every stamped file would silently reject after each restart. Static review caught what the runtime fluked. Fix (will force-push once validated): hash tokenizer content — Your two regressions become the validation plan, run live:
I'll report both results on this PR before re-requesting review. The lesson is noted for the probe-set doc comment as well: pointer-derived inputs must never reach the hash. |
ds4_str holds a pointer and a length; hashing the raw array therefore hashed process mapping state, not tokenizer data. Same-binary restarts can reproduce the allocator layout by luck, but any different mapping would silently change the fingerprint and reject valid stamped checkpoints after an ordinary restart - the opposite of the feature's purpose. Hash content instead: n_vocab, then every token's length + bytes in id order, then the merge-rank table (used/cap plus each used slot's key bytes and rank; open-addressed slot order is a deterministic function of the GGUF merge list). The behavioral probes still ride on top, so code changes are covered even when data is not. Validated live (Apple M5 Max): fingerprint e467e62ceeadf47e identical across separate process launches of the unchanged model, and a stamped checkpoint restored at full depth (32768-token grid, 69 ms load) across a stop/start cycle with zero mismatch lines. A single-byte token mutation on an APFS clone of the model (TABLE->TASLE, same length, outside every probe string) moves the fingerprint to 5f83c7a25a3b184e, proving data sensitivity independent of probe coverage.
|
Fixed and validated — head now What changed in
Your two regressions, run live on Apple M5 Max against the production model file:
One confession your static finding flushed out: my original "validated across restart" claim in the PR description was real but lucky — same binary + same args reproduced the allocator layout on macOS, so the pointer-hash passed its own test. The fix and both regressions above are now the honest basis for that claim. Thanks for catching this before it landed. |
2 participants
Persisted KV checkpoints store a token history that future requests
re-tokenize from text with whatever tokenizer the running engine has.
When tokenizer code or data changes (the JoyAI pre-tokenizer has seen
several rule fixes recently), old checkpoint token histories become
unreproducible: every resumed session diverges from its restored state
at the first changed merge boundary, keeps reloading the same stale
file through the text-keyed tiers, and re-imports the stale
tokenization - a permanent per-turn miss/reload/re-prefill loop.
Production symptom (this week, long multimodal agent session): 29
consecutive turns each live-missed at the same token, each reloaded
the same stale checkpoint by text key, each re-prefilled ~38k tokens.
The session was permanently pinned; only a manual cold rebuild healed
it. A plain text-equality check at load does not work as a detector:
it false-positives on legitimate preserved-thinking continuations,
whose stored sampled tokens intentionally differ from a whole-text
re-tokenization of their bytes (that difference is the bridge's entire
premise). The correct discriminator is engine identity.
This commit adds a behavioral tokenizer fingerprint: an append-only
probe set (apostrophe-punctuation seams, digit runs, indentation,
newline joins, UTF-8 letters, special-token edges) run through the
real tokenizer, hashed together with the token table. It covers
tokenizer data AND code changes with no human-maintained version
number, and it is cheap (computed once, lazily, under a mutex).
Mechanics:
header + u64 fp) and the auxiliary ext bit DS4_KVSTORE_EXT_TOKFP.
offset BEFORE touching the payload: mismatches cost microseconds,
never a 600 ms load, and the session is left untouched.
stamped file under the same text key: the poison class becomes
self-healing instead of permanent.
Verification (live agent session on Apple M5 Max):
38-token re-prefill, session continues at full reuse.
tokenizer edit): old stamped files rejected instantly with a clear
warning; fallback chain restored from the next valid candidate;
fresh stores re-stamped correctly after restoring the probe.
updated for the new leading section).
Independent of #968/#971 (touches ds4.c/ds4.h/ds4_kvstore.*/server
trailer hooks only); no behavior change for files written today.