server: fix incremental DSML tracking across malformed text and split close tags - #997
Open
titan550 wants to merge 1 commit into
Open
server: fix incremental DSML tracking across malformed text and split close tags#997titan550 wants to merge 1 commit into
titan550 wants to merge 1 commit into
Conversation
… close tags dsml_decode_tracker_update() reads the accumulated response text with a byte cursor, dt->pos. The cursor persists across calls. Two paths could stop the cursor permanently. The tracker then could not see later bytes and reported a stale classification for the rest of the generation. The classification selects the sampling policy for the next token. A stuck cursor holds the temperature at 0, also for payload text. The tracker exists to keep payload sampled normally (22ca6ab). 1. The structural: loop returned on an unrecognized non-whitespace byte and did not advance the cursor. Stray prose after the start tag, an <invoke> tag without the sentinel, or a code fence stopped it permanently. The loop now advances past the byte. The partial-literal checks run first, so the loop still holds a tag prefix at the end of the buffer and does not skip it. 2. A piece can end on the lone '<' of a closing parameter tag. raw_partial_lit_min(..., 2) rejects a one-byte prefix, so the body loop consumed the '<'. The close tag then could not match again. A new one-byte check holds the cursor on that byte. The string-body loop reports STRING_BODY. The JSON loop reports JSON_STRUCTURAL. These are the values the reference recognizer returns, so the two recognizers stay in agreement. 3. dsml_decode_state_for_text() is the test-only reference recognizer. It had the same defect and gets the same fix. Without it, the differential tests would compare against the old defect. This commit fixes cursor correctness only. It does not bound a generation whose tool block never closes. That is a separate problem (antirez#895, antirez#48). The tests feed cumulative prefixes because a one-shot feed cannot show a stuck cursor. The tracker cost is unchanged: 0.010 us per piece before and after (71 KB tool call fed in 4-byte pieces, best of 20 runs).
titan550
marked this pull request as ready for review
September 6, 2026 20:58
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
My server decoded inside one open tool block for more than 900 seconds, at temperature 0, near 30,000 generated tokens. I used AI assistance to identify the issue. Two independent defects in
dsml_decode_tracker_update()cause this class of failure. The commit message has the mechanics.Defect 1: an unrecognized byte in the structural section stops the cursor permanently. Defect 2: a piece that ends on the lone
<of a closing parameter tag loses that byte, and the close tag can never match again. Each defect wedges the state that selects the sampling policy. Defect 1 holds greedy sampling for the rest of the generation. The sampler has no repetition penalty, so a greedy repetition loop cannot exit, and decode runs until the context is exhausted.Scope: cursor correctness only. This change does not bound a generation whose block never closes. That is the request in #48 and part of #895. An experimental token-budget backstop is stacked on this branch. I can send it separately if wanted.
The tests feed cumulative prefixes, byte by byte, because a one-shot feed cannot show a stuck cursor. A differential test compares the tracker to the test-only reference recognizer at random split points. Each fix was verified by reverting it alone against the new tests: the structural-loop advance -> 429 failures, the string-body one-byte hold -> 586, the JSON one-byte hold -> 298, the reference-recognizer fix -> 351.
Verification per CONTRIBUTING. Machine: Apple M5 Max, 128 GB, Darwin 27.0, Metal backend. Model:
DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-0731(viaDS4_TEST_MODEL).Tracker cost is unchanged: 0.010 us per piece before and after (71 KB tool call, 4-byte pieces, best of 20). The change is server-side text scanning only. No inference backend is touched.