fix(anthropic): capture input + cache tokens on the streaming path (#433) - #448
Open
initializ-mk wants to merge 1 commit into
Open
fix(anthropic): capture input + cache tokens on the streaming path (#433)#448initializ-mk wants to merge 1 commit into
initializ-mk wants to merge 1 commit into
Conversation
) readAnthropicStream handled only message_delta (output_tokens), so a streamed Anthropic call dropped ALL input tokens — and, under prompt caching, the cache read/creation counts too. This is the streaming sibling of the non-streaming fix in #431/#432. Anthropic reports input_tokens (+ cache_read/creation) on message_start and accumulates output_tokens onto message_delta. Now: - parse message_start usage into locals, - emit ONE complete UsageInfo on the terminal message_delta (input + cache read + creation + output + TotalTokens), so the streamed usage matches the non-streaming path. A single authoritative Usage is correct whether a consumer overwrites (result.Usage = *delta.Usage, the existing pattern) or sums per-delta — avoiding both input-loss on overwrite and double-count on sum. Tests: cache-heavy stream recovers input/cache/output/total; a non-cached stream populates input+output with zero cache fields. golangci-lint clean; full llm suite passes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #433.
Problem
readAnthropicStreamhandled onlymessage_delta(which carriesoutput_tokens), so a streamed Anthropic call dropped all input tokens — and, under prompt caching, thecache_read/cache_creationcounts too. The streaming sibling of the non-streaming fix in #431/#432.Anthropic reports
input_tokens(+cache_read_input_tokens/cache_creation_input_tokens) on themessage_startevent and accumulatesoutput_tokensontomessage_delta— so a parser that ignoresmessage_startsees zero input.Fix
message_startusage (input + cache read/creation) into locals.UsageInfoon the terminalmessage_delta(input + cache read + creation + output +TotalTokens), so the streamed usage matches the non-streaming path.Why one terminal emission (not emit-at-start + emit-at-delta): consumers merge
StreamDelta.Usagedifferently — the existing aggregator (responses.go) overwrites (result.Usage = *delta.Usage), while #433's sketch assumed summing. A single authoritative Usage is correct under both: overwrite ends with the complete value; summing sees it once (no double-count). Emitting input at start and output at delta would lose input under the overwrite consumer.Blast radius
Low/forward-looking: today the only production
ChatStreamconsumer (forge-cli/cmd/ui.go) readsContentonly and ignores usage, so nothing consumed streaming usage before. This makes it correct for when a consumer (or a future streamingAfterLLMCall/accumulator) does.Tests
input=12,cache_read=4000,cache_creation=200,output=25,total=4237,TotalInputTokens()=4212.golangci-lintclean; fullforge-core/llmsuite passes. No doc change — the fix makes streaming match the usage behavior the docs already describe forllm_call.