Skip to content

fix(anthropic): capture input + cache tokens on the streaming path (#433) - #448

Open
initializ-mk wants to merge 1 commit into
mainfrom
fix/anthropic-stream-input-tokens
Open

fix(anthropic): capture input + cache tokens on the streaming path (#433)#448
initializ-mk wants to merge 1 commit into
mainfrom
fix/anthropic-stream-input-tokens

Conversation

@initializ-mk

Copy link
Copy Markdown
Contributor

Fixes #433.

Problem

readAnthropicStream handled only message_delta (which carries output_tokens), so a streamed Anthropic call dropped all input tokens — and, under prompt caching, the cache_read/cache_creation counts too. The streaming sibling of the non-streaming fix in #431/#432.

Anthropic reports input_tokens (+ cache_read_input_tokens / cache_creation_input_tokens) on the message_start event and accumulates output_tokens onto message_delta — so a parser that ignores message_start sees zero input.

Fix

  • Parse message_start usage (input + cache read/creation) into locals.
  • Emit one complete UsageInfo on the terminal message_delta (input + cache read + creation + output + TotalTokens), so the streamed usage matches the non-streaming path.

Why one terminal emission (not emit-at-start + emit-at-delta): consumers merge StreamDelta.Usage differently — the existing aggregator (responses.go) overwrites (result.Usage = *delta.Usage), while #433's sketch assumed summing. A single authoritative Usage is correct under both: overwrite ends with the complete value; summing sees it once (no double-count). Emitting input at start and output at delta would lose input under the overwrite consumer.

Blast radius

Low/forward-looking: today the only production ChatStream consumer (forge-cli/cmd/ui.go) reads Content only and ignores usage, so nothing consumed streaming usage before. This makes it correct for when a consumer (or a future streaming AfterLLMCall/accumulator) does.

Tests

  • Cache-heavy stream recovers input=12, cache_read=4000, cache_creation=200, output=25, total=4237, TotalInputTokens()=4212.
  • Non-cached stream populates input+output with zero cache fields.

golangci-lint clean; full forge-core/llm suite passes. No doc change — the fix makes streaming match the usage behavior the docs already describe for llm_call.

)

readAnthropicStream handled only message_delta (output_tokens), so a
streamed Anthropic call dropped ALL input tokens — and, under prompt
caching, the cache read/creation counts too. This is the streaming
sibling of the non-streaming fix in #431/#432.

Anthropic reports input_tokens (+ cache_read/creation) on message_start
and accumulates output_tokens onto message_delta. Now:
- parse message_start usage into locals,
- emit ONE complete UsageInfo on the terminal message_delta
  (input + cache read + creation + output + TotalTokens),

so the streamed usage matches the non-streaming path. A single
authoritative Usage is correct whether a consumer overwrites
(result.Usage = *delta.Usage, the existing pattern) or sums per-delta —
avoiding both input-loss on overwrite and double-count on sum.

Tests: cache-heavy stream recovers input/cache/output/total; a
non-cached stream populates input+output with zero cache fields.
golangci-lint clean; full llm suite passes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Anthropic streaming path drops input + cache tokens (StreamDelta.Usage carries only output_tokens)

1 participant