feat(langchain): report cached and reasoning token usage in Langchain/Langgraph - #6800
Conversation
The Langchain and Langgraph integrations only recorded input, output, and total token counts, dropping cached input tokens and reasoning output tokens even though SPANDATA defines both and the OpenAI, OpenAI Agents, and Google GenAI integrations already report them. Extract cache_read/cached_tokens and reasoning/reasoning_tokens from both the LangChain usage_metadata detail shape and OpenAI-style details dicts, and set gen_ai.usage.input_tokens.cached and gen_ai.usage.output_tokens.reasoning on spans in both integrations. Fixes getsentryGH-6799
![sentry[bot]](https://avatars.githubusercontent.com/in/12637?s=60&v=4){kind=small}
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit cb26c51. Configure here.
alexander-alderman-webb
left a comment
alexander-alderman-webb
left a comment
There was a problem hiding this comment.
We should come up with a more principled way to record token fields from different providers.
Unfortunately, the token details are a leaky abstraction of LangChain (since they just expose provider-formatted values).
List the known provider spellings per token field once, in precedence order, and read them through a None-aware first-value helper so a legitimate count of 0 is kept instead of falling through to the next provider's key. Langgraph now reuses the same extraction as Langchain, so both integrations understand the same shapes (including LangChain's input/output_token_details).
|
Thanks @alexander-alderman-webb — agreed the provider-formatted detail keys are a leaky abstraction, and the ad-hoc sniffing made it worse. Latest push takes a step toward the principled version within this PR's scope:
If you'd rather see this generalized further (e.g. a shared normalization map in |
| # each call site, the known spellings per field are listed once here, in | ||
| # precedence order (LangChain's own normalized names first, then provider ones). | ||
| # Adding support for another provider's spelling means extending one tuple. | ||
| _INPUT_TOKEN_KEYS = ("prompt_tokens", "input_tokens") | ||
| _OUTPUT_TOKEN_KEYS = ("completion_tokens", "output_tokens") | ||
| _INPUT_DETAILS_KEYS = ("input_token_details", "prompt_tokens_details") |
There was a problem hiding this comment.
Bug: The comment describing key precedence for token extraction is inconsistent with the implementation for _INPUT_TOKEN_KEYS and _OUTPUT_TOKEN_KEYS, which can be misleading.
Severity: LOW
Suggested Fix
Update the comment to accurately reflect the implementation's precedence order, or reorder the keys in _INPUT_TOKEN_KEYS and _OUTPUT_TOKEN_KEYS to match the comment's description. For example, change ("prompt_tokens", "input_tokens") to ("input_tokens", "prompt_tokens").
Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's
not valid.
Location: sentry_sdk/integrations/langchain.py#L733-L738
Potential issue: In `sentry_sdk/integrations/langchain.py`, a comment claims that key
precedence is given to LangChain's normalized names first, followed by provider-specific
names. However, the implementation for `_INPUT_TOKEN_KEYS` and `_OUTPUT_TOKEN_KEYS`
contradicts this by placing the OpenAI-style keys (`prompt_tokens`, `completion_tokens`)
before the LangChain-normalized keys (`input_tokens`, `output_tokens`). While this does
not cause a functional bug because real-world responses do not contain both sets of keys
simultaneously, it makes the code misleading for future maintainers who might rely on
the comment for understanding the logic.
3 participants
Description
The Langchain and Langgraph integrations only recorded input/output/total token counts, dropping cached input tokens and reasoning output tokens — even though
SPANDATAdefines both and the OpenAI, OpenAI Agents, and Google GenAI integrations already report them. This makes cache-discount and reasoning-token cost calculations inaccurate for Langchain/Langgraph apps.This extracts
cache_read/cached_tokensandreasoning/reasoning_tokensfrom both the LangChainusage_metadatadetail shape (input_token_details/output_token_details) and OpenAI-style details dicts (prompt_tokens_details/completion_tokens_details), and setsgen_ai.usage.input_tokens.cached/gen_ai.usage.output_tokens.reasoningin both integrations. Tests cover both detail shapes, span data emission, and the Langgraph accumulation loop (all three fail on master and pass with this change); ruff clean.Issues