Conversation
…jayly/copilot-legacy-json-sessions
…teCost web search param
ozymandiashh
left a comment
There was a problem hiding this comment.
Thanks for taking on the legacy format. I can't merge this as it stands: the tests pass, but the defects are all inside untested branches, and several change numbers for users on the current format too. Everything below was measured on the branch merged with current main (clean merge, full suite green apart from the known #1216 flake).
-
Reasoning tokens are billed as web-search requests.
src/providers/copilot.ts:702-709passesreasoningTokensas the sixth argument tocalculateCost, which iswebSearchRequests. A session with an 8,000-char thinking block on claude-sonnet-4-6 comes out at $20.04 where the true cost is about $0.03, withcostIsEstimated: false. Every other call site in the file passes0there. -
The rewrite regresses the current
.jsonlpath. The old parser readmetadata.promptTokensandmetadata.outputTokens || completionTokensbecause the two fields do not live in the same object (seechatSessionSampleRequestin the tests). The new code requires both in one object, so the repo's own real-data shape (metadata.promptTokens: 32543+ rootcompletionTokens: 490) falls through to char estimation: 32,543 input tokens become 2. Rows with no token fields, previously skipped, are now emitted as estimates. That's not a behavior-preserving refactor. -
Model ids get mangled instead of resolved.
normaliseLegacyModelIdturns every dotted version into dashes and the repo has no dash-to-dot normalisation, so anything not in the aliases you added loses or changes pricing:gemini-2.0-flash-001$0.15/Mtok to $0,gpt-4.5-preview$75 to $30 (matches a different row),gpt-5.3-codex-sparkfolded into the non-Spark SKU. Resolve through the alias mechanism; don't rewrite ids at the provider. -
Unresolvable model now falls back to
claude-sonnet-4-5. Main returnedunknownand $0 plus the unknown-model warning. The branch invents premium spend ($3.00 on a 1M-token request with no model info) and suppresses the diagnostic. -
Double-count with transcripts.
hasChatSessionFiles()(line 3358) still matches.jsonlonly, so a workspace with only legacy.jsonsessions no longer suppressesGitHub.copilot-chat/transcripts/. Both sources are discovered, dedup keys are in different namespaces, anddocs/providers/copilot.md:366documents this exact invariant. Extend the gate to.json, and key legacy sessions onsession.sessionId || basenameso the two formats of one session dedup against each other. -
Scope.
models.tsdisplay-name changes forcopilot-*-auto(affects dash, menubar, JSON),deriveGeminiShortNamenow overriding the short-name table, eight JetBrains model tokens, theinferJetBrainsProjectregex rewrite (which dropsC:/repoon Windows because the downstream filter still requires a leading slash), and awin32early-return added to an unrelated existing test. Please move all of that out; this PR should be the legacy chat sessions and nothing else.
There is no test coverage for the new cost path (grep -c 'renderedUserMessage|toolCallRounds|CacheBreakpoint' tests/ is 0), which is why CI is green. Fixtures exercising 1-5 are required for a re-review.
iamtoruk
left a comment
There was a problem hiding this comment.
Still the same six. No commits since 2026-09-04, so nothing here is addressed.
I re-measured on main plus this branch:
- reasoning passed as
webSearchRequests: $20.00 where the real cost is $0.0045, and the same tokens are also added intooutputTokens - the existing
.jsonlchatSessions path now runs throughparseLegacyChatSession, so a request withmetadata.promptTokens: 32543and rootcompletionTokens: 490exports as 2 in / 4 out, $0.10 to $0.00 gemini-2.0-flash-001$0.15 to $0.00,gpt-4.5-preview$75 to $30,gpt-5.3-codex-spark$1.75 to $1.25- unknown model $0.00 to $3.00 per 1M with the warning gone
hasChatSessionFilesis still.jsonlonly while discovery takes.json, which breaks the invariant in docs/providers/copilot.md:366
One more: the else if (resultObj.usage) branch sets foundExact = true for a usage object with no token fields, so char estimates ship as costIsEstimated: false.
On my machine this PR changes nothing (no legacy sessions), but every one of those defects is live for anyone who has them. The full suite passes because no test asserts a cost.
Scope is also unchanged: 833 lines in copilot.ts and no fixtures. The copilot-*-auto display-name rename, deriveGeminiShortName, the JetBrains tokens, the inferJetBrainsProject regex and the win32 early return in an unrelated test all need to come out. models.ts should be pricing aliases for legacy ids and nothing else.
Fix the six, add fixtures that fail without the fix, drop the rest, then I will look again.
b918ee9 to
b7cbe3b
Compare
found github.copilot-chat prefix in my chat sessions
…om/JaylyDev/codeburn into jayly/copilot-legacy-json-sessions
Summary
Adding support for VS Code's GitHub Copilot chat sessions prior to February 2026 to copilot's parser.
How it works:
The parser scan for
chatSessions/*.jsonand parses the legacy chat sessions if there is arequestsarray in root JSON object. The schema of those JSON files and its definitions likeIToolCallRoundandThinkingDatacan be found in https://github.com/microsoft/vscode-copilot-chat.This old format reads all of input, output, reasoning messages and tool calling outputs to determine number of tokens used by a model, which is also stored in the request object. It reads exact token counts (
promptTokens,completionTokens,usage) when available, otherwise it fallbacks to character-based estimation.This parser also maps the model used in auto model selection to the actual model used.
Testing
npm testpassesnpm run buildsucceedsnpm run dev -- todayshows correct costs and session counts for this providernpm run dev -- models --provider copilotshows correct model names and pricingTerminal output of running
npm run dev -- models --provider copilot -p lifetimeBefore:
After: