Skip to content

fix(copilot): include copilot legacy chat sessions - #1232

Open
JaylyDev wants to merge 11 commits into
getagentseal:mainfrom
JaylyDev:jayly/copilot-legacy-json-sessions
Open

JaylyDev wants to merge 11 commits into
getagentseal:mainfrom
JaylyDev:jayly/copilot-legacy-json-sessions

Conversation

@JaylyDev

@JaylyDev JaylyDev commented Sep 2, 2026

Copy link
Copy Markdown

Summary

Adding support for VS Code's GitHub Copilot chat sessions prior to February 2026 to copilot's parser.

How it works:

The parser scan for chatSessions/*.json and parses the legacy chat sessions if there is a requests array in root JSON object. The schema of those JSON files and its definitions like IToolCallRound and ThinkingData can be found in https://github.com/microsoft/vscode-copilot-chat.

This old format reads all of input, output, reasoning messages and tool calling outputs to determine number of tokens used by a model, which is also stored in the request object. It reads exact token counts (promptTokens, completionTokens, usage) when available, otherwise it fallbacks to character-based estimation.

This parser also maps the model used in auto model selection to the actual model used.

Testing

  • I have tested this locally against real data (not just unit tests)
  • npm test passes
  • npm run build succeeds
  • I installed the tool and generated real sessions by using it
  • npm run dev -- today shows correct costs and session counts for this provider
  • npm run dev -- models --provider copilot shows correct model names and pricing
  • Screenshot or terminal output attached below proving it works with real data

Terminal output of running npm run dev -- models --provider copilot -p lifetime

Before:

image

After:

image

@ozymandiashh ozymandiashh left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for taking on the legacy format. I can't merge this as it stands: the tests pass, but the defects are all inside untested branches, and several change numbers for users on the current format too. Everything below was measured on the branch merged with current main (clean merge, full suite green apart from the known #1216 flake).

  1. Reasoning tokens are billed as web-search requests. src/providers/copilot.ts:702-709 passes reasoningTokens as the sixth argument to calculateCost, which is webSearchRequests. A session with an 8,000-char thinking block on claude-sonnet-4-6 comes out at $20.04 where the true cost is about $0.03, with costIsEstimated: false. Every other call site in the file passes 0 there.

  2. The rewrite regresses the current .jsonl path. The old parser read metadata.promptTokens and metadata.outputTokens || completionTokens because the two fields do not live in the same object (see chatSessionSampleRequest in the tests). The new code requires both in one object, so the repo's own real-data shape (metadata.promptTokens: 32543 + root completionTokens: 490) falls through to char estimation: 32,543 input tokens become 2. Rows with no token fields, previously skipped, are now emitted as estimates. That's not a behavior-preserving refactor.

  3. Model ids get mangled instead of resolved. normaliseLegacyModelId turns every dotted version into dashes and the repo has no dash-to-dot normalisation, so anything not in the aliases you added loses or changes pricing: gemini-2.0-flash-001 $0.15/Mtok to $0, gpt-4.5-preview $75 to $30 (matches a different row), gpt-5.3-codex-spark folded into the non-Spark SKU. Resolve through the alias mechanism; don't rewrite ids at the provider.

  4. Unresolvable model now falls back to claude-sonnet-4-5. Main returned unknown and $0 plus the unknown-model warning. The branch invents premium spend ($3.00 on a 1M-token request with no model info) and suppresses the diagnostic.

  5. Double-count with transcripts. hasChatSessionFiles() (line 3358) still matches .jsonl only, so a workspace with only legacy .json sessions no longer suppresses GitHub.copilot-chat/transcripts/. Both sources are discovered, dedup keys are in different namespaces, and docs/providers/copilot.md:366 documents this exact invariant. Extend the gate to .json, and key legacy sessions on session.sessionId || basename so the two formats of one session dedup against each other.

  6. Scope. models.ts display-name changes for copilot-*-auto (affects dash, menubar, JSON), deriveGeminiShortName now overriding the short-name table, eight JetBrains model tokens, the inferJetBrainsProject regex rewrite (which drops C:/repo on Windows because the downstream filter still requires a leading slash), and a win32 early-return added to an unrelated existing test. Please move all of that out; this PR should be the legacy chat sessions and nothing else.

There is no test coverage for the new cost path (grep -c 'renderedUserMessage|toolCallRounds|CacheBreakpoint' tests/ is 0), which is why CI is green. Fixtures exercising 1-5 are required for a re-review.

@iamtoruk iamtoruk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still the same six. No commits since 2026-09-04, so nothing here is addressed.

I re-measured on main plus this branch:

  • reasoning passed as webSearchRequests: $20.00 where the real cost is $0.0045, and the same tokens are also added into outputTokens
  • the existing .jsonl chatSessions path now runs through parseLegacyChatSession, so a request with metadata.promptTokens: 32543 and root completionTokens: 490 exports as 2 in / 4 out, $0.10 to $0.00
  • gemini-2.0-flash-001 $0.15 to $0.00, gpt-4.5-preview $75 to $30, gpt-5.3-codex-spark $1.75 to $1.25
  • unknown model $0.00 to $3.00 per 1M with the warning gone
  • hasChatSessionFiles is still .jsonl only while discovery takes .json, which breaks the invariant in docs/providers/copilot.md:366

One more: the else if (resultObj.usage) branch sets foundExact = true for a usage object with no token fields, so char estimates ship as costIsEstimated: false.

On my machine this PR changes nothing (no legacy sessions), but every one of those defects is live for anyone who has them. The full suite passes because no test asserts a cost.

Scope is also unchanged: 833 lines in copilot.ts and no fixtures. The copilot-*-auto display-name rename, deriveGeminiShortName, the JetBrains tokens, the inferJetBrainsProject regex and the win32 early return in an unrelated test all need to come out. models.ts should be pricing aliases for legacy ids and nothing else.

Fix the six, add fixtures that fail without the fix, drop the rest, then I will look again.

@JaylyDev
JaylyDev force-pushed the jayly/copilot-legacy-json-sessions branch from b918ee9 to b7cbe3b Compare September 15, 2026 21:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants