Skip to content

feat: preserve prompt cache across effort switches - #48513

Open
nexxeln wants to merge 2 commits into
v2from
effort-cache
Open

feat: preserve prompt cache across effort switches#48513
nexxeln wants to merge 2 commits into
v2from
effort-cache

Conversation

@nexxeln

@nexxeln nexxeln commented Sep 11, 2026

Copy link
Copy Markdown
Member

Issue for this PR

Closes #

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

Switching a model variant (effort) mid-session currently changes the top-level output_config.effort / reasoning.effort, which providers render into the prompt, so the whole prompt cache (tools, system and messages) is invalidated on every switch. Anthropic (Opus 5, Fable/Mythos 5.1) and OpenAI (gpt-6-astra) both expose a per-message alternative that keeps the prefix intact.

  • Core lowers a same-model model-switched history entry to Message.effort({ effort, previous }) instead of dropping it (to-llm-message.ts).
  • applyEffortUpdates runs in prepareRequest next to applyCachePolicy and strips the markers unless the route's protocol declares supportsEffortUpdates, so every other route behaves exactly as today.
  • Anthropic Messages freezes output_config.effort at the first marker's previous, lowers each marker to { role: "system", content: [], output_config: { effort } }, and adds the mid-conversation-output-config-2026-07-01 beta. OpenAI Responses does the same with configuration_update items (coalescing consecutive ones, off when contextManagement is set).
  • If the last marker disagrees with the effort the request asks for (reverted or forked history), markers are stripped and the plain top-level request is sent.
  • Cache-policy tail selection skips markers so a trailing marker never takes the breakpoint.

Gates: Anthropic Opus ≥ 5 / Fable, Mythos ≥ 5.1 and OpenAI gpt-6-astra, with a compatibility.supportsEffortUpdates override. Anthropic applies the change from the next user turn; OpenAI from the next response. Zen strips anthropic-beta today, so the Opus 5 marker 400s through it until Zen forwards the header — that needs to land first or Zen should advertise supportsEffortUpdates: false for Claude models. Vertex Anthropic is deliberately not forwarded (beta acceptance unverified).

How did you verify your code works?

Live against the APIs before implementing (Opus 5, Opus 4.8, Sonnet 4.6/5, gpt-5.5, gpt-6-astra incl. the ChatGPT/Codex backend, Gemini 3.5 Flash): top-level effort changes read 0 cached tokens; markers / configuration_update read the full prior prefix across multiple switches; unsupported models return the documented 400s. Then bun typecheck in ai/core/server/sdk, the full packages/ai suite, and the core runner tests (new: effort-updates.test.ts, to-llm-message cases, a runner scenario asserting the compiled Anthropic body after a switch).

Screenshots / recordings

n/a

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR

@nexxeln nexxeln changed the title feat(ai): preserve prompt cache across effort switches feat: preserve prompt cache across effort switches Sep 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant