Eval the shared agent context on every PR, via the Claude plugin

[ethanpalm]

Note

Medium Risk
Introduces paid Anthropic API usage in CI and changes where the Claude plugin manifest is written for sync, though evals are mocked and the eval job is a soft gate today.

Overview
Adds a behavioral eval suite under agent-context/evals/ (seven claude plugin eval cases with graders and mocked Mintlify MCP servers) to regression-test the shared skill content through the generated Claude Code plugin.

CI gains an eval-shared-content-via-claude job (after validate): sync the Claude target, copy evals, hard-fail on claude plugin validate, optionally run evals when ANTHROPIC_API_KEY is set (continue-on-error, sticky PR comment, job summary via eval-summary.mjs, artifacts). Runs on same-repo PRs or manual workflow_dispatch (optional no-plugin ablation arm). Fork PRs and push/schedule are excluded to avoid secret spend.

Claude packaging now generates .claude-plugin/plugin.json from targets/claude.json using a new per-target pluginManifestFile (Kiro still uses root plugin.json); build/sync tests and sync-agent-context.yml ship the Claude manifest path accordingly. Docs and .gitignore cover eval usage and evals/results/.

^{Reviewed by Cursor Bugbot for commit 3bec023. Bugbot is set up for automated code reviews on this repo. Configure here.}

What this does

Adds a behavioral eval suite for the shared agent context and runs it on every PR that touches agent-context/, before the content syncs to any plugin.

• agent-context/evals/ — seven claude plugin eval cases with graders and mocks for both Mintlify MCP servers. Five test what the skill teaches (docs.json shape, <Columns> vs the retired <CardGroup>, frontmatter and link rules, page mode values, a must-not-fire negative case). Two test the Admin MCP workflow the skill documents (checkout before edits; confirm before live writes) against mocked servers.
• eval-shared-content-via-claude job in agent-context-ci.yml — generates the Claude plugin from the PR branch, claude plugin validates it (free, fails hard), then runs the suite with pinned models (~$3, 5–13 min). Scores are posted as a sticky comment on the PR (one comment, updated every run), in the job summary, and as an artifact with the full HTML report.
• Claude manifest — the claude target now generates .claude-plugin/plugin.json, the only location Claude Code reads a manifest from (a root plugin.json validates but is ignored for name/version). Uses the existing pluginManifest mechanism from the kiro target plus a new per-target pluginManifestFile. The sync workflow ships it.

What it covers, and what it doesn't

The content is byte-identical across all four targets (npm run check enforces it), so this measures the words every plugin ships. It does not exercise the Codex, Cursor, or Kiro agents — only Claude Code has an eval harness. The job name and summary header say so.

It is a soft gate, on purpose

continue-on-error: true. A low score shows in the summary but does not fail the check. The two admin-* cases swing between identical passes (1.00 → 0.42 once), so a hard gate today would flap. Suggested path: merge → watch a few PRs → set the threshold at the observed floor → flip continue-on-error off → require the check in branch protection.

Needs

• An ANTHROPIC_API_KEY Actions secret. Without it the job still generates and validates the plugin, then reports that no eval ran and passes. Spend lands on the API bill.

Findings from building it (Sonnet 5, 11 runs per admin case)

• <Columns> vs <CardGroup>: plugin +0.53 over no plugin. frame page mode: +0.25. keywords: frontmatter: ~+0.67. The mint.json warning: +0.06 — the model already knows docs.json.
• Skill triggers ~73% on "add a tier to the pricing page on my Mintlify docs site." The description covers creating pages, config, components, nav, API refs — not editing a hosted site or the Admin MCP.
• ~45% of runs did an unconfirmed live write with the skill loaded, while stating the write was immediate. "Confirm the intended change first" is being read as information, not an instruction.

Both are agent-context/context/ changes, left for a follow-up.

First run on this PR

Green with no API key: generate → install Claude Code → claude plugin validate (found .claude-plugin/plugin.json) → evals skipped → summary written. Still unverified until a key exists: whether a Write-only grant needs a sandbox backend on ubuntu-latest (docs say only Bash does).

Never

--mocks off or --allow-real-servers on this suite. The Admin server writes to live deployments and eval runs never stop to ask.

🤖 Generated with Claude Code

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated
mintlify	🟢 Ready	View Preview	Sep 11, 2026, 11:50 PM

[github-actions]

Shared content eval (via Claude Code plugin)

No eval result was produced. Either ANTHROPIC_API_KEY is not set for this repository, or the run failed before writing results.

Run log, and the full HTML report under Artifacts

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit 045a50e. Configure here.}