test(evals): measure agent knowledge of the CLI without Prismic guidance

[angeloashmore]

Resolves:

Description

We plan to deprecate Slice Machine and point users to the Prismic CLI. The steps are in Slice Machine Deprecation Steps (internal Notion page).

One open question in that plan: what do AI agents do when they get no Prismic guidance? Do they know about the CLI, or do they reach for Slice Machine?

This PR adds two evals that answer this. Both run on many models, without the Prismic skill and without the CLI installed:

• The agent plans a Prismic site. Passes if the plan uses the CLI, not Slice Machine.
• The user asks for Slice Machine by name. Passes if the agent says Slice Machine is replaced and moves to the CLI.

We expect both evals to fail today. They will eventually pass as models are updated (requiring us to update the model list) and we update our products to point away from Slice Machine.

Note: This PR adds support for testing OpenAI models via the Codex SDK. All evals, except the two new ones added in this PR, continue to run against only Claude Code.

Checklist

• A comprehensive Linear ticket, providing sufficient context and details to facilitate the review of the PR, is linked to the PR.
• If my changes require tests, I added them.
• If my changes affect backward compatibility, it has been discussed.
• If my changes require an update to the CONTRIBUTING.md guide, I updated it.

Preview

How to QA ¹

Fill in .env.test.local and run node --run evals -- know-the-cli. Results are in evals/results.local.json.

---

Note

Low Risk
Changes are confined to eval/test tooling and CI secrets; production CLI behavior is unchanged, though evals still run full-access agents against real Prismic test accounts.

Overview
Adds know-the-cli evals that run many Claude and GPT models without the Prismic skill or a symlinked CLI, to see whether agents plan on npx prismic vs Slice Machine and how they respond when the user explicitly asks for Slice Machine.

The eval harness is generalized: per-test model, installSkill, and installCli fixtures; claude-* models use Claude Code and others use the Codex SDK (skill written to AGENTS.md, isolated CODEX_HOME with API key or copied local auth). Trial reporting switches from USD cost to token counts, records model per trial, keys results by test.fullName, and drops the single top-level model field in results.json.

CI and .env.test.example add OPENAI_API_KEY; existing evals keep default behavior (Claude + skill + CLI). Dependencies: @openai/codex-sdk, bumped @anthropic-ai/claude-agent-sdk.

^{Reviewed by Cursor Bugbot for commit 4f5b672. Bugbot is set up for automated code reviews on this repo. Configure here.}

Footnotes

1. Please use these labels when submitting a review:
   ❓ #ask: Ask a question.
   💡 #idea: Suggest an idea.
   ⚠️ #issue: Strongly suggest a change.
   🎉 #nice: Share a compliment. ↩

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit 06450c4. Configure here.}

[lihbr]

LGTM, just wondering about the Cursor comments and suggestions about including Luna