diff --git a/.claude/harness-candidates.md b/.claude/harness-candidates.md index 8bf1b8af..988cc532 100644 --- a/.claude/harness-candidates.md +++ b/.claude/harness-candidates.md @@ -127,3 +127,62 @@ Deferred lint/test guardrails surfaced during reviews. Promote to a `CExxx` rule that environment injects the criterion. Until then CE030 stays scoped to the four top-level models; the `command_pattern`/`exclude_pattern` contract this PR changed is documented in the Field descriptions and TASK_DEFINITION_GUIDE regardless. + +## From the evalboard Path-to-GA de-tag / mature-passes fix (4e5bbc4…dd5f7e9) — TS-side guards deferred + +Context: the CExxx harness is a **Python** AST runner over `src/coder_eval/`, so none +of the invariants below are mechanizable in it. Each would need a TypeScript lint +harness (eslint config + custom rules) that `evalboard/` does not have today — +standing one up for three call sites fails the KISS/YAGNI gate. Deferring rather +than dropping; promote if a fourth TS-side invariant appears, and stand up the +harness once for all of them. + +> **Update (PR #94 review round 2).** The *execution* half of this gap is closed: +> `evalboard/` is now gated by the `evalboard` job in `.github/workflows/pr-checks.yml` +> and reachable locally via `make evalboard-verify`, so the vitest suite (including +> the pricing drift guard) is enforcement rather than documentation. What remains +> deferred below is the *static-analysis* half — eslint has still not been stood up. +> The review that prompted this round names four more candidate TS rules (raw +> `status === "SUCCESS"` outside `lib/status.ts`; DOM-global shadowing in props; +> inline copies of the tag predicate; per-run tooltip copy reused on aggregate +> surfaces), which meets the "fourth invariant" promotion bar stated above — +> **stand up eslint next time `evalboard/` is touched substantively.** + +- [ ] **"Every consumer of `RunOverviewTask.matureSkipped` must decide explicitly + whether a carry-forward row counts."** Four consumers now, and they deliberately + DISAGREE: `lib/trends.ts` and `app/runs/[id]/run-view.tsx` count a mature skip as + a pass; `lib/overview.ts::buildTagTaskRows` excludes it from both terms + (`/path-to-ga` is a GA-readiness page). A new consumer silently inheriting either + convention is a real hazard. Guard shape: flag a file that reads `.matureSkipped` + without a nearby comment naming its convention — weak, hence the deferral. Closed + for now by unit tests that assert the exclusion from BOTH numerator and denominator + (`lib/__tests__/overview.test.ts` → `describe("buildTagTaskRows")`). + +- [x] ~~**`taskCarriesRepoTag` is the single repo-provenance tag predicate — but one + duplicate survives.**~~ **RESOLVED in PR #94 review round 2.** The predicate moved to + a dependency-free `lib/tags.ts` (structurally typed on `{skill, tags}` so + `RunOverviewTask`, `TaskResultSummary` and `TaskTrend` all satisfy it), re-exported + from `lib/overview.ts` for existing callers. Both inline copies now import it: + `app/runs/[id]/run-view.tsx` (the `"use client"` one that could not before) and + `lib/trends.ts::trendMatchesTag` (a third copy the original deferral missed). + Still worth a lint rule ("no inline `tags.includes(x) || skill === x`") to catch + future copies — folded into the eslint promotion noted above. + +- [ ] **The de-tag rule fails CLOSED on a newest run that loads fine but stamps no + `tags`** (`lib/overview.ts::buildTagTaskRows`): every tagged task would read as + de-tagged and the table would empty, rendering an empty state indistinguishable from + a genuine full de-tagging. Its sibling failure mode (`overview == null`, a transient + blob read failure) IS guarded, with exactly this rationale. Currently unreachable — + 0 of ~116k date-shaped non-ad-hoc task rows in `runs-remote/` lack `tags`, and the + six zero-tag runs found are all ad-hoc (filtered upstream by id shape + `meta.adhoc`) + — so the barrier is two upstream filters rather than a check at the seam. Left + unguarded on purpose: a `if (taggedInRun.size === 0) skip the de-tag signal` guard + would also mask a real, total de-tagging. Revisit if the pipeline ever stops + stamping tags, or if a non-ad-hoc run legitimately carries zero tagged rows. + +- [ ] **Discriminating-test discipline for predicate narrowings.** Two tests in this + change passed for the wrong reason — a downstream rule (the de-tag drop) masked the + mutation they claimed to catch — and the plan leaned on a `grep` acceptance criterion + that CI never runs. Both were found by mutation-testing the suite and fixed. No + mechanizable guard; the durable lesson is: when a test names a narrowing, construct + the fixture so the row SURVIVES every other rule, or the assertion proves nothing. diff --git a/.github/workflows/pr-checks.yml b/.github/workflows/pr-checks.yml index 9f55e5b1..8692a4da 100644 --- a/.github/workflows/pr-checks.yml +++ b/.github/workflows/pr-checks.yml @@ -157,6 +157,45 @@ jobs: echo "✅ Quality gate complete!" echo "📊 All checks passed: formatting, linting, types, security, tests" + evalboard: + # The dashboard's own gate. `evalboard/` ships ~460 vitest assertions, + # including the pricing drift guard that asserts lib/pricing.ts still agrees + # with src/coder_eval/pricing.py — and until this job existed NOTHING ran + # them: not a workflow, not a Makefile target, not a pre-commit hook. The + # guard was consequently red on `main` for weeks while a 3x-wrong Opus rate + # and five unpriced in-use models shipped to the board. An unrun assertion is + # documentation, not enforcement. + # + # Deliberately NOT path-filtered. `paths:` is workflow-scoped in GitHub + # Actions, and the parity guard's whole point is that a reprice in + # src/coder_eval/pricing.py — a pure-Python diff touching no evalboard file — + # must trip it. A `evalboard/**`-only filter would skip exactly the change + # class this job exists to catch. + name: Evalboard (Types, Tests, Build) + runs-on: ubuntu-latest + timeout-minutes: 15 + steps: + - name: Checkout code + uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + + - name: Set up Node.js 20 + uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0 + with: + node-version: "20" + + - name: Enable pnpm + # The version comes from evalboard/package.json's `packageManager` field, + # so corepack pins it without a second place to keep in sync. + run: corepack enable + + - name: Install dependencies (lockfile-pinned) + working-directory: evalboard + run: pnpm install --frozen-lockfile + + - name: Verify (tsc --noEmit && vitest run && next build) + working-directory: evalboard + run: pnpm verify + no-uipath-extra: # Proves that `pip install coder-eval` (without the optional `[uipath]` # extra) yields a working framework: imports succeed, the criterion diff --git a/CLAUDE.md b/CLAUDE.md index 8875b1a6..a1402582 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -199,8 +199,14 @@ make typecheck # pyright make test # pytest make lint # custom architectural lint rules (CE001+) make verify # All of the above + coverage check (CI equivalent) + +# The JS half (evalboard/). Separate because it needs a Node/pnpm toolchain, +# but gated in CI by the `evalboard` job just like the Python side. +make evalboard-verify # tsc --noEmit + vitest + next build ``` +Editing `src/coder_eval/pricing.py` means editing `evalboard/lib/pricing.ts` too — it is a hand-copied mirror, and `evalboard/lib/__tests__/pricing-parity.test.ts` fails the build on drift in either direction. + When fixing a bug, ask: *could a custom lint rule have prevented this?* If the root cause is a mechanically detectable pattern (e.g., "always import from `coder_eval.models`", "never call blocking IO in async"), add a rule to `tests/lint/rules/` following the CE001+ pattern and wire it up in `tests/lint/runner.py`. This turns a one-time fix into permanent enforcement. See `tests/test_custom_lint.py` for how rules are tested. (Doc-surface / whole-tree rules that reason over Markdown/YAML or the entire `src/` tree rather than one `.py` AST at a time — CE026–CE031 — are not `BaseRule`s in the runner; they are wired as dedicated `@pytest.mark.lint` test classes. CE031 guards against dead config: a behavior-driving field on `SimulationConfig`/`RunLimits`/`Dataset` that no code reads by name. CE026 keeps the GitHub Action's three onboarding surfaces honest: a page's *first* Action snippet must show the agent-runtime prerequisite steps (pinned to the `action-dogfood` job that proves them in CI), a zero-install absolute next to such a snippet must name the channel it means, and every `github.com/marketplace/actions/` link plus the shields badge label must match `action.yml`'s `name:`.) Adding a user-facing field to one of the models CE030 tracks (`TaskDefinition`, `RunLimits`, `Dataset`, `SimulationConfig` — see `tests/lint/doc_schema_parity.py`) means documenting it in its guide (mention the field name as inline code) or adding an `EXEMPT` entry with a reason it is not user-authored. `make lint` fails otherwise. diff --git a/Makefile b/Makefile index 974cc6ca..0c4c5fe3 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,4 @@ -.PHONY: help install format check typecheck test test-live test-smoke verify verify-noextra clean run lint docs-indexes docker-image docker-image-full coder-eval-runtime docker-images +.PHONY: help install format check typecheck test test-live test-smoke verify verify-noextra evalboard-verify clean run lint docs-indexes docker-image docker-image-full coder-eval-runtime docker-images # Single source of the installed coder-eval version (used to tag the docker # images). Referenced lazily inside the docker recipes, so it doesn't run on @@ -56,6 +56,14 @@ verify: ## Run all verification steps (CI equivalent) # uv run bandit -r src/ -ll --format json -o bandit-report.json uv run pytest tests/ -n auto -m "not live and not lint" --cov=coder_eval --cov-report=term-missing --cov-report=xml --cov-fail-under=80 +evalboard-verify: ## Run the evalboard (Next.js dashboard) checks: tsc + vitest + next build + # The JS half of the repo. Not folded into `make verify` because it needs a + # Node/pnpm toolchain a Python-only contributor may not have — but it IS + # gated in CI by the `evalboard` job, so a red run here is a red PR. + # Includes the pricing drift guard: a reprice in src/coder_eval/pricing.py + # without the matching edit to evalboard/lib/pricing.ts fails right here. + cd evalboard && pnpm install --frozen-lockfile && pnpm verify + verify-noextra: ## Verify the framework works without the optional [uipath] extra # Build a throwaway venv that has ONLY the [dev] extra (no [uipath]); confirms # `pip install coder-eval` (without the extra) is functional. Mirrors the diff --git a/evalboard/README.md b/evalboard/README.md index 3596d6ff..11cefc5c 100644 --- a/evalboard/README.md +++ b/evalboard/README.md @@ -24,6 +24,24 @@ show up in the index — empty shells and the `latest` symlink are filtered out. daily success-rate chart and tag rails for filtering. - `/trends` — per-task pass rate and avg duration/cost/turns across the last 10 runs, with a tag filter and expandable per-task history. +- `/path-to-ga` — GA-readiness report for the tasks tagged `path-to-ga`. + Its task table deliberately answers under **stricter rules than every other + surface**, and both differences are load-bearing: + 1. **De-tagged tasks are dropped.** A `run.json` tag is a historical stamp, so + elsewhere (including `/trends`) a task de-tagged upstream lingers until the + last run that predates the removal ages out. Here a task is dropped once a + *newer* run in the window carries it without the tag — proof of removal. + A task that merely stopped appearing is unknowable, so it is kept and dated. + 2. **Mature carry-forwards are not passes.** Elsewhere a skipped-but-carried- + forward row counts as a pass; here it is excluded from both the numerator + and the denominator, so the rate reports only measured runs (`—` when + nothing executed). + The headline tile and chart above the table keep the ordinary mature-blind, + union-over-window semantics — they feed the front page — which is why they read + higher than the table, and why the page says so in prose. +- `/watchlist` — what needs attention, ranked over the recent-runs window: tasks + and skills scored on failures, regressions and turn-budget pressure + (`lib/watchlist.ts`). - `/runs/latest` — shortcut that redirects to the newest run id. - `/runs/` — run summary (pass rate, cost, duration) + one row per task. A "Download run (.zip)" button bundles the whole run folder. diff --git a/evalboard/app/path-to-ga/__tests__/task-table.test.tsx b/evalboard/app/path-to-ga/__tests__/task-table.test.tsx new file mode 100644 index 00000000..c67284a6 --- /dev/null +++ b/evalboard/app/path-to-ga/__tests__/task-table.test.tsx @@ -0,0 +1,138 @@ +import { describe, expect, test } from "vitest"; +import { render, screen } from "@testing-library/react"; +import type { TagTaskRow } from "@/lib/overview"; +import { TagTaskTable } from "../task-table"; + +// TagTaskTable is a pure props-in/JSX-out component — no router hooks, so unlike +// run-view.render.test.tsx this needs no next/navigation stub. + +function row(overrides: Partial = {}): TagTaskRow { + return { + taskId: "skill-flow-coded-agent", + skill: "uipath-maestro-flow", + appearances: 20, + matureSkips: 0, + executed: 20, + passRate: 90, + latestStatus: "SUCCESS", + latestScore: 1.0, + latestRunId: "2026-07-31_04-38-51", + latestMatureSkipped: false, + ...overrides, + }; +} + +function renderTable( + rows: TagTaskRow[], + harness: string | null = null, +) { + return render( + , + ); +} + +describe("TagTaskTable", () => { + test("shows a Mature pill instead of Passed when the latest run skipped the task", () => { + renderTable([row({ latestMatureSkipped: true })]); + expect(screen.getByText("Mature")).toBeInTheDocument(); + expect(screen.queryByText("Passed")).not.toBeInTheDocument(); + }); + + test("shows Passed and the numeric score for an ordinary executed row", () => { + renderTable([row({ latestScore: 0.75 })]); + expect(screen.getByText("Passed")).toBeInTheDocument(); + expect(screen.getByText("0.75")).toBeInTheDocument(); + }); + + test("dashes out the latest score on a mature row", () => { + // 1.0 on a carry-forward row is inherited, not measured — showing it + // beside a Mature pill would read as a fresh result. + renderTable([row({ latestMatureSkipped: true, latestScore: 1.0 })]); + expect(screen.queryByText("1.00")).not.toBeInTheDocument(); + // Exactly one cell dashes — the score. Every other column on the default + // row is populated, so this pins WHICH cell went un-measured. + expect(screen.getAllByText("—")).toHaveLength(1); + }); + + test("annotates Appearances with the mature count, and only when non-zero", () => { + renderTable([row({ appearances: 24, matureSkips: 3 })]); + expect(screen.getByText("(3 mature)")).toBeInTheDocument(); + // The raw count stays plain beside it (getByText matches an element's own + // direct text nodes, so this is the cell's "24", not "24 (3 mature)"). + expect(screen.getByText("24")).toBeInTheDocument(); + }); + + test("no mature annotation when nothing was skipped", () => { + renderTable([row({ appearances: 24, matureSkips: 0 })]); + expect(screen.queryByText(/mature\)/)).not.toBeInTheDocument(); + }); + + test("renders an em dash for an unmeasured pass rate", () => { + // Every appearance was a carry-forward → nothing executed → no rate. + // Must not read as NaN% or a measured 0%. `latestMatureSkipped` is true + // by construction here: buildTagTaskRows reads latest* off one of the + // counted appearances, so matureSkips === appearances forces it — the + // score dashes too, hence two dashes rather than one. + renderTable([ + row({ + appearances: 4, + matureSkips: 4, + passRate: null, + latestMatureSkipped: true, + }), + ]); + expect(screen.queryByText(/NaN/)).not.toBeInTheDocument(); + expect(screen.queryByText("0%")).not.toBeInTheDocument(); + expect(screen.getAllByText("—")).toHaveLength(2); + }); + + test("the pass-rate tooltip names the row's own executed denominator", () => { + // Reads `executed` off the row rather than re-deriving it, so the + // caption and the percentage can't describe different rules. The row + // below is deliberately inconsistent (executed=5 against 24-3=21) — + // only a tooltip sourced from the field can report 5. + renderTable([row({ appearances: 24, matureSkips: 3, executed: 5 })]); + expect(screen.getByText("90%").closest("td")).toHaveAttribute( + "title", + expect.stringContaining("Measured over 5 executed appearances"), + ); + }); + + test("the mature annotation speaks in the window's voice, not one run's", () => { + renderTable([row({ appearances: 24, matureSkips: 3 })]); + const title = screen.getByText("(3 mature)").getAttribute("title"); + expect(title).toContain("3 of these 24 appearances"); + // The per-run string would say "this run" on a page that renders none. + expect(title).not.toContain("this run"); + }); + + test("Last appearance shows the date half of the latest run id", () => { + renderTable([row({ latestRunId: "2026-07-16_04-24-15" })]); + expect(screen.getByText("2026-07-16")).toBeInTheDocument(); + }); + + test("empty rows render the empty state naming the tag and window", () => { + // Newly reachable: a tag whose every task was de-tagged yields [] where + // it previously yielded stale rows. + renderTable([]); + expect( + screen.getByText(/No tasks tagged path-to-ga in the last 30d\./), + ).toBeInTheDocument(); + }); + + test("the pooled-across-harnesses note tracks the harness prop", () => { + const { unmount } = renderTable([row()], null); + expect(screen.getByText(/pooled across harnesses/)).toBeInTheDocument(); + unmount(); + + renderTable([row()], "claude-code"); + expect( + screen.queryByText(/pooled across harnesses/), + ).not.toBeInTheDocument(); + }); +}); diff --git a/evalboard/app/path-to-ga/page.tsx b/evalboard/app/path-to-ga/page.tsx index 1432ddb5..5d6f5ceb 100644 --- a/evalboard/app/path-to-ga/page.tsx +++ b/evalboard/app/path-to-ga/page.tsx @@ -1,18 +1,15 @@ -import Link from "next/link"; import { + avgRunSuccessRate, getOverview, getTagTaskBreakdown, listRecentHarnesses, } from "@/lib/overview"; import { parseHarnessScope } from "@/lib/harness"; -import { humanizeTaskId } from "@/lib/format"; -import { passClass } from "@/lib/pass-rate"; import { HarnessSelector } from "../_components/harness-selector"; import { harnessShortLabel } from "../_components/harness-badge"; import { type Window } from "@/lib/reviews-types"; import { DailySuccessChart } from "../_overview/daily-chart"; -import { TableScroll } from "../_components/scroll-table"; -import { StatusPill } from "@/lib/pills"; +import { TagTaskTable } from "./task-table"; export const dynamic = "force-dynamic"; @@ -42,11 +39,7 @@ export default async function PathToGaPage({ ]); const runsInWindow = overview.runs.length; - const avgPassRate = - runsInWindow > 0 - ? overview.runs.reduce((sum, r) => sum + (r.successRate ?? 0), 0) / - runsInWindow - : null; + const avgPassRate = avgRunSuccessRate(overview.runs); return (
@@ -98,10 +91,22 @@ export default async function PathToGaPage({ {taskRows.length}
- distinct task{taskRows.length === 1 ? "" : "s"} + distinct task{taskRows.length === 1 ? "" : "s"} still + tagged
+ {/* The tile above and the chart below keep their original + mature-blind, union-over-the-window semantics (they feed the + front page and every tag-filtered view); the table does not. + Say so, rather than let the two silently disagree. */} +

+ The rate above and the chart cover every run that carried a{" "} + {TAG} task at the time it + ran, counting mature carry-forwards as passes. The table + below is narrower: only tasks still carrying the tag, scored + on runs that actually executed. +

{runsInWindow > 0 ? ( -
-
-

- Tasks -

- {/* Unscoped, a task's appearances span harnesses, so its rate - pools regimes that aren't strictly comparable. Say so - rather than let the number read as one harness's. */} - {!harness && ( - - pooled across harnesses · pick one above to separate - them - - )} -
- - - - - - - - - - - - - - {taskRows.map((r) => ( - - - - - - - - - ))} - {taskRows.length === 0 && ( - - - - )} - -
- Task - - Skill - - Appearances - - Pass rate - - Latest status - - Latest score -
- - {humanizeTaskId(r.taskId)} - -
- {r.taskId} -
- 		- {r.skill ?? "—"} - - {r.appearances} - - - {r.passRate.toFixed(0)}% - - - - - {r.latestScore != null - ? r.latestScore.toFixed(2) - : "—"} -
- No tasks tagged {TAG} in the last{" "} - {WINDOW}. -
- -
+ ); } diff --git a/evalboard/app/path-to-ga/task-table.tsx b/evalboard/app/path-to-ga/task-table.tsx new file mode 100644 index 00000000..82952445 --- /dev/null +++ b/evalboard/app/path-to-ga/task-table.tsx @@ -0,0 +1,197 @@ +import Link from "next/link"; +import type { TagTaskRow } from "@/lib/overview"; +import { fmtRunDate, humanizeTaskId } from "@/lib/format"; +import { passClass } from "@/lib/pass-rate"; +import { matureAggregateTooltip, MaturePill, StatusPill } from "@/lib/pills"; +import { TableScroll } from "../_components/scroll-table"; +import { type Window } from "@/lib/reviews-types"; + +// The Path-to-GA task table. Split out of page.tsx (which stays the async IO +// shell) purely so it is render-testable in jsdom — mirrors the +// app/runs/[id]/page.tsx + run-view.tsx split. No "use client": this holds no +// state and no handlers, and a server component may render the "use client" +// TableScroll as a child. +// +// Every row here is scored on runs that ACTUALLY EXECUTED (see +// lib/overview.ts::buildTagTaskRows), which is narrower than the headline tile +// and chart above it — hence the caveat paragraph in page.tsx. A mature +// carry-forward's inherited status/score are dashed out rather than shown as if +// they were measured (same idiom as app/trends/trends-view.tsx; note +// app/runs/[id]/task-grid.tsx deliberately still shows the carried-forward 1.00 +// beside its own MaturePill, so the two surfaces differ). Not extracted into a +// shared helper — the column shapes differ, so it would be a wrapper around a +// ternary. +function passRateTooltip(r: TagTaskRow): string { + // Read off the row, never re-derived: if the exclusion set ever widens the + // caption must move with the percentage it describes. + const executed = r.executed; + if (executed === 0) { + return ( + `Not executed once in this window — all ${r.appearances} appearance` + + `${r.appearances === 1 ? "" : "s"} were mature carry-forwards, so ` + + "there is no measured pass rate." + ); + } + return ( + `Measured over ${executed} executed appearance` + + `${executed === 1 ? "" : "s"}` + + (r.matureSkips > 0 + ? ` (${r.matureSkips} mature carry-forward${r.matureSkips === 1 ? "" : "s"} excluded).` + : ".") + ); +} + +// `windowLabel`, not `window`: a prop named `window` shadows the DOM global +// inside the component body, and sibling components in this tree genuinely use +// that global (app/_components/scroll-table.tsx, app/_components/search-box.tsx). +export function TagTaskTable({ + rows, + tag, + windowLabel, + harness, +}: { + rows: TagTaskRow[]; + tag: string; + windowLabel: Window; + harness: string | null; +}) { + return ( +
+
+

Tasks

+ {/* Unscoped, a task's appearances span harnesses, so its rate + pools regimes that aren't strictly comparable — and the row + then mixes scopes: Appearances/Pass rate are pooled, while + Last appearance and the two Latest columns describe ONE run (and + maturity is per-harness pipeline state, so a Mature pill can + legitimately sit beside a middling pooled rate). Say both, + rather than let either read as one harness's. */} + {!harness && ( + + pooled across harnesses · Last appearance and Latest + describe a + single run · pick one above to separate them + + )} +
+ + + + + + + + + + + + + + + {rows.map((r) => ( + + + + + {/* No conditional dimming: under harness rotation + "newest" differs per harness, so a relative + highlight would mislead. */} + + {/* The denominator is `appearances - matureSkips`, + which is NOT the Appearances shown two cells + left — a mature task is re-validated about + weekly, so a 24-appearance row can rest on a + single executed run and still tint green. + Name the sample size on hover so a confident + percentage can't hide a tiny denominator. */} + + + + + ))} + {rows.length === 0 && ( + + + + )} + +
TaskSkill + Appearances + + Last appearance + + Pass rate + + Latest status + + Latest score +
+ + {humanizeTaskId(r.taskId)} + +
+ {r.taskId} +
+ 		+ {r.skill ?? "—"} + + {r.appearances} + {r.matureSkips > 0 && ( + + {" "} + ({r.matureSkips} mature) + + )} + + {fmtRunDate(r.latestRunId)} + + + {r.passRate != null + ? `${r.passRate.toFixed(0)}%` + : "—"} + + + {r.latestMatureSkipped ? ( + + ) : ( + + )} + + {r.latestMatureSkipped || + r.latestScore == null + ? "—" + : r.latestScore.toFixed(2)} +
+ No tasks tagged {tag} in the last{" "} + {windowLabel}. +
+ +
+ ); +} diff --git a/evalboard/app/runs/[id]/run-view.tsx b/evalboard/app/runs/[id]/run-view.tsx index 1aac1316..fd73589d 100644 --- a/evalboard/app/runs/[id]/run-view.tsx +++ b/evalboard/app/runs/[id]/run-view.tsx @@ -7,6 +7,7 @@ import type { ReviewIndexEntry } from "@/lib/reviews-types"; import { fmtDuration, humanizeTaskId } from "@/lib/format"; import { passBarClass, passClass } from "@/lib/pass-rate"; import { perTaskPassCounts, statusCategory } from "@/lib/status"; +import { taskCarriesRepoTag } from "@/lib/tags"; import { ChipLegend } from "@/app/_overview/tag-rail"; import { CollapsibleRail } from "@/app/_components/collapsible-rail"; import { ActivationCard } from "./activation-card"; @@ -279,9 +280,7 @@ export function RunView({ // `tags` URL param works for both rails. Robust to new runs where // the skill comes from task_path but is missing from task.tags. arr = arr.filter((t) => - selectedTags.every( - (tag) => t.tags.includes(tag) || t.skill === tag, - ), + selectedTags.every((tag) => taskCarriesRepoTag(t, tag)), ); } if (selectedReviewTags.length > 0) { diff --git a/evalboard/lib/__tests__/format.test.ts b/evalboard/lib/__tests__/format.test.ts index 137a625d..868a6a08 100644 --- a/evalboard/lib/__tests__/format.test.ts +++ b/evalboard/lib/__tests__/format.test.ts @@ -1,5 +1,5 @@ import { describe, expect, test } from "vitest"; -import { fmtRunTime, fmtTimestamp } from "../format"; +import { fmtRunDate, fmtRunTime, fmtTimestamp } from "../format"; describe("fmtRunTime", () => { test("reformats a daily-pipeline run id into a readable timestamp", () => { @@ -23,6 +23,20 @@ describe("fmtRunTime", () => { }); }); +describe("fmtRunDate", () => { + test("keeps only the date half of a daily-pipeline run id", () => { + expect(fmtRunDate("2026-07-16_04-24-15")).toBe("2026-07-16"); + }); + + test("returns ad-hoc run ids verbatim rather than splitting on the underscore", () => { + // A bare .split("_")[0] would hand back "adhoc-2026-07-25" / "codex". + expect(fmtRunDate("adhoc-2026-07-25_09-19-36")).toBe( + "adhoc-2026-07-25_09-19-36", + ); + expect(fmtRunDate("codex_skills_full_v2")).toBe("codex_skills_full_v2"); + }); +}); + describe("fmtTimestamp", () => { test("formats an ISO start_time into the fmtRunTime shape", () => { // run.json start_time carries microseconds and no timezone; the literal diff --git a/evalboard/lib/__tests__/overview.test.ts b/evalboard/lib/__tests__/overview.test.ts index 2f8b9440..c518ee2d 100644 --- a/evalboard/lib/__tests__/overview.test.ts +++ b/evalboard/lib/__tests__/overview.test.ts @@ -1,10 +1,14 @@ import { describe, expect, test, vi } from "vitest"; import { + avgRunSuccessRate, buildAdhocRows, + buildTagTaskRows, collectPipelineRuns, projectRunRow, scopeRunTasks, summarizeListing, + taskCarriesRepoTag, + taskMatchesTag, turnBudgetRateForTasks, type PerRun, type RunListingRow, @@ -491,6 +495,416 @@ describe("buildAdhocRows", () => { }); }); +// buildTagTaskRows drives /path-to-ga. Two behaviours are asserted here that no +// other test covers: a task de-tagged upstream (present in a newer run WITHOUT +// the tag) must disappear, and a mature carry-forward must leave both the +// numerator AND the denominator of passRate — the page-local divergence from +// trends.ts, which counts a carry-forward as a pass. +describe("buildTagTaskRows", () => { + const TAG = "path-to-ga"; + + function perRun(id: string, tasks: RunOverviewTask[]): PerRun { + return { + id, + overview: { + id, + tasks, + totalCostUsd: null, + taskDurationSeconds: null, + componentShas: [], + }, + reviewTagCounts: {}, + reviewTagsByTask: {}, + adhoc: false, + title: null, + }; + } + + test("keeps a task still tagged in the newest run", () => { + const rows = buildTagTaskRows( + [ + perRun("r1", [task({ taskId: "a", tags: [TAG] })]), + perRun("r2", [task({ taskId: "a", tags: [TAG] })]), + ], + TAG, + ); + expect(rows.map((r) => r.taskId)).toEqual(["a"]); + expect(rows[0].appearances).toBe(2); + expect(rows[0].latestRunId).toBe("r2"); + }); + + test("drops a task present-but-untagged in a newer run (the de-tag bug)", () => { + // Models ipe-drive-to-slack: tagged in r1, still running in the newer r2 + // but with the tag removed from its YAML. That is proof of de-tagging, so + // the row must vanish rather than linger until r1 ages out of the window. + const rows = buildTagTaskRows( + [ + perRun("r1", [task({ taskId: "detagged", tags: [TAG] })]), + perRun("r2", [task({ taskId: "detagged", tags: ["other"] })]), + ], + TAG, + ); + expect(rows).toEqual([]); + }); + + test("keeps a task that simply stopped appearing, dated to its newest tagged run", () => { + // Retired / renamed / skip:true is unknowable from run data, so the row + // stays and latestRunId is what the page renders as "Last seen". + const rows = buildTagTaskRows( + [ + perRun("r1", [task({ taskId: "gone", tags: [TAG] })]), + perRun("r2", [task({ taskId: "other", tags: [TAG] })]), + ], + TAG, + ); + expect(rows.map((r) => r.taskId)).toEqual(["gone", "other"]); + expect(rows[0].latestRunId).toBe("r1"); + }); + + test("one untagged replicate in the newest run is not a de-tag", () => { + // A replicated task has several rows per run; the de-tag check collapses + // with any-row semantics, so a single untagged replicate cannot drop it. + const rows = buildTagTaskRows( + [ + perRun("r1", [ + task({ taskId: "a", tags: [TAG] }), + task({ taskId: "a", tags: [] }), + ]), + ], + TAG, + ); + expect(rows.map((r) => r.taskId)).toEqual(["a"]); + // Only the tagged row accumulates. + expect(rows[0].appearances).toBe(1); + }); + + test("matureSkips counts carry-forwards; appearances still includes them", () => { + const rows = buildTagTaskRows( + [ + perRun("r1", [task({ taskId: "a", tags: [TAG] })]), + perRun("r2", [ + task({ taskId: "a", tags: [TAG], matureSkipped: true }), + ]), + ], + TAG, + ); + expect(rows[0].appearances).toBe(2); + expect(rows[0].matureSkips).toBe(1); + }); + + test("passRate excludes mature skips from both numerator and denominator", () => { + // 4 appearances, 1 mature skip, 2 executed passes out of 3 executed rows + // → 66.67%. The old mature-inclusive rule would have said 75% (3/4). + const rows = buildTagTaskRows( + [ + perRun("r1", [task({ taskId: "a", tags: [TAG] })]), + perRun("r2", [ + task({ taskId: "a", tags: [TAG], status: "FAILURE" }), + ]), + perRun("r3", [task({ taskId: "a", tags: [TAG] })]), + perRun("r4", [ + task({ taskId: "a", tags: [TAG], matureSkipped: true }), + ]), + ], + TAG, + ); + expect(rows[0].appearances).toBe(4); + expect(rows[0].matureSkips).toBe(1); + expect(rows[0].passRate).toBeCloseTo(66.6667, 3); + }); + + test("passRate is a measured 0 — not null — when every executed run failed", () => { + // The boundary that separates "no data" from "the worst task on the GA + // board". Both render through the same `passRate != null` ternary, so + // gating on `executedPasses > 0` instead of `executed > 0` would dash + // out the one row that most needs to read red. + const rows = buildTagTaskRows( + [ + perRun("r1", [ + task({ taskId: "a", tags: [TAG], status: "FAILURE" }), + ]), + perRun("r2", [ + task({ taskId: "a", tags: [TAG], status: "FAILURE" }), + ]), + ], + TAG, + ); + expect(rows[0].passRate).toBe(0); + expect(rows[0].matureSkips).toBe(0); + expect(rows[0].executed).toBe(2); + }); + + test("executed is the passRate denominator, carried on the row", () => { + // The table's tooltip names the sample size from this field instead of + // re-deriving it, so the caption cannot describe a different rule than + // the percentage beside it. + const rows = buildTagTaskRows( + [ + perRun("r1", [task({ taskId: "a", tags: [TAG] })]), + perRun("r2", [ + task({ taskId: "a", tags: [TAG], matureSkipped: true }), + ]), + perRun("r3", [ + task({ taskId: "a", tags: [TAG], matureSkipped: true }), + ]), + ], + TAG, + ); + expect(rows[0].appearances).toBe(3); + expect(rows[0].executed).toBe(1); + expect(rows[0].passRate).toBe(100); + }); + + test("a non-SUCCESS terminal status is not a pass", () => { + // Pins the pass predicate to lib/status.ts::isPassStatus rather than a + // raw literal: ERROR and TIMEOUT are executed appearances that failed, + // so they belong in the denominator and out of the numerator. + const rows = buildTagTaskRows( + [ + perRun("r1", [ + task({ taskId: "a", tags: [TAG], status: "ERROR" }), + ]), + perRun("r2", [ + task({ taskId: "a", tags: [TAG], status: "TIMEOUT" }), + ]), + perRun("r3", [task({ taskId: "a", tags: [TAG] })]), + ], + TAG, + ); + expect(rows[0].executed).toBe(3); + expect(rows[0].passRate).toBeCloseTo(33.3333, 3); + }); + + test("passRate is null when every tagged appearance was a mature skip", () => { + // Nothing was measured, so the page must show "—" rather than a + // measured-looking 100% (or a divide-by-zero NaN). + const rows = buildTagTaskRows( + [ + perRun("r1", [ + task({ taskId: "a", tags: [TAG], matureSkipped: true }), + ]), + perRun("r2", [ + task({ taskId: "a", tags: [TAG], matureSkipped: true }), + ]), + ], + TAG, + ); + expect(rows[0].matureSkips).toBe(2); + expect(rows[0].passRate).toBeNull(); + // Producer invariant the table's rendering relies on: latest* is read + // off one of the counted appearances, so "nothing executed" necessarily + // means the latest appearance was a skip. The table dashes BOTH cells on + // the strength of this — it can never show a measured-looking score + // beside an unmeasured rate. + expect(rows[0].latestMatureSkipped).toBe(true); + }); + + test("latestMatureSkipped reflects the newest tagged appearance", () => { + const skippedLatest = buildTagTaskRows( + [ + perRun("r1", [task({ taskId: "a", tags: [TAG] })]), + perRun("r2", [ + task({ taskId: "a", tags: [TAG], matureSkipped: true }), + ]), + ], + TAG, + ); + expect(skippedLatest[0].latestMatureSkipped).toBe(true); + + const executedLatest = buildTagTaskRows( + [ + perRun("r1", [ + task({ taskId: "a", tags: [TAG], matureSkipped: true }), + ]), + perRun("r2", [task({ taskId: "a", tags: [TAG] })]), + ], + TAG, + ); + expect(executedLatest[0].latestMatureSkipped).toBe(false); + }); + + test("a tag matched via skill is never dropped", () => { + // taskCarriesRepoTag's first clause: every run containing the task + // matches, so `tagged` is always true. Using a raw tags.includes() here + // would wrongly drop every skill-tag row. + const rows = buildTagTaskRows( + [ + perRun("r1", [task({ taskId: "a", skill: TAG })]), + perRun("r2", [task({ taskId: "a", skill: TAG })]), + ], + TAG, + ); + expect(rows.map((r) => r.taskId)).toEqual(["a"]); + }); + + test("a task carrying the tag only as a review tag does not appear", () => { + // Review tags are a post-hoc, effectively disjoint namespace; this page + // reports repo-declared tags only. + const r = perRun("r1", [task({ taskId: "a", tags: [] })]); + const rows = buildTagTaskRows( + [{ ...r, reviewTagsByTask: { a: [TAG] } }], + TAG, + ); + expect(rows).toEqual([]); + }); + + test("a review-tagged older appearance is not folded into a kept row", () => { + // Discriminates the accumulate path specifically: the row IS kept (its + // newest run carries the repo tag), so only `appearances` reveals which + // predicate accumulated. Widening the accumulate path back to + // taskMatchesTag would count r1's review-tag-only row too and report 2. + const older = perRun("r1", [task({ taskId: "a", tags: [] })]); + const rows = buildTagTaskRows( + [ + { ...older, reviewTagsByTask: { a: [TAG] } }, + perRun("r2", [task({ taskId: "a", tags: [TAG] })]), + ], + TAG, + ); + expect(rows).toHaveLength(1); + expect(rows[0].appearances).toBe(1); + }); + + test("a task that only recently GAINED the tag is kept", () => { + // Mirror image of the de-tag case, and the reason the newest-appearance + // rule is first-write-wins rather than "tagged in every appearance": a + // task tagged on day 20 of a 30-day window is present-but-untagged in the + // older runs, and must not read as de-tagged. + const rows = buildTagTaskRows( + [ + perRun("r1", [task({ taskId: "a", tags: [] })]), + perRun("r2", [task({ taskId: "a", tags: [TAG] })]), + ], + TAG, + ); + expect(rows.map((r) => r.taskId)).toEqual(["a"]); + expect(rows[0].appearances).toBe(1); + expect(rows[0].latestRunId).toBe("r2"); + }); + + test("a newest run with a null overview neither adds nor drops anything", () => { + // A transient blob failure on the newest run must not read as a de-tag of + // every row. + const broken: PerRun = { + id: "r9", + overview: null, + reviewTagCounts: {}, + reviewTagsByTask: {}, + adhoc: false, + title: null, + }; + const rows = buildTagTaskRows( + [broken, perRun("r1", [task({ taskId: "a", tags: [TAG] })])], + TAG, + ); + expect(rows.map((r) => r.taskId)).toEqual(["a"]); + expect(rows[0].appearances).toBe(1); + expect(rows[0].latestRunId).toBe("r1"); + }); + + test("empty input returns an empty list", () => { + expect(buildTagTaskRows([], TAG)).toEqual([]); + }); + + test("rows come back sorted by taskId", () => { + const rows = buildTagTaskRows( + [ + perRun("r1", [ + task({ taskId: "c", tags: [TAG] }), + task({ taskId: "a", tags: [TAG] }), + task({ taskId: "b", tags: [TAG] }), + ]), + ], + TAG, + ); + expect(rows.map((r) => r.taskId)).toEqual(["a", "b", "c"]); + }); + + test("all three latest* fields come off the same row on replicate disagreement", () => { + // The newest run has an executed replicate and a carried-forward one. + // First-row-wins decides, and the Mature pill must never sit beside a + // measured score from the other replicate. + const rows = buildTagTaskRows( + [ + perRun("r1", [ + task({ + taskId: "a", + tags: [TAG], + matureSkipped: true, + status: "SUCCESS", + weightedScore: 1.0, + }), + task({ + taskId: "a", + tags: [TAG], + status: "FAILURE", + weightedScore: 0.25, + }), + ]), + ], + TAG, + ); + expect(rows[0].latestMatureSkipped).toBe(true); + expect(rows[0].latestStatus).toBe("SUCCESS"); + expect(rows[0].latestScore).toBe(1.0); + // Both replicates are tagged, so `appearances` counts ROWS (2), not runs + // (1) — the semantics the interface comment promises. + expect(rows[0].appearances).toBe(2); + expect(rows[0].matureSkips).toBe(1); + }); +}); + +// The Path-to-GA headline tile. Hoisted out of the page shell so the +// null-successRate rule is assertable without rendering an async server +// component. +describe("avgRunSuccessRate", () => { + test("averages over the runs that report a rate", () => { + expect( + avgRunSuccessRate([{ successRate: 100 }, { successRate: 50 }]), + ).toBe(75); + }); + + test("a run with no measurable rate is excluded, not counted as 0", () => { + // Dividing by the full run count instead would report 50% here and drag + // the headline down every time a run.json fails to load. + expect( + avgRunSuccessRate([{ successRate: 100 }, { successRate: null }]), + ).toBe(100); + }); + + test("null when nothing in scope reports a rate", () => { + expect(avgRunSuccessRate([])).toBeNull(); + expect(avgRunSuccessRate([{ successRate: null }])).toBeNull(); + }); + + test("0 is a measured rate, not an absence", () => { + expect(avgRunSuccessRate([{ successRate: 0 }])).toBe(0); + }); +}); + +// taskMatchesTag was rebuilt on top of the extracted taskCarriesRepoTag so +// buildTagTaskRows could reuse the repo-provenance half. scopeRunTasks and the +// front-page rails still go through taskMatchesTag and legitimately filter on +// review tags, so the extraction has to be behaviour-preserving. +describe("taskCarriesRepoTag / taskMatchesTag", () => { + test("taskCarriesRepoTag matches skill and YAML tags, not review tags", () => { + expect(taskCarriesRepoTag(task({ skill: "x" }), "x")).toBe(true); + expect(taskCarriesRepoTag(task({ tags: ["x"] }), "x")).toBe(true); + expect(taskCarriesRepoTag(task({ tags: ["y"] }), "x")).toBe(false); + }); + + test("taskMatchesTag still matches via skill, tags, or a review tag", () => { + expect(taskMatchesTag(task({ skill: "x" }), {}, "x")).toBe(true); + expect(taskMatchesTag(task({ tags: ["x"] }), {}, "x")).toBe(true); + expect( + taskMatchesTag(task({ taskId: "t" }), { t: ["x"] }, "x"), + ).toBe(true); + expect(taskMatchesTag(task({ taskId: "t", tags: ["y"] }), {}, "x")).toBe( + false, + ); + }); +}); + // projectRunRow is the single definition of "does this run count, and with which // tasks" — the summary tiles (getWindowRollup) and the paged run table // (getRunListing) both go through it. They used to be one loop; if they ever diff --git a/evalboard/lib/__tests__/pricing-parity.test.ts b/evalboard/lib/__tests__/pricing-parity.test.ts index 7e5e1a67..0907fd94 100644 --- a/evalboard/lib/__tests__/pricing-parity.test.ts +++ b/evalboard/lib/__tests__/pricing-parity.test.ts @@ -44,9 +44,20 @@ function parsePythonTable(): Record< describe("pricing.ts ↔ pricing.py parity", () => { const py = parsePythonTable(); - test("parses a non-trivial Python table", () => { - // Guard against a regex/path regression silently passing the test. - expect(Object.keys(py).length).toBeGreaterThan(10); + test("parses every ModelPricing row in the Python table", () => { + // A "> 10" floor is not enough: the guard NARROWS silently if the regex + // stops matching some rows (a `ruff format` reflow onto several lines, a + // switch to keyword args), and a narrowed guard stops reporting exactly + // the class of omission this file exists to catch. Count the constructor + // calls in the source and require the parse to have found all of them. + const declared = ( + readFileSync(PY_PATH, "utf8").match(/^\s*"[^"]+":\s*ModelPricing\(/gm) ?? [] + ).length; + expect(declared).toBeGreaterThan(10); + expect( + Object.keys(py).length, + "ROW_RE missed a pricing.py row — the parity guard is narrower than it looks", + ).toBe(declared); }); test("every model in lib/pricing.ts exists in pricing.py", () => { @@ -71,20 +82,24 @@ describe("pricing.ts ↔ pricing.py parity", () => { }); // Python-priced models we deliberately do NOT mirror to the frontend: heavy - // frontier Claude/GPT variants the evalboard never runs, so pricing them here - // adds nothing. Kept explicit (not a blanket "ignore extras") so a NEW model - // added to pricing.py that ISN'T here and ISN'T in PRICING breaks the build — - // catching a real litellm-relevant omission (e.g. the Bedrock open-weight ids - // that previously rendered "—" for cost). + // frontier variants no harness runs, so pricing them here adds nothing. Kept + // explicit (not a blanket "ignore extras") so a NEW model added to pricing.py + // that ISN'T here and ISN'T in PRICING breaks the build — catching a real + // litellm-relevant omission (e.g. the Bedrock open-weight ids that previously + // rendered "—" for cost). + // + // KEEP THIS SET HONEST. It silences the drift guard, so a stale entry hides a + // live bug rather than a non-issue: `claude-sonnet-5`, `gpt-5.6-sol`, + // `gpt-5.6-terra` and `gpt-5.6-luna` sat here under "the evalboard never runs + // them" while appearing ~32k / ~2k / ~17k / ~2k times in `runs-remote/`, so + // every one of those runs rendered "—" for cost with nothing failing. Before + // adding an id, grep the corpus for it — absence from run data is the ONLY + // justification, and it expires the moment a harness adopts the model. const DELIBERATELY_UNMIRRORED = new Set([ - "claude-sonnet-5", "gpt-5.4-mini", "gpt-5.4-nano", "gpt-5.4-pro", "gpt-5.5-pro", - "gpt-5.6-sol", - "gpt-5.6-terra", - "gpt-5.6-luna", // OpenRouter open-weight models: priced in pricing.py only for the Python // max_usd static fallback. The evalboard deliberately does NOT statically // price them — OpenRouter routes per-request, so it shows the captured @@ -94,6 +109,18 @@ describe("pricing.ts ↔ pricing.py parity", () => { "deepseek/deepseek-v4-pro", ]); + test("every DELIBERATELY_UNMIRRORED id still exists in pricing.py", () => { + // Stale-membership guard. An exemption silences the drift guard for one + // id forever; once the id leaves pricing.py the entry silences nothing + // and only survives to be copied. Making that a build failure is what + // forces the set to be re-read rather than appended to. + const stale = [...DELIBERATELY_UNMIRRORED].filter((m) => !(m in py)); + expect( + stale, + `exempted from the mirror but no longer priced in pricing.py — drop them from DELIBERATELY_UNMIRRORED: ${stale.join(", ")}`, + ).toEqual([]); + }); + test("every pricing.py model is mirrored in pricing.ts or explicitly unmirrored", () => { const missing = Object.keys(py).filter((m) => !(m in PRICING) && !DELIBERATELY_UNMIRRORED.has(m)); expect( diff --git a/evalboard/lib/__tests__/pricing.test.ts b/evalboard/lib/__tests__/pricing.test.ts index 66302cf1..7c6ea53c 100644 --- a/evalboard/lib/__tests__/pricing.test.ts +++ b/evalboard/lib/__tests__/pricing.test.ts @@ -35,8 +35,16 @@ describe("resolvePricing", () => { expect(resolvePricing("__proto__")).toBeNull(); }); - test("knows the current default opus id", () => { - expect(resolvePricing("claude-opus-4-8")?.outputPerMTok).toBe(75); + test("knows the current default opus id, at the repriced tier", () => { + // Opus 4.5 REPRICED the family from $15/$75 to $5/$25 per Mtok. The two + // generations differ 3x, so pin the boundary: an id on the wrong side of + // it triples (or thirds) every Opus cost the board renders, which reads + // as a plausible number rather than an obvious error. + // pricing-parity.test.ts is the authority on the rates themselves; this + // asserts the split survives an edit to the table. + expect(resolvePricing("claude-opus-4-8")?.outputPerMTok).toBe(25); + expect(resolvePricing("claude-opus-5")?.outputPerMTok).toBe(25); + expect(resolvePricing("claude-opus-4-1")?.outputPerMTok).toBe(75); }); test("strips LiteLLM/Bedrock routing + region prefixes (recorded model_used is qualified)", () => { diff --git a/evalboard/lib/format.ts b/evalboard/lib/format.ts index d801c526..a1f6241f 100644 --- a/evalboard/lib/format.ts +++ b/evalboard/lib/format.ts @@ -17,6 +17,15 @@ export function fmtRunTime(id: string): string { return `${d} · ${t.replace(/-/g, ":")}`; } +// Date-only form of fmtRunTime, for columns that need to answer "how old is +// this" rather than "which run exactly". Non-date-shaped ids pass through +// (reusing fmtRunTime's guard rather than a bare split, which would hand back +// "codex" for an ad-hoc id). +export function fmtRunDate(id: string): string { + if (!DAILY_RUN_ID_RE.test(id)) return id; + return id.split("_")[0]; +} + // Format a run.json ISO timestamp (`start_time`, "YYYY-MM-DDTHH:MM:SS[.ffffff]") // into the same "YYYY-MM-DD · HH:MM:SS" shape fmtRunTime renders for date-shaped // run ids — so an ad-hoc run, whose id carries no date, shows a comparable diff --git a/evalboard/lib/overview.ts b/evalboard/lib/overview.ts index e02215d6..5b18cd06 100644 --- a/evalboard/lib/overview.ts +++ b/evalboard/lib/overview.ts @@ -16,6 +16,8 @@ import { withinTurnBudget } from "./turns"; import { humanizeTaskId } from "./format"; import { mapWithConcurrency } from "./concurrency"; import { DEFAULT_HARNESS, normalizeHarness, orderHarnesses } from "./harness"; +import { isPassStatus } from "./status"; +import { taskCarriesRepoTag } from "./tags"; import type { Window } from "./reviews-types"; export interface RunPoint { @@ -515,13 +517,18 @@ function loadWindowData(window: Window): Promise { return loadWindowDataInner(window); } +// Repo-provenance half of taskMatchesTag. Defined in the dependency-free +// lib/tags.ts (this module is server-only — it imports next/cache — so a +// "use client" component could not adopt a copy living here) and re-exported +// for the existing callers. +export { taskCarriesRepoTag }; + export function taskMatchesTag( task: RunOverviewTask, reviewTagsByTask: Record, tag: string, ): boolean { - if (task.skill === tag) return true; - if (task.tags.includes(tag)) return true; + if (taskCarriesRepoTag(task, tag)) return true; const rt = reviewTagsByTask[task.taskId]; return rt ? rt.includes(tag) : false; } @@ -624,76 +631,201 @@ export async function getOverview( export interface TagTaskRow { taskId: string; skill: string | null; - // How many runs in the window carried this task under the tag. + // Tagged task ROWS in the window, not distinct runs: a replicated task + // contributes one per replicate. Unchanged from the previous behaviour + // (the column header stays "Appearances") — the de-tag check below is the + // only place that collapses to one sample per run. Includes rows the + // nightly skipped as mature and carried forward. appearances: number; - passRate: number; // 0-100 across those appearances + // Of `appearances`, how many were mature carry-forwards (not executed). + matureSkips: number; + // `appearances - matureSkips`: the denominator behind passRate, carried on + // the row rather than re-derived by the renderer so the percentage and the + // caption that names its sample size can never describe different rules. + executed: number; + // 0-100 over EXECUTED appearances only (appearances - matureSkips). + // null when nothing in the window actually ran, so the UI shows "—" + // rather than a measured-looking 0% or 100%. + passRate: number | null; latestStatus: string | null; latestScore: number | null; latestRunId: string; + // True when the newest tagged ROW — the same row latestStatus and + // latestScore come from — was a mature carry-forward, so those two are + // inherited, not measured. + latestMatureSkipped: boolean; } -// Per-task breakdown for a single tag, windowed like getOverview but grouped -// by task instead of by run. One row per distinct task_id carrying the tag -// anywhere in the window; "latest" fields come from the newest run the task -// appeared in (runs are walked newest-first, so the first occurrence wins). -export async function getTagTaskBreakdown( - window: Window, - tag: string, - harness: string | null = null, -): Promise { - const perRun = (await loadWindowData(window)).filter( - (r) => - !r.adhoc && - (harness == null || - normalizeHarness(r.overview?.harness) === harness), - ); +// Per-task breakdown for a single tag, windowed like getOverview but grouped by +// task instead of by run. Pure over the PerRun[] it is handed (the caller does +// the fetching and the adhoc/harness filtering), so it unit-tests without +// touching the blob store — mirrors lib/trends.ts::aggregate. +// +// DE-TAGGING. A run.json task row carries `tags` as a historical stamp written +// at execution time; the board never consults the skills repo. So a task +// de-tagged upstream keeps rendering for as long as a run that predates the +// removal stays in the window. The rule here: a task is dropped when it appears +// in a NEWER run in the window whose rows for it do not carry the tag — that is +// proof the tag was removed, not a heuristic. A task that merely stopped +// appearing (retired, renamed, `skip: true`) is unknowable and therefore KEPT, +// with `latestRunId` doubling as "last seen" so its age is visible. +// +// The signal comes from taskCarriesRepoTag, NOT taskMatchesTag: review tags +// (review_index.json) are post-hoc annotations from a separate namespace, so an +// as-yet-unreviewed newest run — the normal case — would otherwise read as a +// de-tag. The same predicate is used to accumulate, so the narrowing is +// symmetric: a task pulled onto this page only by a review tag does not appear. +// +// The "did any row carry the tag this run" collapse is required because a +// replicated task has several rows per run, and one untagged replicate must not +// read as a de-tag. +// +// MATURITY — a DELIBERATE, page-local divergence. lib/trends.ts::aggregate (and +// app/runs/[id]/run-view.tsx) count a mature carry-forward as a pass and exclude +// it only from the cost/duration averages. Here it is excluded from BOTH the +// numerator and the denominator of `passRate`, because /path-to-ga is a +// GA-readiness page and must report MEASURED passes. That difference is +// intentional — do not "harmonise" this with trends.ts. +// +// CROSS-REPO CONTRACT: `matureSkipped` is stamped into run.json by the external +// nightly eval_runner, not by anything in src/coder_eval. If the producer renames +// or drops the field every carry-forward silently reads as an executed pass again +// — the rate inflates, the "(N mature)" annotations and Mature pills vanish, and +// nothing errors. It is the one input here this repo cannot type-check. +export function buildTagTaskRows(perRun: PerRun[], tag: string): TagTaskRow[] { + // Run ids are date-shaped, so a lexical sort is chronological — the same + // assumption lib/trends.ts::aggregate and the previous implementation + // already make. const sorted = [...perRun].sort((a, b) => b.id.localeCompare(a.id)); interface Acc { skill: string | null; - statuses: (string | null)[]; + appearances: number; + matureSkips: number; + executedPasses: number; latestRunId: string; latestStatus: string | null; latestScore: number | null; + latestMatureSkipped: boolean; } const byTask = new Map(); - - for (const { id, overview, reviewTagsByTask } of sorted) { + // taskId -> did its NEWEST appearance in the window carry the tag. First + // write wins because the walk is newest-first, so a task that only gained + // the tag recently reads as tagged (and one that lost it reads as untagged) + // regardless of what the older runs say. + const newestTagged = new Map(); + + for (const { id, overview } of sorted) { + // A run whose run.json failed to load (loadPerRunForId downgrades to a + // null overview) must contribute neither an appearance nor a de-tag + // signal — otherwise a transient blob failure on the newest run would + // drop every row. if (!overview) continue; + + // Two passes over the run's rows, not one: `taggedInRun` must be + // complete before any verdict is recorded, because a replicated task + // has several rows per run and one untagged replicate must not read as + // a de-tag. The `newestTagged.has` guard below is what collapses those + // replicate rows to a single first-write-wins verdict. + const taggedInRun = new Set(); + for (const t of overview.tasks) { + if (taskCarriesRepoTag(t, tag)) taggedInRun.add(t.taskId); + } for (const t of overview.tasks) { - if (!taskMatchesTag(t, reviewTagsByTask, tag)) continue; + if (!newestTagged.has(t.taskId)) { + newestTagged.set(t.taskId, taggedInRun.has(t.taskId)); + } + } + + for (const t of overview.tasks) { + if (!taskCarriesRepoTag(t, tag)) continue; let entry = byTask.get(t.taskId); if (!entry) { + // All three latest* fields come off ONE row — the first tagged + // row of the newest-first walk — so the Mature pill and the + // dashed-out score always describe the same sample. entry = { skill: t.skill, - statuses: [], + appearances: 0, + matureSkips: 0, + executedPasses: 0, latestRunId: id, latestStatus: t.status, latestScore: t.weightedScore, + latestMatureSkipped: t.matureSkipped ?? false, }; byTask.set(t.taskId, entry); } - entry.statuses.push(t.status); + entry.appearances += 1; + if (t.matureSkipped) { + entry.matureSkips += 1; + } else if (isPassStatus(t.status)) { + // lib/status.ts, not a raw "SUCCESS" literal: `status` is an + // untyped string, and this page's pass rate must move with + // every other surface if the passing set ever widens. + entry.executedPasses += 1; + } } } const rows: TagTaskRow[] = []; for (const [taskId, e] of byTask) { - const appearances = e.statuses.length; - const passed = e.statuses.filter((s) => s === "SUCCESS").length; + // Provably de-tagged: the task is still running, and its newest run does + // not carry the tag. (A task in byTask always has a newestTagged entry — + // both are written from the same non-null-overview iteration — so the + // `?? true` only satisfies Map.get's `| undefined`; it is not a real + // "unknown ⇒ keep" case.) + if (!(newestTagged.get(taskId) ?? true)) continue; + const executed = e.appearances - e.matureSkips; rows.push({ taskId, skill: e.skill, - appearances, - passRate: appearances ? (passed / appearances) * 100 : 0, + appearances: e.appearances, + matureSkips: e.matureSkips, + executed, + passRate: executed > 0 ? (e.executedPasses / executed) * 100 : null, latestStatus: e.latestStatus, latestScore: e.latestScore, latestRunId: e.latestRunId, + latestMatureSkipped: e.latestMatureSkipped, }); } return rows.sort((a, b) => a.taskId.localeCompare(b.taskId)); } +// IO wrapper around buildTagTaskRows: fetch the window, drop ad-hoc runs and +// (optionally) scope to one harness, then aggregate. Harness scoping happens +// HERE, before the pure function sees the runs, so a newer run on a different +// harness cannot de-tag a row in a harness-scoped view. +export async function getTagTaskBreakdown( + window: Window, + tag: string, + harness: string | null = null, +): Promise { + const perRun = (await loadWindowData(window)).filter( + (r) => + !r.adhoc && + (harness == null || + normalizeHarness(r.overview?.harness) === harness), + ); + return buildTagTaskRows(perRun, tag); +} + +// Mean of the per-run success rates, over the runs that HAVE one. A run whose +// successRate is null has no measurable outcome (no tasks, or a run.json that +// failed to load) — folding it in as 0 would drag the headline tile down and +// make "no data" indistinguishable from "everything failed". null when no run +// in scope reports a rate at all. +export function avgRunSuccessRate( + runs: readonly { successRate: number | null }[], +): number | null { + const rates = runs + .map((r) => r.successRate) + .filter((r): r is number => r != null); + if (rates.length === 0) return null; + return rates.reduce((sum, r) => sum + r, 0) / rates.length; +} + // The slice of a run that the active tag/q filter selects: which tasks count, // and the cost/duration summed over exactly those. null means the run has // nothing matching and drops out entirely. diff --git a/evalboard/lib/pills.tsx b/evalboard/lib/pills.tsx index 82fc266d..1ab93e04 100644 --- a/evalboard/lib/pills.tsx +++ b/evalboard/lib/pills.tsx @@ -17,6 +17,20 @@ export function matureLinkTooltip(sourceRunLabel: string): string { ); } +// Aggregate-voice variant, for a windowed count of mature appearances rather +// than one run's row. MATURE_TOOLTIP says "this run" twice, which is right on a +// single-run surface and wrong on a page that renders no single run at all. +export function matureAggregateTooltip( + matureSkips: number, + appearances: number, +): string { + return ( + `${matureSkips} of these ${appearances} appearances were mature ` + + "carry-forwards: skipped to save cost and carried forward as a pass " + + "(re-validated about weekly on a fixed slot), not executed." + ); +} + // Fallback tooltip when no recent execution was found within the look-back // window, so the id stays non-clickable. export const MATURE_NO_SOURCE_TOOLTIP = diff --git a/evalboard/lib/pricing.ts b/evalboard/lib/pricing.ts index ad6d0293..a96f165a 100644 --- a/evalboard/lib/pricing.ts +++ b/evalboard/lib/pricing.ts @@ -19,28 +19,55 @@ export interface Pricing { // build on drift — this hand-copied mirror is otherwise guarded only by a // comment. Not part of the consumer API; use resolvePricing() instead. export const PRICING: Record = { - "claude-opus-4-8": p(15, 75, 18.75, 1.5), - "claude-opus-4-7": p(15, 75, 18.75, 1.5), - "claude-opus-4-6": p(15, 75, 18.75, 1.5), - "claude-opus-4-6-20250514": p(15, 75, 18.75, 1.5), - "claude-opus-4-5-20251101": p(15, 75, 18.75, 1.5), + // Claude. Opus 4.5 and later are priced at the POST-repricing $5/$25 rates, + // not Opus 4.1's $15/$75 — the two generations differ 3x, so an undated + // alias must never inherit the older tier. Undated aliases each need their + // own key: resolvePricing's fallback only strips a trailing date (dated → + // undated), it cannot invent one. + "claude-fable-5": p(10, 50, 12.5, 1), + "claude-opus-5": p(5, 25, 6.25, 0.5), + "claude-opus-4-8": p(5, 25, 6.25, 0.5), + "claude-opus-4-7": p(5, 25, 6.25, 0.5), + "claude-opus-4-6": p(5, 25, 6.25, 0.5), + "claude-opus-4-5": p(5, 25, 6.25, 0.5), + "claude-opus-4-5-20251101": p(5, 25, 6.25, 0.5), + "claude-opus-4-1": p(15, 75, 18.75, 1.5), + "claude-opus-4": p(15, 75, 18.75, 1.5), "claude-opus-4-20250514": p(15, 75, 18.75, 1.5), + "claude-sonnet-5": p(3, 15, 3.75, 0.3), "claude-sonnet-4-6": p(3, 15, 3.75, 0.3), - "claude-sonnet-4-6-20250514": p(3, 15, 3.75, 0.3), + "claude-sonnet-4-5": p(3, 15, 3.75, 0.3), "claude-sonnet-4-5-20250929": p(3, 15, 3.75, 0.3), "claude-sonnet-4-20250514": p(3, 15, 3.75, 0.3), - "claude-haiku-4-5-20251001": p(0.8, 4, 1, 0.08), + "claude-haiku-4-5": p(1, 5, 1.25, 0.1), + "claude-haiku-4-5-20251001": p(1, 5, 1.25, 0.1), + "claude-haiku-3-5": p(0.8, 4, 1, 0.08), "claude-3-7-sonnet-20250219": p(3, 15, 3.75, 0.3), "claude-3-5-sonnet-20241022": p(3, 15, 3.75, 0.3), "claude-3-5-sonnet-20240620": p(3, 15, 3.75, 0.3), "claude-3-opus-20240229": p(15, 75, 18.75, 1.5), "claude-3-sonnet-20240229": p(3, 15, 3.75, 0.3), "claude-3-haiku-20240307": p(0.25, 1.25, 0.3, 0.03), + // OpenAI (CodexAgent). cacheWrite == input on every entry below is + // DELIBERATE, not a copy-paste slip: OpenAI bills no separate cache-write + // fee, so the fresh prompt slice is plain input. It is also inert — the + // Codex agent records cache_creation_tokens as 0 (codex_agent.py), so this + // rate always multiplies zero. Rationale mirrored from pricing.py, which + // states it once for the whole block. "gpt-5-codex": p(1.25, 10, 1.25, 0.125), "gpt-5": p(1.25, 10, 1.25, 0.125), + "gpt-5.1-codex-max": p(1.25, 10, 1.25, 0.125), + "gpt-5.1-codex": p(1.25, 10, 1.25, 0.125), + "gpt-5.1-codex-mini": p(0.25, 2, 0.25, 0.025), + "codex-mini-latest": p(1.5, 6, 1.5, 0.375), "gpt-5.3-codex": p(1.75, 14, 1.75, 0.175), + "gpt-5.2-codex": p(1.75, 14, 1.75, 0.175), "gpt-5.4": p(2.5, 15, 2.5, 0.25), "gpt-5.5": p(5, 30, 5, 0.5), + // Terra and Luna repriced 2026-07-30 (-20% / -80%); post-cut rates. + "gpt-5.6-sol": p(5, 30, 5, 0.5), + "gpt-5.6-terra": p(2, 12, 2, 0.2), + "gpt-5.6-luna": p(0.2, 1.2, 0.2, 0.02), // Google Gemini (AntigravityAgent). Gemini bills no separate cache-write // fee (cache_write == input, effectively unused); cache_read is the cached- // input rate. Pro's >200K-token tier is higher — this flat rate reads low @@ -48,8 +75,12 @@ export const PRICING: Record = { "gemini-3-pro-preview": p(2, 12, 2, 0.2), "gemini-3.1-pro-preview": p(2, 12, 2, 0.2), "gemini-3.1-pro-preview-customtools": p(2, 12, 2, 0.2), + "gemini-3.6-flash": p(1.5, 7.5, 1.5, 0.15), "gemini-3.5-flash": p(1.5, 9, 1.5, 0.15), - "gemini-3-flash-preview": p(1.5, 9, 1.5, 0.15), + "gemini-3.5-flash-lite": p(0.3, 2.5, 0.3, 0.03), + "gemini-3.1-flash-lite": p(0.25, 1.5, 0.25, 0.025), + "gemini-3.1-flash-lite-preview": p(0.25, 1.5, 0.25, 0.025), + "gemini-3-flash-preview": p(0.5, 3, 0.5, 0.05), // OpenRouter open-weight models (litellm backend) are DELIBERATELY NOT priced // here. OpenRouter routes per-request, so a static headline rate is wrong (the // billed rate depends on the provider it landed on), and there is no per-bucket diff --git a/evalboard/lib/tags.ts b/evalboard/lib/tags.ts new file mode 100644 index 00000000..610c3214 --- /dev/null +++ b/evalboard/lib/tags.ts @@ -0,0 +1,24 @@ +// The repo-provenance tag predicate, and nothing else. +// +// Deliberately dependency-free (no next/*, no blob readers) so a "use client" +// component can import it. That is the whole reason it is not in lib/overview.ts: +// that module imports next/cache and the blob readers, so anything defined there +// is server-only, which is why app/runs/[id]/run-view.tsx and lib/trends.ts each +// grew their own inline copy of these two lines. +// +// The parameter is structurally typed rather than RunOverviewTask so all three +// shapes that carry repo provenance — RunOverviewTask, TaskResultSummary and the +// aggregated TaskTrend — satisfy it without importing each other. + +export interface RepoTagged { + skill: string | null; + tags: string[]; +} + +// The tag as the task's own YAML declared it, stamped into run.json at execution +// time. This is the repo-provenance HALF of taskMatchesTag: the only half whose +// absence in a newer run proves the tag was removed upstream, since review tags +// are post-hoc annotations and an unreviewed run carries none. +export function taskCarriesRepoTag(task: RepoTagged, tag: string): boolean { + return task.skill === tag || task.tags.includes(tag); +} diff --git a/evalboard/lib/trends.ts b/evalboard/lib/trends.ts index f1b25b72..c1b669a4 100644 --- a/evalboard/lib/trends.ts +++ b/evalboard/lib/trends.ts @@ -11,6 +11,7 @@ import { type TagCount, } from "./overview"; import { DEFAULT_HARNESS } from "./harness"; +import { taskCarriesRepoTag } from "./tags"; import type { ComponentSha } from "./runs"; export const TRENDS_RECENT_RUN_COUNT = 10; @@ -254,9 +255,16 @@ export function aggregateTaskTrends( // Predicate matching getOverview's tag scoping logic, but operating on the // aggregated TaskTrend (we don't keep per-run reviewTagsByTask around after // aggregation; dominantFailureTags is the equivalent task-level signal). +// +// NOTE — divergence from /path-to-ga, which answers the same question under +// stricter rules. `trend.tags` is a UNION over the window, so a task de-tagged +// upstream keeps matching here until every run that predates the removal ages +// out; lib/overview.ts::buildTagTaskRows instead drops a task proven de-tagged +// by a newer run. Likewise `passRate` below counts mature carry-forwards as +// passes, which that page excludes. Both are deliberate: this is the general +// browsing surface, that one is a GA-readiness report. export function trendMatchesTag(trend: TaskTrend, tag: string): boolean { - if (trend.skill === tag) return true; - if (trend.tags.includes(tag)) return true; + if (taskCarriesRepoTag(trend, tag)) return true; return trend.dominantFailureTags.some((t) => t.tag === tag); } diff --git a/src/coder_eval/pricing.py b/src/coder_eval/pricing.py index 27132444..c8cf478c 100644 --- a/src/coder_eval/pricing.py +++ b/src/coder_eval/pricing.py @@ -3,7 +3,8 @@ Anthropic/OpenAI/Google built-in rates; plugins contribute additional rates via ``register_pricing()``. Prices are per million tokens (MTok). Sources: https://claude.com/pricing#api, https://developers.openai.com/api/docs/pricing, -https://ai.google.dev/gemini-api/docs/pricing (all verified 2026-07-29). +https://ai.google.dev/gemini-api/docs/pricing (all verified 2026-07-29; +the GPT-5.6 rows re-verified 2026-08-09 after the 2026-07-30 Terra/Luna cut). """ from collections.abc import Iterable @@ -20,7 +21,7 @@ class ModelPricing: cache_read_per_mtok: float # prompt caching read -# Official vendor rate cards, verified 2026-07-29. +# Official vendor rate cards, verified 2026-07-29 (GPT-5.6 rows: 2026-08-09). # Key: CLI model name (before gateway mapping) _PRICING: dict[str, ModelPricing] = { "claude-fable-5": ModelPricing(10.0, 50.0, 12.50, 1.0), @@ -79,9 +80,14 @@ class ModelPricing: "gpt-5.4-mini": ModelPricing(0.75, 4.5, 0.75, 0.075), "gpt-5.4-nano": ModelPricing(0.20, 1.25, 0.20, 0.02), # GPT-5.6: sol flagship / terra balanced (Codex default) / luna economy. + # Terra and Luna were REPRICED on 2026-07-30 (-20% and -80%); these are the + # post-cut rates. The pre-cut $2.50/$15 and $1.00/$6 are what a historical run + # was actually billed, but this table is a single current-rate card with no + # notion of an effective date — so old runs re-price low, the same tradeoff + # the Sonnet promo comment above already accepts. "gpt-5.6-sol": ModelPricing(5.0, 30.0, 5.0, 0.50), - "gpt-5.6-terra": ModelPricing(2.5, 15.0, 2.5, 0.25), - "gpt-5.6-luna": ModelPricing(1.0, 6.0, 1.0, 0.10), + "gpt-5.6-terra": ModelPricing(2.0, 12.0, 2.0, 0.20), + "gpt-5.6-luna": ModelPricing(0.20, 1.20, 0.20, 0.02), # Google Gemini (AntigravityAgent, via the Gemini Developer API), keyed on the # literal ids the ListModels endpoint returns. No cache-write fee, so # cache_write == input (unused: the agent maps cache_creation_tokens to 0).