CI: comment when a benchmark exceeds the 1 ms iteration budget

[joseph-isaacs]

Rationale for this change

The 1 ms per-iteration budget documented in #9176 had nothing enforcing it, so benchmarks landed at 23.8 ms and 123.4 ms and were caught only by a maintainer reading CodSpeed's table by hand and commenting "This benchmark has a too long runtime" (#9136).

The obvious implementation — rebuild the benchmarks in walltime mode and re-run them to get wall-clock numbers — costs a second 8-shard build-and-run matrix. It isn't needed: CodSpeed already measures per-iteration time and publishes it in its sticky PR comment. So this reads that comment instead. Nothing is rebuilt, nothing is re-run.

What changes are included in this PR?

.github/workflows/bench-budget.yml — one job, on issue_comment (created and edited, since CodSpeed edits its comment in place as runs land). Gated on the comment author being codspeed-hq[bot] and the body carrying CodSpeed's marker: anyone can post text containing that marker, so pinning the author is what makes the parsed table trustworthy.

scripts/check-bench-budget.py — parses the "Performance Changes" table: status emoji, mode, the percent-encoded uri query parameter from each benchmark link, and the HEAD per-iteration time. Handles 1,182.5 µs (thousands separator), both micro-sign variants, N/A, and the | ... | truncation row.

scripts/tests/test_check_bench_budget.py — 27 tests. Fixtures are trimmed copies of the real CodSpeed comments from #9136 and #9060, so the parser is tested against markup CodSpeed actually posts.

ci.yml — a bench-budget-script job running those tests. The workflow itself lives outside PR CI, so a parser bug would otherwise surface as a wrong comment on someone else's PR rather than as a red check.

docs/developer-guide/benchmarking.md — documents the check under the rule added in #9176.

On #9136 this would have rendered, in place of the manual comment:

Benchmark	Per-iteration	Over budget
vortex-geo/…::contains::constant_x_polygons_overlapping	123.4 ms	123.4×	🆕
vortex-geo/…::contains::column_x_column_polygons	23.8 ms	23.8×	🆕
vortex-array/benches/like.rs::like_per_row_distinct_patterns	1.1 ms	1.1×	🆕

Deliberate choices worth reviewing

• Report-only, never a failing check. --fail-on-violation exists but is unused.
• Quiet when clean. Nothing over budget updates an existing complaint to an all-clear but never opens a new comment, so in-budget PRs get no noise.
• Improvements are not flagged. A benchmark this PR made faster but still over budget is not a reason to open a complaint (--include-improved overrides).
• Two scope limits, stated in the comment itself rather than hidden: only benchmarks CodSpeed reports as new or changed are checked (an untouched benchmark already over budget is never caught — the budget is enforced going forward, not retroactively), and CodSpeed truncates its own table at 20 rows, in which case the comment says so instead of implying the rest were checked.
• Checkout is the default branch, not the PR head. The job holds a pull-requests: write token, so it must run this repository's script rather than a version a fork could edit.

Testing

• 27 parser tests pass (also wired into CI as Benchmark budget script).
• End-to-end run of the script against the real comment body from benchmarks: add string compression benchmark #9060.
• ruff check / ruff format clean; yamllint --strict -c .yamllint.yaml .github/ clean; git diff --check clean.
• Verified pr-number and create-if-not-exists are real inputs on thollander/actions-comment-pull-request at the SHA this repo already pins.

Important

This cannot be exercised end-to-end on its own PR. issue_comment and workflow_dispatch workflows only ever run the copy of the workflow on the default branch, so neither entry point exists until this merges. CodSpeed will post its report here and nothing will react to it — that is expected, not a failure.

That is why the workflow_dispatch entry point exists. After merge, dispatch it with pr-number: 9136 to replay it against a real report and confirm the comment it produces. If it misbehaves, it comments on pull requests and nothing else — it fails no checks and blocks no merges, so the blast radius of a bad parse is a wrong comment.

What APIs are changed? Are there any user-facing changes?

No Rust, no public API. CI and tooling only. The user-facing effect is a new PR comment when a new or changed benchmark exceeds 1 ms per iteration.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FZJAp4wVgwTVhLyErCWFZn

[codspeed-hq]

Merging this PR will not alter performance

✅ 1842 untouched benchmarks
⏩ 44 skipped benchmarks¹

---

_{Comparing claude/branch-benchmark-ci-l7j32h (f4d6cf4) with develop (4ebe623)}

Footnotes

1. 44 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩