Skip to content

docs: make the 1 ms benchmark iteration budget an explicit maximum - #9176

Merged
connortsui20 merged 1 commit into
developfrom
claude/bench-1ms-target-docs
Aug 4, 2026
Merged

docs: make the 1 ms benchmark iteration budget an explicit maximum#9176
connortsui20 merged 1 commit into
developfrom
claude/bench-1ms-target-docs

Conversation

@joseph-isaacs

Copy link
Copy Markdown
Contributor

Rationale for this change

The benchmarking guide asked for "less than 1ms" per iteration without saying whether that was an aspiration or a limit, and without saying which number to measure against it.

That gap showed up in review on #9136, where benchmarks landed at 23.8 ms and 123.4 ms per iteration. The review conversation there had no documented rule to point at — as noted on that PR, the guide "has nothing about benchmarks being too long".

This documents the rule first, on its own, so it can be cited independently of any tooling that enforces it.

What changes are included in this PR?

Docs only — one section of docs/developer-guide/benchmarking.md:

  • States that 1 ms is the maximum, not a soft target.
  • Says what to do when a benchmark does not fit: shrink the input, split it into smaller parameterized cases, or gate it with #[cfg(not(codspeed))].
  • Names the number to check: the per-iteration time CodSpeed reports under HEAD for every benchmark a PR adds or changes — not the runtime of the whole benchmark binary, which is what "this benchmark takes too long" usually gets confused with.

The heading loses its ~ (under ~1 msunder 1 ms), which leaves the existing anchor #keep-per-iteration-execution-time-under-1-ms unchanged.

A follow-up branch adds CI that reads CodSpeed's report and comments when a new or changed benchmark exceeds this budget. It is deliberately kept out of this PR so the rule can be agreed on its own.

What APIs are changed? Are there any user-facing changes?

No code, no API changes. Documentation only.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FZJAp4wVgwTVhLyErCWFZn


Generated by Claude Code

The guide asked for "less than 1ms" without saying whether that was a target or a limit,
and without saying which number to measure. Benchmarks have landed at 23.8 ms and 123.4 ms
per iteration, and the review conversation on those has no documented rule to point at.

State that 1 ms is the maximum, say what to do when a benchmark does not fit, and point at
CodSpeed's per-iteration number under HEAD as the value to check -- not the runtime of the
whole benchmark binary, which is what "too long" usually gets confused with.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FZJAp4wVgwTVhLyErCWFZn
@joseph-isaacs
joseph-isaacs force-pushed the claude/bench-1ms-target-docs branch from 3192635 to 4fdf68a Compare August 4, 2026 15:37
@codspeed-hq

codspeed-hq Bot commented Aug 4, 2026

Copy link
Copy Markdown

Merging this PR will regress 1 benchmark

⚡ 5 improved benchmarks
❌ 1 regressed benchmark
✅ 1836 untouched benchmarks
⏩ 44 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation chunked_varbinview_opt_into_canonical[(10, 1000)] 5.4 ms 6.2 ms -12.63%
Simulation chunked_canonicalize_into[(1000, 50, 8, 4)] 16.2 ms 14.5 ms +11.63%
Simulation chunked_canonicalize_into[(1000, 50, 8, 64)] 18.9 ms 17.1 ms +10.46%
Simulation chunked_canonicalize_into[(1000, 100, 16, 16)] 26.8 ms 24.3 ms +10.42%
Simulation chunked_canonicalize_into[(1000, 50, 8, 16)] 17.1 ms 15.5 ms +10.37%
Simulation chunked_canonicalize_into[(1000, 100, 16, 4)] 22.9 ms 20.8 ms +10.34%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing claude/bench-1ms-target-docs (4fdf68a) with develop (bc9aa10)2

Open in CodSpeed

Footnotes

  1. 44 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

  2. No successful run was found on develop (2bae710) during the generation of this report, so bc9aa10 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report.

@connortsui20
connortsui20 merged commit 4ebe623 into develop Aug 4, 2026
70 of 71 checks passed
@connortsui20
connortsui20 deleted the claude/bench-1ms-target-docs branch August 4, 2026 15:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/docs A docs change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants