docs: make the 1 ms benchmark iteration budget an explicit maximum - #9176
Conversation
The guide asked for "less than 1ms" without saying whether that was a target or a limit, and without saying which number to measure. Benchmarks have landed at 23.8 ms and 123.4 ms per iteration, and the review conversation on those has no documented rule to point at. State that 1 ms is the maximum, say what to do when a benchmark does not fit, and point at CodSpeed's per-iteration number under HEAD as the value to check -- not the runtime of the whole benchmark binary, which is what "too long" usually gets confused with. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FZJAp4wVgwTVhLyErCWFZn
3192635 to
4fdf68a
Compare
Merging this PR will regress 1 benchmark
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing Footnotes
|
Rationale for this change
The benchmarking guide asked for "less than 1ms" per iteration without saying whether that was an aspiration or a limit, and without saying which number to measure against it.
That gap showed up in review on #9136, where benchmarks landed at 23.8 ms and 123.4 ms per iteration. The review conversation there had no documented rule to point at — as noted on that PR, the guide "has nothing about benchmarks being too long".
This documents the rule first, on its own, so it can be cited independently of any tooling that enforces it.
What changes are included in this PR?
Docs only — one section of
docs/developer-guide/benchmarking.md:#[cfg(not(codspeed))].HEADfor every benchmark a PR adds or changes — not the runtime of the whole benchmark binary, which is what "this benchmark takes too long" usually gets confused with.The heading loses its
~(under ~1 ms→under 1 ms), which leaves the existing anchor#keep-per-iteration-execution-time-under-1-msunchanged.A follow-up branch adds CI that reads CodSpeed's report and comments when a new or changed benchmark exceeds this budget. It is deliberately kept out of this PR so the rule can be agreed on its own.
What APIs are changed? Are there any user-facing changes?
No code, no API changes. Documentation only.
🤖 Generated with Claude Code
https://claude.ai/code/session_01FZJAp4wVgwTVhLyErCWFZn
Generated by Claude Code