Skip to content

fix: show uncertainty in benchmark comparisons - #2476

Merged
zeitlinger merged 1 commit into
mainfrom
fix/benchmark-report-confidence
Sep 15, 2026
Merged

zeitlinger merged 1 commit into
mainfrom
fix/benchmark-report-confidence

Conversation

@zeitlinger

Copy link
Copy Markdown
Member

Improve benchmark comparison reporting without changing benchmark execution or sample configuration.

  • Display JMH 99.9% confidence intervals beside PR and base means.
  • Distinguish overlapping intervals, below-threshold changes, and missing/invalid uncertainty.
  • Keep the verdict explicitly conservative and heuristic, not a significance test.
  • Add coverage for rendering, malformed data, thresholds, metadata mismatch, and the supplied overlapping example.

This is separate from #2468 and does not rerun benchmarks.

Signed-off-by: Gregor Zeitlinger <gregor.zeitlinger@grafana.com>
@zeitlinger
zeitlinger merged commit 398d087 into main Sep 15, 2026
18 checks passed
@zeitlinger
zeitlinger deleted the fix/benchmark-report-confidence branch September 15, 2026 16:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants