Skip to content

perf: update otel benchmark dependencies, add otel bound instrument scenarios - #2451

Open
jack-berg wants to merge 3 commits into
prometheus:mainfrom
jack-berg:update-otel-benchmarks
Open

perf: update otel benchmark dependencies, add otel bound instrument scenarios#2451
jack-berg wants to merge 3 commits into
prometheus:mainfrom
jack-berg:update-otel-benchmarks

Conversation

@jack-berg

Copy link
Copy Markdown
Collaborator

Updating benchmarks after substantial efforts to improve otel performance under contention. Changes:

  • Add scenarios for experimental bound instruments. I understand if you want to reject this until stable.
  • Disable exemplars for apples to apples comparison. My understanding is that the prom client_java cases don't have exemplars enabled, can change if I'm wrong.
  • Update to opentelemetry-java v1.66.0. If we wait until opentelemetry-java-instrumentation version is released which bundles the v1.66.0 version, the clutter in pom.xml goes away.

Benchmark results on my machine:

Benchmark                                          Mode  Cnt      Score      Error  Units
CounterBenchmark.codahaleIncNoLabels              thrpt   25  95236.680 ±  755.718  ops/s
CounterBenchmark.openTelemetryAdd                 thrpt   25  54598.238 ±  290.633  ops/s
CounterBenchmark.openTelemetryBoundAdd            thrpt   25  82770.849 ±  342.576  ops/s
CounterBenchmark.openTelemetryBoundInc            thrpt   25  85165.317 ±    7.693  ops/s
CounterBenchmark.openTelemetryInc                 thrpt   25  75136.160 ±  552.123  ops/s
CounterBenchmark.openTelemetryIncNoLabels         thrpt   25  85517.108 ±  538.105  ops/s
CounterBenchmark.prometheusAdd                    thrpt   25  84533.092 ±  680.646  ops/s
CounterBenchmark.prometheusInc                    thrpt   25  85135.753 ±   23.184  ops/s
CounterBenchmark.prometheusNoLabelsInc            thrpt   25  85061.556 ±  194.011  ops/s
CounterBenchmark.simpleclientAdd                  thrpt   25   9174.173 ± 1542.709  ops/s
CounterBenchmark.simpleclientInc                  thrpt   25   8860.946 ± 1306.154  ops/s
CounterBenchmark.simpleclientNoLabelsInc          thrpt   25  17502.624 ±  830.879  ops/s
HistogramBenchmark.openTelemetryBoundClassic      thrpt   25  25261.802 ± 6559.742  ops/s
HistogramBenchmark.openTelemetryBoundExponential  thrpt   25   2367.637 ±  381.165  ops/s
HistogramBenchmark.openTelemetryClassic           thrpt   25  28697.261 ± 4082.434  ops/s
HistogramBenchmark.openTelemetryExponential       thrpt   25   1868.494 ±  101.732  ops/s
HistogramBenchmark.prometheusClassic              thrpt   25  23600.746 ± 7292.571  ops/s
HistogramBenchmark.prometheusClassicPerThread     thrpt   25  58674.695 ±   98.620  ops/s
HistogramBenchmark.prometheusClassicSingleThread  thrpt   25  14865.898 ±   12.295  ops/s
HistogramBenchmark.prometheusNative               thrpt   25  10726.819 ±  677.244  ops/s
HistogramBenchmark.simpleclient                   thrpt   25  17340.988 ±  266.176  ops/s

The improvements to otel explicit histogram performance came from a design inspired by prometheus client_java. The code contains attribution. I did manage to slightly improve on the design - If you agree, I'm happy to chat about it and encourage prom client_java to copy any of the bits you think are helpful.

@jack-berg jack-berg changed the title Update otel benchmark dependencies, add otel bound instrument scenarios perf: update otel benchmark dependencies, add otel bound instrument scenarios Sep 11, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Benchmark results

Benchmark run succeeded for 6de043a8e903d7c24b89bafdc38c6b787db98b30.

Prometheus Java Client Benchmarks

Run Information

  • Date: 2026-09-14T15:36:54Z
  • Commit: 6de043a
  • JDK: 25.0.3 (OpenJDK 64-Bit Server VM)
  • Benchmark config: 3 fork(s), 3 warmup, 5 measurement, 4 threads
  • Hardware: AMD EPYC 9V74 80-Core Processor, 4 cores, 16 GB RAM
  • OS: Linux 6.17.0-1022-azure

Comparison with base

  • Head: 6de043a
  • Base: f71915f
  • Metric direction: Throughput scores are higher-is-better; positive Head vs base deltas indicate faster performance.
  • Note: Base and head are compared on the same runner within each topic.
Benchmark PR Base Head vs base Regression verdict
CounterBenchmark.codahaleIncNoLabels 45.61K 41.98K +8.6% meaningful improvement
CounterBenchmark.openTelemetryAdd 14.61K 12.13K +20.4% meaningful improvement
CounterBenchmark.openTelemetryInc 16.66K 14.13K +17.9% meaningful improvement
CounterBenchmark.openTelemetryIncNoLabels 20.87K 17.07K +22.2% meaningful improvement
CounterBenchmark.prometheusAdd 48.61K 49.29K -1.4% within noise
CounterBenchmark.prometheusInc 60.03K 58.06K +3.4% within noise
CounterBenchmark.prometheusNoLabelsInc 51.87K 51.87K -0.0% within noise
CounterBenchmark.simpleclientAdd 5.91K 5.82K +1.6% within noise
CounterBenchmark.simpleclientInc 6.12K 6.21K -1.4% within noise
CounterBenchmark.simpleclientNoLabelsInc 5.89K 5.92K -0.5% within noise
HistogramBenchmark.openTelemetryClassic 2.34K 552.56 +322.9% meaningful improvement
HistogramBenchmark.openTelemetryExponential 500.73 444.95 +12.5% meaningful improvement
HistogramBenchmark.prometheusClassic 3.98K 3.62K +10.0% within noise
HistogramBenchmark.prometheusClassicPerThread 9.37K 9.35K +0.2% within noise
HistogramBenchmark.prometheusClassicSingleThread 4.64K 4.65K -0.3% within noise
HistogramBenchmark.prometheusNative 2.31K 1.86K +24.1% meaningful improvement
HistogramBenchmark.simpleclient 6.03K 6.08K -0.8% within noise
HistogramTextFormatBenchmark.openMetricsWriteToNull 24.12K 22.96K +5.0% meaningful improvement
HistogramTextFormatBenchmark.prometheusWriteToNull 24.10K 24.42K -1.3% within noise
TextFormatUtilBenchmark.openMetricsWriteToByteArray 461.97K 466.04K -0.9% within noise
TextFormatUtilBenchmark.openMetricsWriteToNull 472.81K 472.49K +0.1% within noise
TextFormatUtilBenchmark.prometheusWriteToByteArray 492.58K 487.94K +0.9% within noise
TextFormatUtilBenchmark.prometheusWriteToNull 501.20K 493.19K +1.6% within noise
  • Benchmarks only in PR results (listed separately below): CounterBenchmark.openTelemetryBoundAdd, CounterBenchmark.openTelemetryBoundInc, HistogramBenchmark.openTelemetryBoundClassic, HistogramBenchmark.openTelemetryBoundExponential

Results for PR head

CounterBenchmark

Benchmark Score Error Units
prometheusInc 60.03K ± 67.66 ops/s
prometheusNoLabelsInc 51.87K ± 755.18 ops/s
prometheusAdd 48.61K ± 431.95 ops/s
codahaleIncNoLabels 45.61K ± 904.71 ops/s
openTelemetryIncNoLabels 20.87K ± 275.85 ops/s
openTelemetryInc 16.66K ± 40.06 ops/s
openTelemetryAdd 14.61K ± 115.41 ops/s
simpleclientInc 6.12K ± 54.10 ops/s
simpleclientAdd 5.91K ± 263.21 ops/s
simpleclientNoLabelsInc 5.89K ± 6.51 ops/s

HistogramBenchmark

Benchmark Score Error Units
prometheusClassicPerThread 9.37K ± 78.32 ops/s
simpleclient 6.03K ± 112.00 ops/s
prometheusClassicSingleThread 4.64K ± 63.20 ops/s
prometheusClassic 3.98K ± 2.15K ops/s
openTelemetryClassic 2.34K ± 957.53 ops/s
prometheusNative 2.31K ± 182.03 ops/s
openTelemetryExponential 500.73 ± 9.52 ops/s

HistogramTextFormatBenchmark

Benchmark Score Error Units
openMetricsWriteToNull 24.12K ± 466.20 ops/s
prometheusWriteToNull 24.10K ± 165.43 ops/s

TextFormatUtilBenchmark

Benchmark Score Error Units
prometheusWriteToNull 501.20K ± 9.16K ops/s
prometheusWriteToByteArray 492.58K ± 5.52K ops/s
openMetricsWriteToNull 472.81K ± 4.25K ops/s
openMetricsWriteToByteArray 461.97K ± 4.56K ops/s

New benchmarks in PR head

These benchmarks have no base counterpart; scores are descriptive only and have no regression verdict.

Benchmark Score Error Units
CounterBenchmark.openTelemetryBoundAdd 30.19K ± 606.96 ops/s
CounterBenchmark.openTelemetryBoundInc 34.64K ± 118.20 ops/s
HistogramBenchmark.openTelemetryBoundClassic 4.12K ± 167.25 ops/s
HistogramBenchmark.openTelemetryBoundExponential 553.17 ± 16.16 ops/s

Raw Results

Benchmark                                            Mode  Cnt          Score        Error  Units
CounterBenchmark.codahaleIncNoLabels                thrpt   15      45606.374    ± 904.713  ops/s
CounterBenchmark.openTelemetryAdd                   thrpt   15      14610.287    ± 115.408  ops/s
CounterBenchmark.openTelemetryBoundAdd              thrpt   15      30185.103    ± 606.964  ops/s
CounterBenchmark.openTelemetryBoundInc              thrpt   15      34637.698    ± 118.196  ops/s
CounterBenchmark.openTelemetryInc                   thrpt   15      16662.639     ± 40.058  ops/s
CounterBenchmark.openTelemetryIncNoLabels           thrpt   15      20867.544    ± 275.853  ops/s
CounterBenchmark.prometheusAdd                      thrpt   15      48611.919    ± 431.947  ops/s
CounterBenchmark.prometheusInc                      thrpt   15      60025.349     ± 67.663  ops/s
CounterBenchmark.prometheusNoLabelsInc              thrpt   15      51865.237    ± 755.185  ops/s
CounterBenchmark.simpleclientAdd                    thrpt   15       5910.856    ± 263.215  ops/s
CounterBenchmark.simpleclientInc                    thrpt   15       6120.718     ± 54.102  ops/s
CounterBenchmark.simpleclientNoLabelsInc            thrpt   15       5891.804      ± 6.506  ops/s
HistogramBenchmark.openTelemetryBoundClassic        thrpt   15       4119.817    ± 167.247  ops/s
HistogramBenchmark.openTelemetryBoundExponential    thrpt   15        553.168     ± 16.164  ops/s
HistogramBenchmark.openTelemetryClassic             thrpt   15       2336.606    ± 957.535  ops/s
HistogramBenchmark.openTelemetryExponential         thrpt   15        500.729      ± 9.519  ops/s
HistogramBenchmark.prometheusClassic                thrpt   15       3978.460   ± 2148.301  ops/s
HistogramBenchmark.prometheusClassicPerThread       thrpt   15       9371.457     ± 78.321  ops/s
HistogramBenchmark.prometheusClassicSingleThread    thrpt   15       4636.831     ± 63.196  ops/s
HistogramBenchmark.prometheusNative                 thrpt   15       2309.747    ± 182.028  ops/s
HistogramBenchmark.simpleclient                     thrpt   15       6030.195    ± 112.004  ops/s
HistogramTextFormatBenchmark.openMetricsWriteToNull  thrpt   15      24117.658    ± 466.196  ops/s
HistogramTextFormatBenchmark.prometheusWriteToNull  thrpt   15      24099.842    ± 165.428  ops/s
TextFormatUtilBenchmark.openMetricsWriteToByteArray  thrpt   15     461966.352   ± 4560.627  ops/s
TextFormatUtilBenchmark.openMetricsWriteToNull      thrpt   15     472812.630   ± 4249.055  ops/s
TextFormatUtilBenchmark.prometheusWriteToByteArray  thrpt   15     492577.786   ± 5515.902  ops/s
TextFormatUtilBenchmark.prometheusWriteToNull       thrpt   15     501197.837   ± 9161.278  ops/s

Notes

  • Score = the JMH primary metric; throughput is higher-is-better and latency is lower-is-better.
  • Error = 99.9% confidence interval
  • Regression verdict requires comparable benchmark metadata, non-overlapping JMH confidence intervals, and a change of at least 5%; otherwise it is marked "within noise" or "inconclusive".
  • Scores for different benchmark methods are not ranked against one another; they may measure different workloads.

Benchmark Descriptions

Benchmark Description
CounterBenchmark Counter increment performance: Prometheus, OpenTelemetry, simpleclient, Codahale
HistogramBenchmark Histogram observation performance (classic vs native/exponential)
TextFormatUtilBenchmark Metric exposition format writing speed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants