Skip to content

JMH convention plugin - #12048

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 1 commit into
masterfrom
bdu/test-jvm-constraints-apply-to-jmh
Jul 29, 2026
Merged

JMH convention plugin#12048
gh-worker-dd-mergequeue-cf854d[bot] merged 1 commit into
masterfrom
bdu/test-jvm-constraints-apply-to-jmh

Conversation

@bric3

@bric3 bric3 commented Jul 23, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

What Does This Do

Introduces a JMH convention plugin, that support passing properties via command line for jmh runs. In particular it supports the dd-trace-java.test-jvm-constraints testJvm property to select the JVM.

Other properties are also supported to adjust the JMH benchmark configuration

Property Effect
jmh.includes Comma-separated benchmark name patterns to run (a subset run)
jmh.profilers Comma-separated JMH profilers to attach (e.g. stack, gc)
jmh.forks Overrides the fork count for a spot-check run
jmh.threads Overrides the thread count for a spot-check run

Motivation

Provide a regular "standardized" way to adjust benchmark runs via command line.
Added as a convention plugin so it avoids repetition. Follow what's done in #11703, among others.

Additional Notes

Due to the way classloader works between buildsrc/ and regular projects, the plugin code has to use reflection.

Contributor Checklist

Jira ticket: [PROJ-IDENT]

@bric3 bric3 added tag: no release notes Changes to exclude from release notes comp: tooling Build & Tooling labels Jul 23, 2026
@bric3

bric3 commented Jul 23, 2026

Copy link
Copy Markdown
Contributor Author

@codex review

@bric3 bric3 added tag: no release notes Changes to exclude from release notes comp: tooling Build & Tooling labels Jul 23, 2026
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Jul 23, 2026

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Can't wait for the next one!

Reviewed commit: 9f4b41d4dc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@datadog-datadog-prod-us1-2

This comment has been minimized.

Sign in to view
@dd-octo-sts

dd-octo-sts Bot commented Jul 23, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.01 s 13.97 s [-0.4%; +1.1%] (no difference)
startup:insecure-bank:tracing:Agent 12.95 s 13.01 s [-1.2%; +0.3%] (no difference)
startup:petclinic:appsec:Agent 16.76 s 16.77 s [-1.2%; +1.1%] (no difference)
startup:petclinic:iast:Agent 16.33 s 16.85 s [-7.6%; +1.4%] (no difference)
startup:petclinic:profiling:Agent 16.65 s 16.70 s [-1.5%; +0.9%] (no difference)
startup:petclinic:sca:Agent 16.93 s 16.82 s [-0.3%; +1.6%] (no difference)
startup:petclinic:tracing:Agent 15.91 s 16.13 s [-2.4%; -0.3%] (maybe better)

Commit: 42af6e14 · CI Pipeline · Benchmarking Platform UI

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@bric3
bric3 requested a review from dougqh July 23, 2026 13:28
@bric3
bric3 marked this pull request as ready for review July 23, 2026 13:29
@bric3
bric3 requested a review from a team as a code owner July 23, 2026 13:29
datadog-datadog-prod-us1-2[bot]
datadog-datadog-prod-us1-2 Bot reviewed Jul 23, 2026
View reviewed changes

@datadog-datadog-prod-us1-2 datadog-datadog-prod-us1-2 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Datadog Autotest: PASS

More details

The JMH configuration is now applied centrally without changing the existing testJvm semantics: Gradle's withPlugin callback handles either plugin application order, and an unset testJvm leaves the JMH JVM unset. Focused execution was blocked because this checkout requires a Java 25 Gradle daemon while the sandbox has only JDK 8, 11, 17, and 21.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Datadog Autotest · Commit 9f4b41d · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest

@bric3
bric3 force-pushed the bdu/test-jvm-constraints-apply-to-jmh branch from 9f4b41d to e5d4e59 Compare July 29, 2026 10:43
@bric3
bric3 requested review from a team as code owners July 29, 2026 10:43
@bric3
bric3 requested review from claponcet, dromanol, jordan-wong and mcculls and removed request for a team July 29, 2026 10:43
@bric3 bric3 changed the title Test jvm constraints plugin now configures jmh plugin JMH convention plugin Jul 29, 2026
@bric3
feat: introduces a jmh convention plugin
42af6e1 
* Use `testJvm` as the default JMH launcher while keeping
* Use `jmh.*` properties when provided

  * Without `jmh.fork` or `jmh.threads`, forks and threads come from `@Fork` and `@Threads`; otherwise JMH defaults apply.
  * Without `jmh.includes`, JMH runs all discovered benchmarks.
  * Without `jmh.profilers`, no profiler is attached unless configured elsewhere.
@bric3
bric3 force-pushed the bdu/test-jvm-constraints-apply-to-jmh branch from e5d4e59 to 42af6e1 Compare July 29, 2026 10:48
@dougqh

dougqh commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

This looks good to me. It covers the things that I use frequently: selecting benchmarks, running with fewer forks, running with prof gc.

It could also be nice to have the option to adjust warmupIterations and measurementIterations, too, but that's just nice to have.

@bric3

bric3 commented Jul 29, 2026

Copy link
Copy Markdown
Contributor Author

/merge

@gh-worker-devflow-routing-ef8351

gh-worker-devflow-routing-ef8351 Bot commented Jul 29, 2026 •
edited
Loading

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2026-07-29 12:37:28 UTC ℹ️ Start processing command /merge

2026-07-29 12:37:33 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 1h (p90).

2026-07-29 13:28:14 UTC ℹ️ MergeQueue: This merge request was merged

@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot merged commit 8d7bf3a into master Jul 29, 2026
589 checks passed
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot deleted the bdu/test-jvm-constraints-apply-to-jmh branch July 29, 2026 13:28
@github-actions github-actions Bot added this to the 1.65.0 milestone Jul 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@datadog-datadog-prod-us1-2 datadog-datadog-prod-us1-2[bot] datadog-datadog-prod-us1-2[bot] left review comments
@dougqh dougqh dougqh approved these changes
@claponcet claponcet Awaiting requested review from claponcet claponcet is a code owner automatically assigned from DataDog/asm-java
@dromanol dromanol Awaiting requested review from dromanol dromanol is a code owner automatically assigned from DataDog/asm-java
@mcculls mcculls Awaiting requested review from mcculls mcculls is a code owner automatically assigned from DataDog/apm-sdk-capabilities-java
@jordan-wong jordan-wong Awaiting requested review from jordan-wong jordan-wong is a code owner automatically assigned from DataDog/apm-idm-java

Assignees

No one assigned

Labels

comp: tooling Build & Tooling tag: no release notes Changes to exclude from release notes

Projects

None yet

Milestone

1.65.0

Development

Successfully merging this pull request may close these issues.

2 participants

@bric3 @dougqh