Add APM test agent trace decoding by PerfectSlayer · Pull Request #12089 · DataDog/dd-trace-java

PerfectSlayer · 2026-07-28T09:58:15Z

What Does This Do

This PR adds APM test agent trace decoding.

Motivation

The goal is to retrieve the full traces captured by the APM test agent in order to evaluate them during smoke tests.
Unlike our mock backend, using the test agent will avoid testing against ourselves and our decoders.

Additional Notes

Contributor Checklist

Format the title according to the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any other useful labels
Avoid using close, fix, or any linking keywords when referencing an issue
Use solves instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, migration, or deletion
Update public documentation with any new configuration flags or behaviors
Once approved, use merge queue to merge the PR

Jira ticket: APMLP-1247

datadog-datadog-prod-us1-2

The new JSON decoder leaves v0.4 meta_struct entries as base64 strings, although the shared DecodedSpan contract and existing v0.4 decoder expose their decoded JSON values. Any smoke test that inspects AppSec or other structured span metadata will receive the wrong type or fail when casting it, so this path needs a custom decode before shipping.

_{Open Bits AI session}

_{🤖 Datadog Autotest · Commit 0bfd4d7 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

datadog-datadog-prod-us1-2 · 2026-07-28T10:01:14Z

+  public long getDuration() {
+    return this.duration;
+  }
+


Decode base64 meta_struct payloads

Smoke tests inspecting AppSec or other structured span metadata will see the wrong type and can fail or silently miss the metadata they are intended to validate.

Assertion details

Input: A /test/traces JSON response containing v0.4 meta_struct, for example meta_struct:{"appsec":"<base64 of JSON {"triggers":3}>"}. The test-agent wire representation uses binary meta_struct values, and JSON encoding represents those byte arrays as base64 strings.

Expected: getMetaStruct() returns the decoded structured value, matching SpanV04 and the DecodedSpan contract, so callers can inspect nested maps and lists.

Actual: SpanJson declares metaStruct as Map<String,Object> and returns it directly. Moshi therefore exposes the base64 text as a String instead of decoding the bytes and parsing the JSON payload.

Was this helpful? React 👍 or 👎
_{🤖 Datadog Autotest · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

#12089 (comment)

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0bfd4d7e74

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-07-28T10:07:12Z

+  @Json(name = "meta_struct")
+  Map<String, Object> metaStruct;


Decode binary meta_struct entries before exposing them

When a trace contains meta_struct, each v0.4 entry is a MessagePack-encoded binary payload (see TraceMapperV0_4.MetaStructWriter), which /test/traces represents as a base64 JSON string. Deserializing directly into Map<String, Object> therefore exposes strings instead of the structured objects returned by the existing v0.4 decoder, so AppSec/LLM smoke tests that inspect or cast these values will fail or evaluate the wrong data. Decode each base64 value and unpack its MessagePack payload; the current synthetic test should also use the wire representation rather than an already-decoded object.

Useful? React with 👍 / 👎.

It does not apply here. From Claude:

Thanks — this is the right instinct in general (in the raw msgpack v0.4/v1 wire formats, meta_struct values are binary MessagePack blobs, which is exactly why SpanV04/SpanV1 decode them). But it doesn't apply to the JSON this decoder consumes: the dd-apm-test-agent decodes meta_struct server-side before serving /test/traces and /test/session/traces, so the JSON already carries nested objects, not base64.

From ddapm_test_agent/trace.py at the pinned v1.44.0:
meta_struct: NotRequired[Dict[str, Dict[str, Any]]] # decoded dicts, not strings

def _parse_meta_struct(value):
return {key: msgpack.unpackb(val_bytes) for key, val_bytes in value.items()}
_parse_meta_struct is applied via _flexible_decode_meta_struct(payload) inside decode_v04(...), and handle_session_traces / handle_test_traces simply web.json_response(traces) those already-decoded traces. Source: https://github.com/DataDog/dd-apm-test-agent/blob/v1.44.0/ddapm_test_agent/trace.py#L363-L367

So typing meta_struct as base64 Map<String,String> and re-decoding would actually break against the real agent (the value is a nested JSON object, not a string). Keeping Map<String,Object> and letting the value deserialize directly is correct here. Not making this change; verified against the agent source (and the existing metaStructEmptyWhenAbsentAndAMapWhenPresent fixture uses the real nested shape).

datadog-datadog-prod-us1-2 · 2026-07-28T10:16:52Z

🎯 Code Coverage (details)
• Patch Coverage: 1.01%
• Overall Coverage: 1.49% (-56.36%)

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 7a8b69b | Docs | Datadog PR Page | Give us feedback!}

dd-octo-sts · 2026-07-28T14:49:24Z

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.01 s	14.02 s	[-0.9%; +0.7%] (no difference)
startup:insecure-bank:tracing:Agent	12.88 s	13.02 s	[-1.7%; -0.5%] (maybe better)
startup:petclinic:appsec:Agent	16.80 s	16.81 s	[-1.1%; +0.9%] (no difference)
startup:petclinic:iast:Agent	16.30 s	16.86 s	[-7.7%; +1.0%] (no difference)
startup:petclinic:profiling:Agent	16.69 s	16.83 s	[-2.0%; +0.4%] (no difference)
startup:petclinic:sca:Agent	16.89 s	16.40 s	[-1.3%; +7.3%] (no difference)
startup:petclinic:tracing:Agent	15.55 s	16.18 s	[-8.0%; +0.1%] (no difference)

Commit: 7a8b69b4 · CI Pipeline · Benchmarking Platform UI

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

PerfectSlayer · 2026-07-29T12:40:44Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8ff8af25e4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

PerfectSlayer · 2026-07-29T13:04:25Z

@codex review

chatgpt-codex-connector · 2026-07-29T13:08:21Z

Codex Review: Didn't find any major issues. Bravo.

Reviewed commit: 3e09e7c480

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

datadog-datadog-prod-us1-2 · 2026-07-29T13:19:11Z

View session in Datadog

Bits Code status: ✅ Done

Comment @DataDog to request changes

AlexeyKuznetsov-DD

LGTM, left minor notes.

AlexeyKuznetsov-DD · 2026-07-29T14:55:07Z

+ * MessageJson decodes a JSON trace payload — a JSON array of traces, each a JSON array of spans in
+ * the v0.4 shape, as exposed by the dd-apm-test-agent — into the shared {@link DecodedMessage}
+ * model. Unlike the msgpack formats there is no message envelope, so the payload maps directly to
+ * the list of traces.


Q: Would be nice to have a bit more info how this class used with dd-apm-test-agent ?

Currently it isn't, that's why it might be confusing.
It's part of a bigger changes shown here: #12051 and will be used with the new trace / span assert rules and smoke tests backend abstraction over mocked backend and dd apm test agent backend.

PerfectSlayer · 2026-07-30T04:29:30Z

What just happened? Why did GH added so many reviewers for a single commit with 7 files?
Sorry all for the notification… 🙇

datadog-datadog-prod-us1-2

More details

The decoder changes preserve the existing DecodedSpan contract across the exercised malformed, unsigned-ID, numeric-precision, and nested meta-struct cases, and the session token is applied only to the shared agent client. No diff-specific production regression was confirmed; the focused suite could not run because this sandbox lacks Java 25 and cannot download the Gradle wrapper.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Datadog Autotest · Commit 98b712f · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

PerfectSlayer · 2026-07-30T07:02:22Z

Last minute fix: numbers over Long.MAX_VALUE are now decoded as BigInteger rather than falling back to Double - it should keep the original precision. That will still work nicely with the Matcher<? extend Number> for the trace assert rules.

PerfectSlayer · 2026-07-30T07:02:36Z

/merge

gh-worker-devflow-routing-ef8351 · 2026-07-30T07:02:40Z

View all feedbacks in Devflow UI.

2026-07-30 07:02:40 UTC ℹ️ Start processing command /merge

2026-07-30 07:02:45 UTC ℹ️ MergeQueue: waiting for PR to be ready

This pull request is not mergeable according to GitHub. Common reasons include pending required checks, missing approvals, or merge conflicts — but it could also be blocked by other repository rules or settings.
It will be added to the queue as soon as checks pass and/or get approvals. View in MergeQueue UI.
Note: if you pushed new commits since the last approval, you may need additional approval.
You can remove it from the waiting list with /remove command.

2026-07-30 07:49:08 UTC ℹ️ MergeQueue: merge request added to the queue

The expected merge time in master is approximately 1h (p90).

2026-07-30 08:54:36 UTC ℹ️ MergeQueue: This merge request was merged

PerfectSlayer requested a review from a team as a code owner July 28, 2026 09:58

PerfectSlayer added the type: feature Enhancements and improvements label Jul 28, 2026

PerfectSlayer requested a review from amarziali July 28, 2026 09:58

PerfectSlayer added the comp: testing Testing label Jul 28, 2026

PerfectSlayer mentioned this pull request Jul 28, 2026

Improving smoke test capabilities #12051

Draft

datadog-datadog-prod-us1-2 Bot reviewed Jul 28, 2026

View reviewed changes

chatgpt-codex-connector Bot reviewed Jul 28, 2026

View reviewed changes

PerfectSlayer force-pushed the bbujon/smoke-tests-test-agent-decoding branch from 0bfd4d7 to d14912e Compare July 28, 2026 14:17

PerfectSlayer force-pushed the bbujon/smoke-tests-test-agent-decoding branch from d14912e to fa90299 Compare July 29, 2026 09:59

PerfectSlayer added the tag: ai generated Largely based on code generated by an AI or LLM label Jul 29, 2026

PerfectSlayer force-pushed the bbujon/smoke-tests-test-agent-decoding branch from fa90299 to 8ff8af2 Compare July 29, 2026 12:40

chatgpt-codex-connector Bot reviewed Jul 29, 2026

View reviewed changes

PerfectSlayer force-pushed the bbujon/smoke-tests-test-agent-decoding branch from 8ff8af2 to 3e09e7c Compare July 29, 2026 13:02

dougqh approved these changes Jul 29, 2026

View reviewed changes

AlexeyKuznetsov-DD approved these changes Jul 29, 2026

View reviewed changes

PerfectSlayer added 3 commits July 30, 2026 06:15

feat(ci): Add test agent sidecar to smoke tests

29eb526

feat(core): Add support for test session token header

2edebf3

feat(testing): Avoid null collection

a6e2545

PerfectSlayer force-pushed the bbujon/smoke-tests-test-agent-decoding branch from 3e09e7c to 6cba0d6 Compare July 30, 2026 04:16

PerfectSlayer requested review from a team as code owners July 30, 2026 04:16

PerfectSlayer requested review from mhlidd and randomanderson and removed request for a team July 30, 2026 04:16

PerfectSlayer force-pushed the bbujon/smoke-tests-decoded-spans branch from d1a2872 to a6e2545 Compare July 30, 2026 04:17

PerfectSlayer removed request for a team, ValentinZakharov, amarziali, dd-oleksii, dudikeleti, leoromanovsky, manuel-alvarez-alvarez, mhlidd and randomanderson July 30, 2026 04:28

PerfectSlayer force-pushed the bbujon/smoke-tests-test-agent-decoding branch from 6cba0d6 to 98b712f Compare July 30, 2026 05:43

Base automatically changed from bbujon/smoke-tests-decoded-spans to master July 30, 2026 06:59

feat(test-agent): Add APM test agent trace decoding

7a8b69b

PerfectSlayer force-pushed the bbujon/smoke-tests-test-agent-decoding branch from 98b712f to 7a8b69b Compare July 30, 2026 07:00

datadog-datadog-prod-us1-2 Bot reviewed Jul 30, 2026

View reviewed changes

gh-worker-dd-mergequeue-cf854d Bot merged commit db219d3 into master Jul 30, 2026
780 of 782 checks passed

gh-worker-dd-mergequeue-cf854d Bot deleted the bbujon/smoke-tests-test-agent-decoding branch July 30, 2026 08:54

github-actions Bot added this to the 1.65.0 milestone Jul 30, 2026

Uh oh!

Conversation

PerfectSlayer commented Jul 28, 2026 • edited by atlassian Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

datadog-datadog-prod-us1-2 Bot left a comment

Choose a reason for hiding this comment

Uh oh!

datadog-datadog-prod-us1-2 Bot Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

PerfectSlayer Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

PerfectSlayer Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

datadog-datadog-prod-us1-2 Bot commented Jul 28, 2026 • edited by datadog-datadog-prod-us1 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dd-octo-sts Bot commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🟢 Java Benchmark SLOs — All performance SLOs passed

Uh oh!

PerfectSlayer commented Jul 29, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

PerfectSlayer commented Jul 29, 2026

Uh oh!

chatgpt-codex-connector Bot commented Jul 29, 2026

Uh oh!

datadog-datadog-prod-us1-2 Bot commented Jul 29, 2026 • edited by datadog-datadog-us1-prod Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AlexeyKuznetsov-DD left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

AlexeyKuznetsov-DD Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

PerfectSlayer Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

PerfectSlayer commented Jul 30, 2026

Uh oh!

datadog-datadog-prod-us1-2 Bot left a comment

Choose a reason for hiding this comment

Uh oh!

PerfectSlayer commented Jul 30, 2026

Uh oh!

PerfectSlayer commented Jul 30, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

PerfectSlayer commented Jul 28, 2026 •

edited by atlassian Bot

Loading

datadog-datadog-prod-us1-2 Bot commented Jul 28, 2026 •

edited by datadog-datadog-prod-us1 Bot

Loading

dd-octo-sts Bot commented Jul 28, 2026 •

edited

Loading

datadog-datadog-prod-us1-2 Bot commented Jul 29, 2026 •

edited by datadog-datadog-us1-prod Bot

Loading

gh-worker-devflow-routing-ef8351 Bot commented Jul 30, 2026 •

edited

Loading