Skip to content

Drop APM traces when APM tracing is disabled and keep LLM Observability working - #12473

Draft
ncybul wants to merge 1 commit into
masterfrom
nicole.cybul/llmobs-apm-tracing-disabled
Draft

Drop APM traces when APM tracing is disabled and keep LLM Observability working#12473
ncybul wants to merge 1 commit into
masterfrom
nicole.cybul/llmobs-apm-tracing-disabled

Conversation

@ncybul

@ncybul ncybul commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

What Does This Do

Two fixes for running LLM Observability without APM tracing (solves #10051):

  1. DD_APM_TRACING_ENABLED=false now drops APM traces. The standalone sampler was only selected when ASM/IAST/SCA was enabled; every other configuration — including LLM Observability — fell through to the default sampler, which keeps traces, so APM traces kept flowing.
  2. DD_TRACE_ENABLED=false with DD_LLMOBS_ENABLED=true no longer throws. LLMObsSystem.start() installed the real SDK implementation even with no tracer installed, and AgentTracer.get().buildSpan() returns null on the no-op tracer, so every LLMObs.start*Span() call threw a NullPointerException out of application code.

Motivation

DD_APM_TRACING_ENABLED=false is documented as working in all SDKs. LLM Observability users who don't want APM tracing had no working way to turn it off: the APM flag was a no-op, and DD_TRACE_ENABLED=false broke the LLMObs SDK.

Additional Notes

Approach. This is a reduced port of #10989, which had gone stale. That PR introduced a ProductTraceSource.LLMOBS bit plus a StandaloneProduct enum and a unified StandaloneSampler replacing AsmStandaloneSampler. Two things changed since: 0x20 is now taken by AI_GUARD, and no other tracer allocates an LLMObs _dd.p.ts bit — in dd-trace-js the products are exactly APM/ASM/DSM/DJM/DBM/AI_GUARD, and in dd-trace-py exactly the same set. LLM Observability doesn't need one: its spans ship to the LLM Observability intake (/evp_proxy/v2/api/v2/llmobs), which is independent of APM sampling priority, so the correct behavior is to drop the APM traces and let the LLMObs events through.

That leaves the fix as the selection logic that other tracers already have — compare getProductRateLimiter in dd-trace-js (1/minute for appsec/iast, RateLimiter(0), i.e. drop-all, otherwise) and LLMObsProcessor.process_trace in dd-trace-py (returns None for the whole APM trace when APM tracing is disabled):

  • ASM enabled → unchanged: AsmStandaloneSampler, 1 APM trace per minute for service catalog / billing.
  • otherwise → ForcePrioritySampler(SAMPLER_DROP, DEFAULT).

Because a DEFAULT-mechanism priority cannot overwrite a priority that is already set, products that force-keep their own traces are unaffected — AI Guard (ai_guard.keep) and ASM (asm.keep) keep working exactly as before, which AIGuardSamplingTest covers.

Testing.

  • SamplerTest: APM traces are dropped when APM tracing is disabled, with and without LLM Observability enabled.
  • LLMObsSystemTest: the SDK stays no-op when tracing is disabled.
  • LlmObsApmTracingDisabledSmokeTest: end-to-end — the APM trace is dropped (SAMPLER_DROP, _dd.apm.enabled:0) while the LLMObs span still reaches the LLM Observability intake.
  • LlmObsTraceDisabledSmokeTest: end-to-end — calling the LLMObs SDK with dd.trace.enabled=false no longer fails the request.
  • Existing AIGuardSamplingTest, ApmTracingDisabledChunkMarkerTest and the other apm-tracing-disabled smoke tests pass unchanged.

Contributor Checklist

Claude session: ba8ff7f0-b73e-41d6-a806-3603ab1c7329
Resume: claude --resume ba8ff7f0-b73e-41d6-a806-3603ab1c7329

🤖 Generated with Claude Code

…ty working

Selecting a standalone sampler only when ASM/IAST/SCA is enabled left every
other DD_APM_TRACING_ENABLED=false configuration on the default sampler, which
keeps traces, so APM traces kept flowing. Drop them instead, as dd-trace-js and
dd-trace-py do; LLM Observability spans are unaffected because they ship to the
LLM Observability intake, which is independent of APM sampling priority.

DD_TRACE_ENABLED=false installed the real LLM Observability SDK with no tracer
to back it, and the no-op tracer's buildSpan() returns null, so every
LLMObs.start*Span() call threw out of application code. Keep the SDK no-op.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@dd-octo-sts

dd-octo-sts Bot commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

❌ New Groovy Files Detected

Please avoid introducing new .groovy files to this repository.

  • dd-smoke-tests/apm-tracing-disabled/src/test/groovy/datadog/smoketest/apmtracingdisabled/LlmObsApmTracingDisabledSmokeTest.groovy
  • dd-smoke-tests/apm-tracing-disabled/src/test/groovy/datadog/smoketest/apmtracingdisabled/LlmObsTraceDisabledSmokeTest.groovy

Instead, rewrite the new file(s) in Java / JUnit. See the How to Test With JUnit Guide for more details.

If this PR needs an exception, add the tag: override groovy enforcement label to bypass this workflow.

@ncybul ncybul added type: bug fix Bug fix tag: ai generated Largely based on code generated by an AI or LLM comp: core Tracer core comp: mlobs ML Observability (LLMObs) labels Sep 11, 2026
@datadog-datadog-prod-us1

This comment has been minimized.

@dd-octo-sts

dd-octo-sts Bot commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.79 s 14.77 s [-0.6%; +0.9%] (no difference)
startup:insecure-bank:tracing:Agent 13.65 s 13.69 s [-1.2%; +0.5%] (no difference)
startup:petclinic:appsec:Agent 17.58 s 17.28 s [+0.8%; +2.7%] (maybe worse)
startup:petclinic:iast:Agent 16.87 s 17.57 s [-8.0%; +0.1%] (no difference)
startup:petclinic:profiling:Agent 17.29 s 17.35 s [-1.5%; +0.9%] (no difference)
startup:petclinic:sca:Agent 17.57 s 17.48 s [-0.5%; +1.4%] (no difference)
startup:petclinic:tracing:Agent 16.53 s 16.30 s [-3.1%; +6.0%] (no difference)

Commit: 6b02ea98 · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: core Tracer core comp: mlobs ML Observability (LLMObs) tag: ai generated Largely based on code generated by an AI or LLM type: bug fix Bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant