Drop APM traces when APM tracing is disabled and keep LLM Observability working - #12473
Draft
ncybul wants to merge 1 commit into
Draft
Drop APM traces when APM tracing is disabled and keep LLM Observability working#12473ncybul wants to merge 1 commit into
ncybul wants to merge 1 commit into
Conversation
…ty working Selecting a standalone sampler only when ASM/IAST/SCA is enabled left every other DD_APM_TRACING_ENABLED=false configuration on the default sampler, which keeps traces, so APM traces kept flowing. Drop them instead, as dd-trace-js and dd-trace-py do; LLM Observability spans are unaffected because they ship to the LLM Observability intake, which is independent of APM sampling priority. DD_TRACE_ENABLED=false installed the real LLM Observability SDK with no tracer to back it, and the no-op tracer's buildSpan() returns null, so every LLMObs.start*Span() call threw out of application code. Keep the SDK no-op. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
|
❌ New Groovy Files Detected Please avoid introducing new
Instead, rewrite the new file(s) in Java / JUnit. See the How to Test With JUnit Guide for more details. If this PR needs an exception, add the |
This comment has been minimized.
This comment has been minimized.
Contributor
🟢 Java Benchmark SLOs — All performance SLOs passed
PR vs. master results
Commit: Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What Does This Do
Two fixes for running LLM Observability without APM tracing (solves #10051):
DD_APM_TRACING_ENABLED=falsenow drops APM traces. The standalone sampler was only selected when ASM/IAST/SCA was enabled; every other configuration — including LLM Observability — fell through to the default sampler, which keeps traces, so APM traces kept flowing.DD_TRACE_ENABLED=falsewithDD_LLMOBS_ENABLED=trueno longer throws.LLMObsSystem.start()installed the real SDK implementation even with no tracer installed, andAgentTracer.get().buildSpan()returnsnullon the no-op tracer, so everyLLMObs.start*Span()call threw aNullPointerExceptionout of application code.Motivation
DD_APM_TRACING_ENABLED=falseis documented as working in all SDKs. LLM Observability users who don't want APM tracing had no working way to turn it off: the APM flag was a no-op, andDD_TRACE_ENABLED=falsebroke the LLMObs SDK.Additional Notes
Approach. This is a reduced port of #10989, which had gone stale. That PR introduced a
ProductTraceSource.LLMOBSbit plus aStandaloneProductenum and a unifiedStandaloneSamplerreplacingAsmStandaloneSampler. Two things changed since:0x20is now taken byAI_GUARD, and no other tracer allocates an LLMObs_dd.p.tsbit — in dd-trace-js the products are exactlyAPM/ASM/DSM/DJM/DBM/AI_GUARD, and in dd-trace-py exactly the same set. LLM Observability doesn't need one: its spans ship to the LLM Observability intake (/evp_proxy/v2/api/v2/llmobs), which is independent of APM sampling priority, so the correct behavior is to drop the APM traces and let the LLMObs events through.That leaves the fix as the selection logic that other tracers already have — compare
getProductRateLimiterin dd-trace-js (1/minutefor appsec/iast,RateLimiter(0), i.e. drop-all, otherwise) andLLMObsProcessor.process_tracein dd-trace-py (returnsNonefor the whole APM trace when APM tracing is disabled):AsmStandaloneSampler, 1 APM trace per minute for service catalog / billing.ForcePrioritySampler(SAMPLER_DROP, DEFAULT).Because a
DEFAULT-mechanism priority cannot overwrite a priority that is already set, products that force-keep their own traces are unaffected — AI Guard (ai_guard.keep) and ASM (asm.keep) keep working exactly as before, whichAIGuardSamplingTestcovers.Testing.
SamplerTest: APM traces are dropped when APM tracing is disabled, with and without LLM Observability enabled.LLMObsSystemTest: the SDK stays no-op when tracing is disabled.LlmObsApmTracingDisabledSmokeTest: end-to-end — the APM trace is dropped (SAMPLER_DROP,_dd.apm.enabled:0) while the LLMObs span still reaches the LLM Observability intake.LlmObsTraceDisabledSmokeTest: end-to-end — calling the LLMObs SDK withdd.trace.enabled=falseno longer fails the request.AIGuardSamplingTest,ApmTracingDisabledChunkMarkerTestand the otherapm-tracing-disabledsmoke tests pass unchanged.Contributor Checklist
type:and (comp:orinst:) labels in addition to any other useful labelsclose,fix, or any linking keywords when referencing an issueClaude session:
ba8ff7f0-b73e-41d6-a806-3603ab1c7329Resume:
claude --resume ba8ff7f0-b73e-41d6-a806-3603ab1c7329🤖 Generated with Claude Code