Skip to content

Add in LLMObs Tool Definitions API - #12072

Open
sabrenner wants to merge 6 commits into
masterfrom
sabrenner/llmobs-tool-definitions
Open

Add in LLMObs Tool Definitions API#12072
sabrenner wants to merge 6 commits into
masterfrom
sabrenner/llmobs-tool-definitions

Conversation

@sabrenner

Copy link
Copy Markdown
Contributor

What Does This Do

Adds in an API to manually tag tool definitions on LLMObs / Agent Observability spans.

Motivation

Continued parity with the Node.js and Python SDKs.

Contributor Checklist

Jira ticket: MLOB-7900

@sabrenner sabrenner added the comp: mlobs ML Observability (LLMObs) label Jul 24, 2026
@sabrenner
sabrenner marked this pull request as ready for review July 27, 2026 15:04
@sabrenner
sabrenner requested review from a team as code owners July 27, 2026 15:04
@sabrenner
sabrenner requested a review from dougqh July 27, 2026 15:04
@dd-octo-sts

dd-octo-sts Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Hi! 👋 Thanks for your pull request! 🎉

To help us review it, please make sure to:

  • Add at least one type, and one component or instrumentation label to the pull request

If you need help, please check our contributing guidelines.

@datadog-prod-us1-3 datadog-prod-us1-3 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Datadog Autotest: FAIL

The new abstract method on the public LLMObsSpan interface breaks source compatibility for applications that provide their own span implementation: a realistic legacy implementation no longer compiles. Existing tool-definition serialization coverage is present, but the Gradle test suites could not run because this environment lacks the required Java 25 toolchain.

📊 Validated against 1 scenarios · Open Bits AI session

🤖 Datadog Autotest · Commit 6e7642c · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest

Comment thread dd-trace-api/src/main/java/datadog/trace/api/llmobs/LLMObsSpan.java Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6e7642c588

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread dd-trace-api/src/main/java/datadog/trace/api/llmobs/LLMObsSpan.java Outdated

@dougqh dougqh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The API compat breakage raised by codex needs to be addressed.

@sabrenner sabrenner added the type: feature Enhancements and improvements label Jul 27, 2026
@datadog-prod-us1-3

datadog-prod-us1-3 Bot commented Jul 28, 2026

Copy link
Copy Markdown

🎯 Code Coverage (details)
Patch Coverage: 89.66%
Overall Coverage: 57.38% (-0.44%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 89a06a5 | Docs | Datadog PR Page | Give us feedback!

@dd-octo-sts

dd-octo-sts Bot commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.01 s 13.97 s [-0.7%; +1.3%] (no difference)
startup:insecure-bank:tracing:Agent 12.99 s 12.99 s [-0.9%; +0.9%] (no difference)
startup:petclinic:appsec:Agent 16.95 s 16.84 s [-0.2%; +1.5%] (no difference)
startup:petclinic:iast:Agent 16.92 s 16.91 s [-0.9%; +1.0%] (no difference)
startup:petclinic:profiling:Agent 16.94 s 16.98 s [-1.5%; +1.0%] (no difference)
startup:petclinic:sca:Agent 16.85 s 16.77 s [-0.4%; +1.3%] (no difference)
startup:petclinic:tracing:Agent 16.04 s 15.66 s [-1.9%; +6.7%] (no difference)

Commit: 89a06a59 · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@sabrenner

Copy link
Copy Markdown
Contributor Author

@dougqh the issue should be addressed!

@ygree

ygree commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Another round soon, please!

Reviewed commit: d28bfcd3af

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@sabrenner
sabrenner added this pull request to the merge queue Jul 28, 2026
@dd-octo-sts

dd-octo-sts Bot commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

/merge

@gh-worker-devflow-routing-ef8351

gh-worker-devflow-routing-ef8351 Bot commented Jul 28, 2026

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2026-07-28 19:35:00 UTC ℹ️ Start processing command /merge


2026-07-28 19:35:05 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 1h (p90).


2026-07-28 19:54:18 UTCMergeQueue: This merge request was updated

This PR is rejected because it was updated

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jul 28, 2026
@sabrenner
sabrenner added this pull request to the merge queue Jul 28, 2026
@dd-octo-sts

dd-octo-sts Bot commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

/merge

@gh-worker-devflow-routing-ef8351

gh-worker-devflow-routing-ef8351 Bot commented Jul 28, 2026

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2026-07-28 19:50:54 UTC ℹ️ Start processing command /merge


2026-07-28 19:50:57 UTC ❌ MergeQueue

PR already in the queue with status in_progress

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jul 28, 2026
@sabrenner
sabrenner enabled auto-merge July 28, 2026 20:15
@sabrenner
sabrenner added this pull request to the merge queue Jul 28, 2026
@dd-octo-sts

dd-octo-sts Bot commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

/merge

@gh-worker-devflow-routing-ef8351

gh-worker-devflow-routing-ef8351 Bot commented Jul 28, 2026

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2026-07-28 20:54:23 UTC ℹ️ Start processing command /merge


2026-07-28 20:54:27 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 1h (p90).


2026-07-28 21:19:47 UTCMergeQueue: The build pipeline contains failing jobs for this merge request

Build pipeline has failing jobs for 95b9aea:

⚠️ Do NOT retry failed jobs directly (why?).

What to do next?

  • Investigate the failures and when ready, re-add your pull request to the queue!
  • If your PR checks are green, try to rebase/merge. It might be because the CI run is a bit old.
  • Any question, go check the FAQ.
Details

Since those jobs are not marked as being allowed to fail, the pipeline will most likely fail.
Therefore, and to allow other builds to be processed, this merge request has been rejected and the pipeline got canceled.

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jul 28, 2026
@sabrenner
sabrenner added this pull request to the merge queue Jul 29, 2026
@dd-octo-sts

dd-octo-sts Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

/merge

@gh-worker-devflow-routing-ef8351

gh-worker-devflow-routing-ef8351 Bot commented Jul 29, 2026

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2026-07-29 14:31:13 UTC ℹ️ Start processing command /merge


2026-07-29 14:31:19 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 1h (p90).


2026-07-29 14:44:46 UTCMergeQueue: This merge request was updated

This PR is rejected because it was updated

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jul 29, 2026
@sabrenner
sabrenner enabled auto-merge July 29, 2026 15:30
@sabrenner
sabrenner added this pull request to the merge queue Jul 29, 2026
@dd-octo-sts

dd-octo-sts Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

/merge

@gh-worker-devflow-routing-ef8351

gh-worker-devflow-routing-ef8351 Bot commented Jul 29, 2026

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2026-07-29 15:46:08 UTC ℹ️ Start processing command /merge


2026-07-29 15:46:13 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 1h (p90).


2026-07-29 16:00:33 UTCMergeQueue: The build pipeline contains failing jobs for this merge request

Build pipeline has failing jobs for 746d837:

⚠️ Do NOT retry failed jobs directly (why?).

What to do next?

  • Investigate the failures and when ready, re-add your pull request to the queue!
  • If your PR checks are green, try to rebase/merge. It might be because the CI run is a bit old.
  • Any question, go check the FAQ.
Details

Since those jobs are not marked as being allowed to fail, the pipeline will most likely fail.
Therefore, and to allow other builds to be processed, this merge request has been rejected and the pipeline got canceled.

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jul 29, 2026
@sabrenner

Copy link
Copy Markdown
Contributor Author

/merge

@gh-worker-devflow-routing-ef8351

gh-worker-devflow-routing-ef8351 Bot commented Jul 29, 2026

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2026-07-29 16:03:08 UTC ℹ️ Start processing command /merge


2026-07-29 16:03:13 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 1h (p90).


2026-07-29 16:15:38 UTCMergeQueue: The build pipeline contains failing jobs for this merge request

Build pipeline has failing jobs for de53cac:

⚠️ Do NOT retry failed jobs directly (why?).

What to do next?

  • Investigate the failures and when ready, re-add your pull request to the queue!
  • If your PR checks are green, try to rebase/merge. It might be because the CI run is a bit old.
  • Any question, go check the FAQ.
Details

Since those jobs are not marked as being allowed to fail, the pipeline will most likely fail.
Therefore, and to allow other builds to be processed, this merge request has been rejected and the pipeline got canceled.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: mlobs ML Observability (LLMObs) type: feature Enhancements and improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants