feat(data-pipeline): add runtime-independent agentless sending - #2389
Conversation
|
✅ All CI checks and tests passed. 🎉 All green!🧪 All tests passed 🎯 Code Coverage (details) 🔗 Commit SHA: 85cf00d | Docs | View more details | Give us feedback! |
Artifact Size Benchmark Reportaarch64-alpine-linux-musl
aarch64-unknown-linux-gnu
libdatadog-x64-windows
libdatadog-x86-windows
x86_64-alpine-linux-musl
x86_64-unknown-linux-gnu
|
BenchmarksComparisonBenchmark execution time: 2026-08-26 22:29:06 Comparing candidate commit 85cf00d in PR branch Found 5 performance improvements and 19 performance regressions! Performance is the same for 129 metrics, 0 unstable metrics.
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fc9ff139a3
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
1c04286 to
52dfcbb
Compare
bengl
left a comment
There was a problem hiding this comment.
Seems fine to me, but @DataDog/apm-common-components-core should probably review.
Implementation changed significantly to keep retries on the Rust side.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 322d9336fe
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
Overall, I'd like to push back against this pull request.
First without the matching libdatadog-nodejs PR it is very hard to reason about which APIs are necessary and which are not.
But more than this, I don't understand why we need the PreparedRequest level of abstraction, and this is not really motivated in the PR description.
What is the goal here
- Is it getting just a bag of data with the request and then doing the http request fully in NodeJS?
- Or is it to have a
send_agentlessfunction that takes messagepack &[u8] serialized trace chunk and uses the existing capabilities implementation to drive calling to the node http stack from rust?
While I am not completely against 1), I would much prefer option 2) as it doesn't expose as much details of libdatadog to nodeJS and you should just be able to hook the exisitng WASM capabilities implementation.
We should just do something like
NodeJS | Rust | NodeJS
send(chunk) -> encode msp -> send(traces: &[u8]) -> encode() -> send_with_retry -> http capability impl
and not
NodeJS | Rust | NodeJS | Rust | NodeJS
prepare(chunk) -> encode msp -> prepare(traces: &[u8]) -> returns prepared request -> send_rust(prepared_request) -> send_with_retry -> http capability
b442108 to
85cf00d
Compare
|
/merge |
|
View all feedbacks in Devflow UI.
It will be processed automatically as soon as GitHub reports it as mergeable. View in MergeQueue UI.
roch.devost@datadoghq.com unqueued this merge request |
|
/code blockers |
|
View all feedbacks in Devflow UI.
Checking merge blockers for #2389...
No merge blockers detected. |
|
/merge |
|
View all feedbacks in Devflow UI.
PR already in the queue with status waiting |
|
/merge -c |
|
View all feedbacks in Devflow UI.
|
|
/merge |
|
View all feedbacks in Devflow UI.
The expected merge time in
|
What does this PR do?
Adds a runtime-independent agentless data-pipeline send path. Rust owns trace
preparation, JSON encoding, compression, retry policy, timeouts, backoff, and
response handling; the host provides only HTTP and sleep capabilities.
The existing
libdd-data-pipelineexporter now uses the same high-level sendpath, so native and WASM consumers share the behavior without requiring a
specific async runtime.
Motivation
Node.js needs equivalent native and WASM data-pipeline implementations without
embedding Tokio and Hyper in the native extension. Keeping the send lifecycle
in Rust preserves retry behavior and makes the capability boundary reusable by
future targets.
Additional Notes
libdatadog-nodejs.This PR was generated by Codex.
How to test the change?
cargo test -p libdd-data-pipeline-core --features compressioncargo test -p libdd-trace-utils send_with_retrycargo test -p libdd-data-pipeline agentlesscargo clippy -p libdd-data-pipeline-core --all-targets --features compression -- -D warningscargo clippy -p libdd-trace-utils --lib --tests -- -D warningscargo clippy -p libdd-data-pipeline --lib -- -D warnings