perf(profiling): speed up ddprof_time sample processing by realFlowControl · Pull Request #4194 · DataDog/dd-trace-php

realFlowControl · 2026-09-15T08:43:16Z

Description

The profiler sends samples through a bounded 100-message channel using non-blocking sends. The original /dev/null I/O upscaling workload can produce samples faster than ddprof_time aggregates them, which fills the channel and drops samples. I profiled the complete PHP process with macOS sample and optimized the measured ddprof_time hot paths.

This PR is intentionally split into five commits and can be reviewed commit by commit:

01c52f4 uses the already available FxHashMap for the trusted, low-cardinality internal profile map. The initial profile showed 32% of active message-processing samples in SipHash.
0cb42e1 caches ProfileIndex and its semantic hash. Samples clone the cached Arc, while equal identities from different requests still aggregate into the same profile.
be742e8 drains up to the bounded channel capacity after select! wakes the consumer. This amortizes channel registration and timer checks without allowing unbounded timer starvation.
cd72040 keeps normal frame and label payloads directly owned. It promotes them to Arc only when experimental heap-live tracking actually needs shared ownership.
8dbc671 creates the profile identity cache lazily from the first sample and refreshes it when the sample context changes.

Median PHP user CPU for the same 100x stress workload improved from 14.94s to 11.90s, a 20.3% reduction. The incremental medians were 14.07s after the map change, 12.57s after cached identity, 12.35s after batch draining, and 11.90s after owned payloads.

Reviewer checklist

Test coverage seems ok.
Appropriate labels assigned.

Profiling the full PHP process under the reverted I/O upscaling workload showed that ddprof_time spent 912 of 2823 active samples (32%) hashing ProfileIndex with SipHash. Use the already available FxHashMap for the internal profile map. The map normally contains one trusted, low-cardinality key, so SipHash's collision resistance adds cost without useful protection. In matched process profiles, active ddprof_time samples fell from 2823 to 2056 (-27%). Across three 100x workloads, median PHP user CPU fell from 14.94s to 14.07s (-5.8%).

Build ProfileIndex and its semantic hash once per request instead of rebuilding and hashing all sample types and tags for every sample. Equal profile identities from different requests still compare equal and aggregate into the same profile. The full-process profile showed ddprof_time message-processing samples fall from 2056 to 1451 (-29.4%). Median PHP user CPU fell from 14.07s to 12.57s (-10.7%).

After select wakes ddprof_time, process up to the bounded channel capacity before returning to select. This amortizes channel registration and timer checks while limiting timer delay to 100 messages. Estimated active ddprof_time CPU fell from 2061 to 1984 samples (-3.7%). Median PHP user CPU fell from 12.57s to 12.35s (-1.8%), and stress-run drops fell from 3/68/350 to 0/0/1.

Keep normal sample frames and labels directly owned instead of allocating two Arc wrappers per sample. Promote them to Arc only when experimental heap-live tracking needs to share the payload with the allocation tracker. Median PHP user CPU fell from 12.35s to 11.90s (-3.6%). The full-process profile no longer showed Arc<Backtrace>::drop_slow, and stress-run drops fell from 0/0/1 to 0/0/0.

datadog-prod-us1-4 · 2026-09-15T08:51:01Z

✨ Unblock PR with BitsAI

❌ Errors

Your PR has failed checks. Please review the issues below and take necessary action before merging.

🚦 10 Pipeline jobs failed

DataDog/apm-reliability/dd-trace-php | check libxml2 version — 🔧 Needs a code fix, caused by this PR

View more details · View in GitLab

DataDog/apm-reliability/dd-trace-php | test_extension_ci: [7.3, valgrind] — 🔧 Needs a code fix, caused by this PR

View more details · View in GitLab

DataDog/apm-reliability/dd-trace-php | test_extension_ci: [8.3] — 🔧 Needs a code fix, caused by this PR

View more details · View in GitLab

View all 10 failed jobs.

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 67.44% (-0.05%)

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 6c35136 | Docs | View more details | Give us feedback!}

pr-commenter · 2026-09-15T09:08:42Z

Benchmarks [ profiler ]

Benchmark execution time: 2026-09-18 15:03:50

Comparing candidate commit 6c35136 in PR branch florian/optimize-ddprof-time with baseline commit 8097433 in branch master.

Found 6 performance improvements and 0 performance regressions! Performance is the same for 21 metrics, 9 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

🟩 = significantly better candidate vs. baseline
🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:php-profiler-timeline-memory-with-profiler

🟩 cpu_usage_percentage [-5.712%; -4.231%]
🟩 cpu_user_time [-157.520ms; -109.829ms] or [-11.232%; -7.831%]
🟩 execution_time [-60.841ms; -42.639ms] or [-5.128%; -3.594%]

scenario:php-profiler-timeline-memory-with-profiler-and-timeline

🟩 cpu_usage_percentage [-6.040%; -4.593%]
🟩 cpu_user_time [-179.938ms; -141.118ms] or [-11.623%; -9.115%]
🟩 execution_time [-72.768ms; -52.173ms] or [-5.229%; -3.749%]

pr-commenter · 2026-09-17T11:21:42Z

Benchmarks [ tracer ]

Benchmark execution time: 2026-09-17 11:20:08

Comparing candidate commit 8dbc671 in PR branch florian/optimize-ddprof-time with baseline commit 4622bea in branch master.

Found 1 performance improvements and 0 performance regressions! Performance is the same for 193 metrics, 0 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

🟩 = significantly better candidate vs. baseline
🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:MessagePackSerializationBench/benchMessagePackSerialization-opcache

🟩 execution_time [-4.247µs; -2.973µs] or [-3.804%; -2.663%]

…ion" This reverts commit 23277c9.

…trace-php into florian/optimize-ddprof-time

realFlowControl added 4 commits September 15, 2026 10:09

github-actions Bot added profiling Relates to the Continuous Profiler tracing labels Sep 15, 2026

fix(profiling): refresh cached profile identity

8dbc671

realFlowControl mentioned this pull request Sep 17, 2026

fix(profiling): avoid holding request locals across callbacks #4208

Closed

2 tasks

realFlowControl added 2 commits September 17, 2026 17:07

fix(profiling): release request locals before trace notification

23277c9

Merge branch 'master' into florian/optimize-ddprof-time

19184e5

morrisonlevi mentioned this pull request Sep 17, 2026

test(profiling): avoid I/O sample queue overflow #4193

Merged

2 tasks

realFlowControl and others added 4 commits September 18, 2026 13:12

Revert "fix(profiling): release request locals before trace notificat…

9d730d9

…ion" This reverts commit 23277c9.

Merge branch 'master' into florian/optimize-ddprof-time

741eb01

fix re-entrancy in minfo()

de670c8

Merge branch 'florian/optimize-ddprof-time' of github.com:DataDog/dd-…

6c35136

…trace-php into florian/optimize-ddprof-time

Navigation Menu

perf(profiling): speed up ddprof_time sample processing - #4194

perf(profiling): speed up ddprof_time sample processing#4194
realFlowControl wants to merge 11 commits into
masterfrom
florian/optimize-ddprof-time

realFlowControl commented Sep 15, 2026 •

edited

Loading

Uh oh!

datadog-prod-us1-4 Bot commented Sep 15, 2026 •

edited by datadog-prod-us1-6 Bot

Loading

Uh oh!

pr-commenter Bot commented Sep 15, 2026 •

edited

Loading

Explanation

More details about the CI and significant changes

Uh oh!

pr-commenter Bot commented Sep 17, 2026

Explanation

More details about the CI and significant changes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

realFlowControl commented Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Reviewer checklist

Uh oh!

datadog-prod-us1-4 Bot commented Sep 15, 2026 • edited by datadog-prod-us1-6 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ Errors

ℹ️ Info

Uh oh!

pr-commenter Bot commented Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks [ profiler ]

Explanation

More details about the CI and significant changes

scenario:php-profiler-timeline-memory-with-profiler

scenario:php-profiler-timeline-memory-with-profiler-and-timeline

Uh oh!

pr-commenter Bot commented Sep 17, 2026

Benchmarks [ tracer ]

Explanation

More details about the CI and significant changes

scenario:MessagePackSerializationBench/benchMessagePackSerialization-opcache

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

realFlowControl commented Sep 15, 2026 •

edited

Loading

datadog-prod-us1-4 Bot commented Sep 15, 2026 •

edited by datadog-prod-us1-6 Bot

Loading

pr-commenter Bot commented Sep 15, 2026 •

edited

Loading