Skip to content

perf(pipeline): use regex-lite in the wasm build - #192

Open
bengl wants to merge 1 commit into
mainfrom
bengl/shrink-pipeline-wasm
Open

perf(pipeline): use regex-lite in the wasm build#192
bengl wants to merge 1 commit into
mainfrom
bengl/shrink-pipeline-wasm

Conversation

@bengl

@bengl bengl commented Jul 31, 2026

Copy link
Copy Markdown
Collaborator

What

Enables libdd-common's regex-lite feature for the pipeline wasm crate. This halves pipeline_bg.wasm.

All figures below are with wasm-opt -O --all-features, i.e. what CI actually publishes:

raw gzip
before 1,507,189 564,771
after 754,457 304,880
delta −752,732 (−49.9%) −259,891 (−46.0%)

For reference, pipeline_bg.wasm in the published @datadog/libdatadog@0.18.1 is 1,509,952 / 565,867 — the "before" row reproduces it.

Section breakdown:

section before after
code 1,020,947 614,571 (−39.8%)
data 477,819 131,060 (−72.6%)
└ binary static tables within data ~393,704 ~71,944 (−81.7%)

Why it was so big

libdd-common declares regex = "1.5" as a non-optional dependency, so every libdd crate drags in the full regex engine. cargo tree --target wasm32-unknown-unknown resolved it with every Unicode feature enabled — unicode-age, unicode-bool, unicode-case, unicode-gencat, unicode-perl, unicode-script, unicode-segment — plus all perf-* engines (DFA, onepass, backtrack, literal, cache).

The result was ~394 KB of Unicode range tables and a full regex engine in a module that never evaluates a regex. Confirmed by parsing the published artifact's data section: it contained regex-syntax's property-name tables (ASCII_Hex_Digit, Bidi_Class, Bidi_Mirrored), script long names (Anatolian_Hieroglyphs, Caucasian_Albanian) and general-category tables (Close_Punctuation, Connector_Punctuation). After this change all three groups of markers are gone from the artifact.

Why it is safe

Upstream built this exact escape hatch: libdd-common's regex_engine module re-exports either regex or regex_lite, and its only use regex:: is inside that cfg gate.

  • libdd-trace-obfuscation (replacer.rs, ip_address.rs) and libdd-trace-utils::trace_filter import libdd_common::regex_engine, not regex directly.
  • The call sites that compile user-supplied patterns — replacer.rs:42/:126 and trace_filter.rs:145/:172 — are not reachable from this module. Obfuscation in libdd-data-pipeline sits behind its stats-obfuscation feature, which we do not enable (default-features = false), and crates/pipeline/src/stats.rs already documents stats obfuscation as disabled. Nothing in crates/pipeline or crates/capabilities constructs a Regex or calls trace_filter. That unreachability is exactly why LTO can drop the engine.
  • require-regex-full, which would override regex-lite, is only enabled by datadog-ffe — not in this dependency graph (0 dependency paths to pipeline).
  • Grepped the obfuscation and filter sources for constructs regex-lite does not support (\p{…}, look-around, backreferences): none present.

regex stays in Cargo.lock because it is still a non-optional dependency of libdd-common; the win comes from it becoming unreferenced and being dead-stripped. The lockfile gains regex-lite 0.1.9.

Verification

Built locally on macOS (node scripts/build-wasm.js pipeline) before and after, then ran wasm-opt over both to match the CI artifact.

  • node --test --test-force-exit test/pipeline.js51/51 pass
  • node --test --test-force-exit test/http_transport.js18/18 pass

Rejected: wasm-opt -Oz

I also measured raising the optimisation level, since crates/pipeline/Cargo.toml currently sets wasm-opt = ['-O', '--all-features']. Not worth it — on the same input -Oz saves 2.2% raw and is larger on the wire:

raw gzip
unoptimised 1,657,600 552,858
-O 1,507,189 564,771
-Oz 1,474,098 568,015

wasm-opt's transforms trade compressibility for raw size here, so -O stays. (Note both optimised builds gzip worse than the unoptimised module — worth revisiting separately if download size is what we care about, since npm ships gzipped tarballs.)

Follow-ups not in this PR

  • Two hashbrown versions (0.15.5 and 0.17.0) resolve in the graph, so two hash-table implementations compile in — needs upstream version alignment.
  • The http crate's standard-header table appeared twice in the published data section, and http::header::name::StandardHeader::from_bytes is a single 12,120-byte function.
  • ~33 KB of embedded panic-location strings (absolute ~/.cargo/git/checkouts/libdatadog-… paths).
  • Making regex optional in libdd-common upstream would remove it from the lockfile rather than relying on dead-code elimination.

libdd-common declares regex as a non-optional dependency, resolved with
every unicode feature enabled, so the pipeline wasm carried ~394 KB of
unicode range tables and a full regex engine for code that never
evaluates a regex. Enabling libdd-common's regex-lite feature leaves the
full regex crate unreferenced, and LTO drops it.

With wasm-opt -O, matching what CI publishes, pipeline_bg.wasm goes from
1,507,189 to 754,457 bytes raw (-49.9%) and 564,771 to 304,880 bytes
gzipped (-46.0%). The code section shrinks 39.8% and the data section
72.6%.

This is safe because the call sites that compile user-supplied patterns
are not reachable here: obfuscation in libdd-data-pipeline sits behind
its stats-obfuscation feature, which this crate does not enable, and
nothing in crates/pipeline or crates/capabilities builds a Regex or calls
trace_filter. require-regex-full, which would override regex-lite, is
only enabled by datadog-ffe, which is not in this dependency graph.

test/pipeline.js passes 51/51 and test/http_transport.js 18/18 against
the rebuilt module.
@bengl
bengl requested review from a team as code owners July 31, 2026 20:29
@ekump

ekump commented Jul 31, 2026

Copy link
Copy Markdown
Collaborator

FYI, regex-lite is much slower and you probably do want obfuscation turned on for better CSS cardinality and for agentless export. I'd recommend keeping a close eye on benchmarks if you really want to use regex-lite.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants