perf(pipeline): use regex-lite in the wasm build - #192
Open
bengl wants to merge 1 commit into
Open
Conversation
libdd-common declares regex as a non-optional dependency, resolved with every unicode feature enabled, so the pipeline wasm carried ~394 KB of unicode range tables and a full regex engine for code that never evaluates a regex. Enabling libdd-common's regex-lite feature leaves the full regex crate unreferenced, and LTO drops it. With wasm-opt -O, matching what CI publishes, pipeline_bg.wasm goes from 1,507,189 to 754,457 bytes raw (-49.9%) and 564,771 to 304,880 bytes gzipped (-46.0%). The code section shrinks 39.8% and the data section 72.6%. This is safe because the call sites that compile user-supplied patterns are not reachable here: obfuscation in libdd-data-pipeline sits behind its stats-obfuscation feature, which this crate does not enable, and nothing in crates/pipeline or crates/capabilities builds a Regex or calls trace_filter. require-regex-full, which would override regex-lite, is only enabled by datadog-ffe, which is not in this dependency graph. test/pipeline.js passes 51/51 and test/http_transport.js 18/18 against the rebuilt module.
rochdev
approved these changes
Jul 31, 2026
Collaborator
|
FYI, regex-lite is much slower and you probably do want obfuscation turned on for better CSS cardinality and for agentless export. I'd recommend keeping a close eye on benchmarks if you really want to use regex-lite. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Enables
libdd-common'sregex-litefeature for thepipelinewasm crate. This halvespipeline_bg.wasm.All figures below are with
wasm-opt -O --all-features, i.e. what CI actually publishes:For reference,
pipeline_bg.wasmin the published@datadog/libdatadog@0.18.1is 1,509,952 / 565,867 — the "before" row reproduces it.Section breakdown:
codedatadataWhy it was so big
libdd-commondeclaresregex = "1.5"as a non-optional dependency, so every libdd crate drags in the fullregexengine.cargo tree --target wasm32-unknown-unknownresolved it with every Unicode feature enabled —unicode-age, unicode-bool, unicode-case, unicode-gencat, unicode-perl, unicode-script, unicode-segment— plus allperf-*engines (DFA, onepass, backtrack, literal, cache).The result was ~394 KB of Unicode range tables and a full regex engine in a module that never evaluates a regex. Confirmed by parsing the published artifact's data section: it contained
regex-syntax's property-name tables (ASCII_Hex_Digit,Bidi_Class,Bidi_Mirrored), script long names (Anatolian_Hieroglyphs,Caucasian_Albanian) and general-category tables (Close_Punctuation,Connector_Punctuation). After this change all three groups of markers are gone from the artifact.Why it is safe
Upstream built this exact escape hatch:
libdd-common'sregex_enginemodule re-exports eitherregexorregex_lite, and its onlyuse regex::is inside that cfg gate.libdd-trace-obfuscation(replacer.rs,ip_address.rs) andlibdd-trace-utils::trace_filterimportlibdd_common::regex_engine, notregexdirectly.replacer.rs:42/:126andtrace_filter.rs:145/:172— are not reachable from this module. Obfuscation inlibdd-data-pipelinesits behind itsstats-obfuscationfeature, which we do not enable (default-features = false), andcrates/pipeline/src/stats.rsalready documents stats obfuscation as disabled. Nothing incrates/pipelineorcrates/capabilitiesconstructs aRegexor callstrace_filter. That unreachability is exactly why LTO can drop the engine.require-regex-full, which would overrideregex-lite, is only enabled bydatadog-ffe— not in this dependency graph (0 dependency paths topipeline).regex-litedoes not support (\p{…}, look-around, backreferences): none present.regexstays inCargo.lockbecause it is still a non-optional dependency oflibdd-common; the win comes from it becoming unreferenced and being dead-stripped. The lockfile gainsregex-lite 0.1.9.Verification
Built locally on macOS (
node scripts/build-wasm.js pipeline) before and after, then ranwasm-optover both to match the CI artifact.node --test --test-force-exit test/pipeline.js→ 51/51 passnode --test --test-force-exit test/http_transport.js→ 18/18 passRejected:
wasm-opt -OzI also measured raising the optimisation level, since
crates/pipeline/Cargo.tomlcurrently setswasm-opt = ['-O', '--all-features']. Not worth it — on the same input-Ozsaves 2.2% raw and is larger on the wire:-O-Ozwasm-opt's transforms trade compressibility for raw size here, so
-Ostays. (Note both optimised builds gzip worse than the unoptimised module — worth revisiting separately if download size is what we care about, since npm ships gzipped tarballs.)Follow-ups not in this PR
hashbrownversions (0.15.5and0.17.0) resolve in the graph, so two hash-table implementations compile in — needs upstream version alignment.httpcrate's standard-header table appeared twice in the published data section, andhttp::header::name::StandardHeader::from_bytesis a single 12,120-byte function.~/.cargo/git/checkouts/libdatadog-…paths).regexoptional inlibdd-commonupstream would remove it from the lockfile rather than relying on dead-code elimination.