fs: implement glob natively

[avivkeller]

Ref: https://openjs-foundation.slack.com/archives/C019Y2T6STH/p1787069174254509

(AI assistance used to help port minimatch to C++ and take improvements from other implementations, e.g. Rust's fast-glob [oxc] and picomatch, etc).

I'd love some pretty pedantic reviews since this is substantial. It's 1:1 with the existing glob, so it's non-breaking, but please tear this apart just in case. This implementation is over 2x faster than the old one.

Benchmarking Result (`fs/bench-glob.js`)

                                                                                       confidence improvement accuracy (*)    (**)   (***)
fs/bench-glob.js recursive='false' mode='callback' pattern='*.js' dir='lib' n=1000            ***    110.46 %      ±48.61% ±66.81% ±91.49%
fs/bench-glob.js recursive='false' mode='callback' pattern='**/*' dir='lib' n=1000            ***    110.84 %      ±37.10% ±50.86% ±69.36%
fs/bench-glob.js recursive='false' mode='callback' pattern='**/**.js' dir='lib' n=1000        ***    138.06 %      ±33.34% ±45.04% ±60.01%
fs/bench-glob.js recursive='false' mode='promise' pattern='*.js' dir='lib' n=1000                     -4.74 %      ±15.02% ±20.35% ±27.26%
fs/bench-glob.js recursive='false' mode='promise' pattern='**/*' dir='lib' n=1000                     -0.38 %      ±15.90% ±21.46% ±28.55%
fs/bench-glob.js recursive='false' mode='promise' pattern='**/**.js' dir='lib' n=1000                  3.81 %      ±19.36% ±26.23% ±35.13%
fs/bench-glob.js recursive='false' mode='sync' pattern='*.js' dir='lib' n=1000                ***    110.10 %      ±34.61% ±47.48% ±64.84%
fs/bench-glob.js recursive='false' mode='sync' pattern='**/*' dir='lib' n=1000                 **     50.12 %      ±28.78% ±39.06% ±52.44%
fs/bench-glob.js recursive='false' mode='sync' pattern='**/**.js' dir='lib' n=1000            ***     75.92 %      ±24.18% ±32.63% ±43.39%
fs/bench-glob.js recursive='true' mode='callback' pattern='*.js' dir='lib' n=1000             ***    124.92 %      ±38.32% ±52.47% ±71.43%
fs/bench-glob.js recursive='true' mode='callback' pattern='**/*' dir='lib' n=1000             ***    103.80 %      ±38.52% ±52.93% ±72.43%
fs/bench-glob.js recursive='true' mode='callback' pattern='**/**.js' dir='lib' n=1000         ***    136.45 %      ±31.32% ±42.26% ±56.22%
fs/bench-glob.js recursive='true' mode='promise' pattern='*.js' dir='lib' n=1000                      -5.92 %      ±17.80% ±24.01% ±31.92%
fs/bench-glob.js recursive='true' mode='promise' pattern='**/*' dir='lib' n=1000                       3.44 %      ±19.30% ±26.04% ±34.62%
fs/bench-glob.js recursive='true' mode='promise' pattern='**/**.js' dir='lib' n=1000                   3.60 %      ±16.46% ±22.28% ±29.80%
fs/bench-glob.js recursive='true' mode='sync' pattern='*.js' dir='lib' n=1000                 ***    114.06 %      ±34.05% ±46.87% ±64.34%
fs/bench-glob.js recursive='true' mode='sync' pattern='**/*' dir='lib' n=1000                  **     45.08 %      ±29.24% ±39.76% ±53.53%
fs/bench-glob.js recursive='true' mode='sync' pattern='**/**.js' dir='lib' n=1000             ***     61.45 %      ±27.48% ±37.16% ±49.59%

Be aware that when doing many comparisons the risk of a false-positive
result increases. In this case, there are 18 comparisons, you can thus
expect the following amount of false-positive results:
  0.90 false positives, when considering a   5% risk acceptance (*, **, ***),
  0.18 false positives, when considering a   1% risk acceptance (**, ***),
  0.02 false positives, when considering a 0.1% risk acceptance (***)

---

When compared to existing tools "on-the-market", this implementation also shows a nice improvement.

Benchmarking Result (A fixture written with `mitata`)

• match: literal "foo/bar/baz/asdf/quux.html"	avg	min	p75	p99	max
ours (path.matchesGlob)	115.65 ns/iter	112.17 ns	116.77 ns	126.67 ns	188.02 ns
picomatch	129.99 ns/iter	15.72 ns	16.84 ns	2.96 µs	3.13 µs
minimatch	350.84 ns/iter	193.97 ns	200.49 ns	3.23 µs	3.25 µs

• match: basename star "**/*.html"	avg	min	p75	p99	max
ours (path.matchesGlob)	99.92 ns/iter	94.84 ns	100.61 ns	110.28 ns	180.62 ns
picomatch	295.60 ns/iter	97.50 ns	101.80 ns	6.57 µs	6.82 µs
minimatch	693.57 ns/iter	250.81 ns	260.74 ns	6.86 µs	6.89 µs

• match: globstar hit "**/*.js"	avg	min	p75	p99	max
ours (path.matchesGlob)	110.30 ns/iter	106.86 ns	111.46 ns	118.16 ns	142.25 ns
picomatch	421.55 ns/iter	140.63 ns	151.12 ns	10.22 µs	10.38 µs
minimatch	959.55 ns/iter	287.52 ns	294.57 ns	10.26 µs	10.36 µs

• match: globstar miss "**/*.js"	avg	min	p75	p99	max
ours (path.matchesGlob)	105.53 ns/iter	102.23 ns	106.72 ns	113.43 ns	124.67 ns
picomatch	364.62 ns/iter	156.70 ns	169.64 ns	10.70 µs	12.95 µs
minimatch	1.22 µs/iter	261.57 ns	341.41 ns	14.94 µs	15.42 µs

• match: prefix + globstar "lib/internal/**/*.js"	avg	min	p75	p99	max
ours (path.matchesGlob)	130.76 ns/iter	124.11 ns	130.69 ns	164.40 ns	195.94 ns
picomatch	488.59 ns/iter	109.83 ns	115.52 ns	13.61 µs	14.94 µs
minimatch	1.49 µs/iter	314.71 ns	325.68 ns	15.31 µs	15.40 µs

• match: char class "foo/[bc]ar/baz/**"	avg	min	p75	p99	max
ours (path.matchesGlob)	236.00 ns/iter	224.22 ns	238.94 ns	254.77 ns	284.88 ns
picomatch	534.01 ns/iter	74.33 ns	87.20 ns	15.56 µs	16.40 µs
minimatch	1.63 µs/iter	324.28 ns	336.46 ns	16.45 µs	16.55 µs

• match: negated class "**/[^_]*.js"	avg	min	p75	p99	max
ours (path.matchesGlob)	295.56 ns/iter	288.08 ns	297.49 ns	314.01 ns	342.31 ns
picomatch	571.17 ns/iter	98.33 ns	104.45 ns	16.58 µs	17.09 µs
minimatch	1.34 µs/iter	276.71 ns	301.73 ns	15.57 µs	15.69 µs

• match: braces "src/**/*.{js,mjs,cjs,ts}"	avg	min	p75	p99	max
ours (path.matchesGlob)	135.89 ns/iter	130.60 ns	136.42 ns	168.18 ns	189.51 ns
picomatch	609.29 ns/iter	88.90 ns	102.36 ns	16.24 µs	17.62 µs
minimatch	2.11 µs/iter	426.18 ns	437.44 ns	19.55 µs	19.98 µs

| • match: extglob + braces "foo/+(bar|baz)/**/*.{html,js}" | avg | min | p75 | p99 | max |
| ----------------------- | ---------------- | ----------- | ----------- | ----------- | ----------- |
| ours (path.matchesGlob) | 436.93 ns/iter | 423.99 ns | 439.49 ns | 464.45 ns | 481.09 ns |
| picomatch | 603.44 ns/iter | 85.91 ns | 89.78 ns | 16.76 µs | 17.67 µs |
| minimatch | 1.80 µs/iter | 349.27 ns | 365.06 ns | 20.80 µs | 21.20 µs |

• match: question marks "file-???.log"	avg	min	p75	p99	max
ours (path.matchesGlob)	305.51 ns/iter	292.63 ns	310.04 ns	324.35 ns	335.83 ns
picomatch	552.86 ns/iter	54.58 ns	59.16 ns	18.33 µs	19.87 µs
minimatch	848.20 ns/iter	113.11 ns	117.96 ns	19.97 µs	20.75 µs

• match: star vs long name "*"	avg	min	p75	p99	max
ours (path.matchesGlob)	96.86 ns/iter	93.32 ns	96.83 ns	123.80 ns	132.66 ns
picomatch	773.27 ns/iter	174.68 ns	182.25 ns	22.29 µs	24.21 µs
minimatch	699.20 ns/iter	111.01 ns	117.18 ns	23.31 µs	23.80 µs

• match: interior stars "a//c/**/f/.txt"	avg	min	p75	p99	max
ours (path.matchesGlob)	108.49 ns/iter	105.86 ns	109.03 ns	118.33 ns	123.97 ns
picomatch	703.31 ns/iter	62.55 ns	69.33 ns	24.25 µs	25.31 µs
minimatch	1.82 µs/iter	315.17 ns	340.18 ns	25.59 µs	25.63 µs

• filter 473 paths: "**/*.js"	avg	min	p75	p99	max
ours (path.matchesGlob)	46.75 µs/iter	43.92 µs	46.79 µs	65.75 µs	147.54 µs
picomatch	522.44 µs/iter	480.70 µs	531.32 µs	547.48 µs	600.84 µs
minimatch	683.26 µs/iter	101.25 µs	103.83 µs	130.08 µs	126.54 ms

• filter 473 paths: "internal/**/*.js"	avg	min	p75	p99	max
ours (path.matchesGlob)	53.18 µs/iter	52.80 µs	53.25 µs	53.56 µs	53.59 µs
picomatch	673.80 µs/iter	624.79 µs	692.34 µs	719.43 µs	733.56 µs
minimatch	765.62 µs/iter	110.83 µs	114.29 µs	188.17 µs	159.80 ms

• filter 473 paths: "/{internal,modules}//*.{js,mjs}"	avg	min	p75	p99	max
ours (path.matchesGlob)	98.90 µs/iter	91.33 µs	100.54 µs	130.75 µs	204.67 µs
picomatch	794.39 µs/iter	766.46 µs	801.54 µs	827.51 µs	828.21 µs
minimatch	2.22 ms/iter	174.75 µs	231.50 µs	681.04 µs	165.86 ms

| • filter 473 paths: "**/*.@(md|json)" | avg | min | p75 | p99 | max |
| ----------------------- | ---------------- | ----------- | ----------- | ----------- | ----------- |
| ours (path.matchesGlob) | 211.23 µs/iter | 197.08 µs | 209.67 µs | 266.04 µs | 371.67 µs |
| picomatch | 521.65 µs/iter | 483.29 µs | 523.77 µs | 563.30 µs | 634.47 µs |
| minimatch | 792.63 µs/iter | 117.17 µs | 120.25 µs | 143.08 µs | 180.25 ms |

• compile + match once (600 distinct patterns, all caches missed)	avg	min	p75	p99	max
ours (path.matchesGlob)	3.61 µs/iter	3.50 µs	3.61 µs	3.65 µs	3.67 µs
picomatch	33.06 µs/iter	2.46 µs	7.54 µs	32.38 µs	149.31 ms
minimatch	12.69 µs/iter	5.21 µs	14.42 µs	38.75 µs	137.63 µs

• stress: star-heavy miss "aaaaab"	avg	min	p75	p99	max
ours (path.matchesGlob)	1.73 µs/iter	1.65 µs	1.74 µs	1.83 µs	1.83 µs
picomatch	4.76 ms/iter	4.74 ms	4.76 ms	4.81 ms	4.84 ms
minimatch	4.35 ms/iter	4.24 ms	4.32 ms	4.70 ms	4.81 ms

| • stress: repeated extglob "+(a|b)+(a|b)+(a|b)*c" | avg | min | p75 | p99 | max |
| ----------------------- | ---------------- | ----------- | ----------- | ----------- | ----------- |
| ours (path.matchesGlob) | 4.69 µs/iter | 4.55 µs | 4.74 µs | 4.87 µs | 4.97 µs |
| picomatch | 25.05 µs/iter | 15.83 µs | 16.57 µs | 16.82 µs | 122.37 µs |
| minimatch | 24.64 µs/iter | 14.84 µs | 15.56 µs | 15.92 µs | 125.47 µs |

• stress: stacked globstars "////*.js"	avg	min	p75	p99	max
ours (path.matchesGlob)	138.40 ns/iter	133.54 ns	138.21 ns	163.71 ns	193.29 ns
picomatch	225.15 ns/iter	209.12 ns	232.94 ns	244.57 ns	246.80 ns
minimatch	3.56 µs/iter	500.14 ns	519.71 ns	522.99 ns	43.14 µs

• stress: two long segments "/abc"	avg	min	p75	p99	max
ours (path.matchesGlob)	489.51 ns/iter	470.98 ns	497.42 ns	517.31 ns	533.83 ns
picomatch	520.94 ns/iter	96.63 ns	110.26 ns	505.96 ns	38.76 µs
minimatch	962.92 ns/iter	205.40 ns	215.36 ns	317.59 ns	36.19 µs

• walk: lib/**/*.js	avg	min	p75	p99	max
ours (fs.globSync)	1.67 ms/iter	1.61 ms	1.67 ms	2.12 ms	2.39 ms
glob (minimatch)	2.89 ms/iter	2.56 ms	2.99 ms	3.44 ms	3.81 ms
tinyglobby (picomatch)	1.78 ms/iter	1.69 ms	1.79 ms	2.03 ms	2.74 ms

• walk: */ in lib	avg	min	p75	p99	max
ours (fs.globSync)	1.67 ms/iter	1.61 ms	1.66 ms	1.97 ms	2.07 ms
glob (minimatch)	2.27 ms/iter	2.15 ms	2.24 ms	2.83 ms	2.99 ms
tinyglobby (picomatch)	1.75 ms/iter	1.67 ms	1.76 ms	2.14 ms	2.34 ms

• walk: *.js in lib	avg	min	p75	p99	max
ours (fs.globSync)	79.67 µs/iter	77.00 µs	79.42 µs	101.21 µs	266.54 µs
glob (minimatch)	503.25 µs/iter	118.38 µs	136.38 µs	228.04 µs	163.89 ms
tinyglobby (picomatch)	84.56 µs/iter	76.46 µs	87.21 µs	119.38 µs	167.63 µs

• walk: **/*.{js,mjs} in test/parallel	avg	min	p75	p99	max
ours (fs.globSync)	6.19 ms/iter	6.10 ms	6.23 ms	6.33 ms	6.35 ms
glob (minimatch)	98.22 ms/iter	12.69 ms	196.59 ms	199.43 ms	478.81 ms
tinyglobby (picomatch)	5.53 ms/iter	4.95 ms	5.78 ms	6.02 ms	6.21 ms

AI-Assisted-By: Claude Fable 5, Claude Opus 5

[nodejs-github-bot]

Review requested:

• @nodejs/actions
• @nodejs/performance
• @nodejs/security-wg
• @nodejs/tsc

[avivkeller]

cc @isaacs @nodejs/fs

[codecov]

Codecov Report

❌ Patch coverage is 58.54241% with 1388 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.71%. Comparing base (55e4ca3) to head (fd617c2).
⚠️ Report is 13 commits behind head on main.

Files with missing lines	Patch %	Lines
src/glob/glob_parser.cc	42.23%	422 Missing and 110 partials ⚠️
src/glob/glob_program.cc	55.05%	203 Missing and 77 partials ⚠️
src/glob/glob_matcher.cc	56.99%	131 Missing and 81 partials ⚠️
src/glob/node_glob.cc	64.90%	88 Missing and 45 partials ⚠️
src/glob/glob_walker.cc	80.91%	46 Missing and 67 partials ⚠️
src/glob/glob_unicode.cc	19.64%	88 Missing and 2 partials ⚠️
src/glob/glob_ast.h	69.23%	10 Missing and 2 partials ⚠️
src/glob/glob_unicode.h	16.66%	5 Missing ⚠️
src/glob/node_glob.h	42.85%	4 Missing ⚠️
lib/internal/fs/glob.js	96.73%	2 Missing and 1 partial ⚠️
... and 2 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #65392      +/-   ##
==========================================
- Coverage   90.14%   89.71%   -0.44%     
==========================================
  Files         752      764      +12     
  Lines      251870   254755    +2885     
  Branches    47365    48034     +669     
==========================================
+ Hits       227037   228541    +1504     
- Misses      16177    17147     +970     
- Partials     8656     9067     +411     

Files with missing lines	Coverage Δ
lib/internal/fs/watchers.js	88.06% <100.00%> (-0.22%)	⬇️
lib/internal/test_runner/runner.js	94.80% <100.00%> (-0.01%)	⬇️
src/glob/glob_program.h	100.00% <100.00%> (ø)
src/glob/glob_walker.h	100.00% <100.00%> (ø)
src/lru_cache-inl.h	90.47% <100.00%> (+1.58%)	⬆️
src/node_binding.cc	82.42% <ø> (ø)
src/node_external_reference.h	100.00% <ø> (ø)
src/node_snapshotable.cc	73.53% <ø> (ø)
lib/internal/test_runner/coverage.js	59.38% <94.11%> (-0.20%)	⬇️
lib/internal/fs/glob.js	97.22% <96.73%> (+5.94%)	⬆️
... and 10 more

... and 52 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[aduh95]

take improvements from other implementations, e.g. Rust's fast-glob [oxc] and picomatch, etc

I would feel more comfortable adopting one of those instead of having to maintain our own, have you explored that route?

[avivkeller]

I would feel more comfortable adopting one of those instead of having to maintain our own, have you explored that route?

I have, but the perf wins here are dependent on this being in native, and not in JS:

1. (Minor) We are calling libuv directly, so we don't need to flip-flop between C++ and JS

2. (Major) We are writing code highly optimized for Node.js's usage. Existing JS implementations compile to a RegEx and rely on V8 to optimize it, however, we can skip that whole step by not relying on V8 at all, and manually compiling the glob to an AST.

   If a directory has 100k entries and 500 match, existing implementations are forced to create and discard ~99.5k strings to get there (since they are written in JS), in a native implementation, we don't need to create, allocate, or garbage collect anything inside of V8, we can handle it all natively.

I also think this can bring some improvements to things like the test runner. For instance, --test-coverage-include/exclude is globbed on the native side (like a supercharged version of #65371), we should be able to bring this performance improvement elsewhere. Maybe this can help in similar ways in other places to?

[avivkeller]

(Whoops!)