metro-file-map: Lazy plugin workers, run on demand

[robhogan]

Summary

A file map plugin with a worker has it run on every file its filter accepts, as files are crawled and as they change. That's right for something like dependency extraction, where nearly every file is needed, but it rules out plugins where only a small, unpredictable subset of files is ever used. Parsing package.json is the motivating case - on our benchmark app a build reads about 9% of the package.json files in the file map (93 of 1,045).

This adds lazy to a plugin's worker description. A lazy plugin's worker isn't run at crawl time or on change, and its filter isn't consulted. Instead the plugin is given processFile(mixedPath) when it's initialised, which runs its worker on that one file - synchronously, in-band - stores the result as the file's plugin data, and returns it. It also emits metadata, as lazy SHA-1 does, so the cache knows there's something to save and processed files stay processed across restarts.

The plugin decides when to call it. Plugin data is undefined for a file until a worker has run on it, and is undefined again once the file changes. null is a result like any other - it's a non-Haste file's Haste name, for example - so what a worker returns is stored with undefined replaced by null.

Making that hold needed one fix. The crawlers and the watcher created file metadata with the first plugin slot already set to null, a relic of when that slot was the Haste name. So an unprocessed file was null to the first plugin and undefined to any other. They now leave plugin slots unset.

On-demand processing bypasses two rules that apply to batches - the plugin's filter, and the exclusion of plugin workers from files in node_modules - because the plugin asked for this file. It writes only that plugin's slot, and doesn't mark the file as visited, since the other workers haven't seen it.

FileDataPlugin takes lazy, gains processFile, and is now exported, so that a plugin can be defined outside metro-file-map without reaching into private paths.

This revives the lazy mode from #1612, on top of the plugin machinery that has landed since.

Test plan

yarn jest packages/metro-file-map packages/metro/src/node-haste packages/metro/src/DeltaBundler packages/metro/src/integration_tests
yarn flow check
yarn typecheck-ts
yarn verify-api-snapshots

• FileProcessor-test.js: a lazy plugin is left out of a batch and its filter isn't called, a file matched only by a lazy plugin isn't processed at all, and processFileForPlugin runs just that plugin in-band for a node_modules file, writes only its slot, and stores null for undefined.
• index-test.js: end to end with a real worker - plugin data is undefined after the crawl, processFile stores it and emits metadata, and a change to the file leaves it undefined again.
• FileDataPlugin-test.js: lazy reaches the worker description, and processFile throws before initialisation and delegates after.
• The crawler tests now expect fresh metadata without a pre-filled plugin slot. Metadata for files already visited keeps its null, which is a real result there.

Changelog: Internal