Batch the per-feature database round trips in feature import

[njakobsen]

Stacked on #67, which it takes as its base. The traversal is fast after that PR; this is where the time goes next.

Pure performance: identical output, and that is measured rather than asserted. Nothing here changes what is imported, so neither #68 nor #69 is touched.

Where the time goes

Profiling one import, with SQL bucketed by statement shape, 9,164 features:

Stage	Wall
parse + pre-pass	0.03s	0.2%
build (traverse + parse geometry)	2.92s	19.0%
save loop	10.73s	69.7%
cache_derivatives	1.18s	7.7%
aggregate	0.25s	1.6%
area	0.28s	1.8%

SQL	Calls	Time	Per call	% of db
INSERT feature	9,165	3.40s	0.371ms	32.4%
validate geometry	9,654	1.98s	0.205ms	18.9%
ST_Force2D	9,654	1.32s	0.137ms	12.6%
ST_GeomFromKML	9,164	1.23s	0.135ms	11.8%
UPDATE (cache_derivatives)	8	1.17s	146ms	11.2%
savepoints and other	18,823	0.90s	0.048ms	8.6%
aggregate union	2	0.48s	240ms	4.6%

6.16 statements per feature. Database time was 68% of wall clock, Ruby the other 32%.

What this changes

Geometry validity, read a batch at a time. ::precompute_geometry_validation asks for a batch in one query and hands each record its own answer. save is untouched and still runs every callback. A record whose geometry is repaired discards the answer, because make_valid replaces the geometry it was about, and falls back to asking for itself.

KML geometry, parsed a batch at a time. Elements are held until there are enough to parse together. The savepoint moves with them: a batch holding an element PostGIS rejects fails as a whole, is caught, and is re-read one element at a time so only that element is lost — the same containment the per-element savepoints gave.

The inserts are left alone, at 32.4% the largest single item. Batching them means insert_all, which bypasses the callback chain, and the import path would then silently skip any callback a future change adds to Feature. That is a maintenance hazard rather than a correctness one, but it is not "no judgment" in the way the rest of this is, so it wants its own discussion. ST_Force2D is in the same position, since it runs as a before_save.

Proving the output is unchanged

Both branches were run over the same four real source files, 136,769 features, and digested:

	Baseline (#67 head)	This branch
Features built	136,769	136,769
Features saved	136,769	136,769
Importer output stream	de693be5…	de693be5…
Stored rows	16a43844…	16a43844…
Warnings	f3d76280…	f3d76280…

The stream digest covers geometry, name, metadata and image paths for every feature in yield order. The rows digest covers the geometry as stored, read back as WKB hex, with name, metadata and source identifier, ordered by id. All three match exactly.

Numbers

	Baseline	This branch
Build (traverse + parse geometry)	39.4s	15.2s	2.6x
Save loop	111.5s	72.6s	1.5x
Combined	150.9s	87.8s	1.7x

Measured on a laptop against local Postgres, where a round trip is cheap. Statements per feature fall from 6.16 to roughly 2.4, so the gain should be larger where round trips cost more than they do here.

Batch sizes and memory

Both sizes are bounded by statement size rather than record count, since each batch is sent as a literal list of geometries — a file of few but very large geometries reaches the ceiling first. At the shipped sizes (500 features, 200 geometry elements), over the 136,769-feature import:

• Largest statement: 0.39 MB, against 0.05 MB on the baseline
• Peak RSS: 1184 MB against 1157 MB, up 2.3%

The 1.1 GB is pre-existing and is the importer holding every feature in memory before saving; this branch adds a bounded buffer and one statement string on top of it.

Verification

Full suite: 315 examples, 0 failures, 19 pending.

New specs cover the batch falling back to one query per element when it holds an unreadable one, that batched validity matches the per-record answer, that nothing is queried again while the answer holds, and that a repair discards it.

🤖 Generated with Claude Code

https://claude.ai/code/session_012b7n8NbLAd6s7c3atii4Mv