feat(array): builders no longer canonicalize their children [builders-child-stack]

[robert3005]

Rationale for this change

Canonical form is not recursive — Canonical only promises a canonical top level. Nested builders did not honour that: every appended child array was pushed through append_to_builder, which decoded it into the child's canonical builder. Struct fields, list elements and extension storage were all decompressed on the way through a builder that had no reason to look at them.

Third PR in the stack, after #9046 and #9064. The eventual goal is to delete the hand-written pack_struct_chunks / swizzle_list_chunks / swizzle_fixed_size_list_chunks special cases in ChunkedArray's canonicalization and route every dtype through append_to_builder — those helpers exist only because the builders used to decode children, and their doc comments describe exactly what this PR makes the builders do.

What changes are included in this PR?

ChildBuilder (vortex-array/src/builders/child.rs) accumulates a nested builder's child from the two sources such a builder receives:

• appended arrays are kept as chunks, in whatever encoding they arrived in
• appended scalars (and zeros/nulls/defaults) are materialized into a canonical builder

finish stitches the two back together, returning a ChunkedArray only when more than one chunk accumulated. StructBuilder, ListBuilder, ListViewBuilder, FixedSizeListBuilder and ExtensionBuilder now hold their children this way. Those are all the builders with array children; bool/primitive/decimal/varbinview/null have none, and Union/Variant builders do not exist yet.

However short the appended array, it becomes a chunk. An earlier revision of this PR kept a MIN_CHUNK_LEN (64) below which arrays were copied into the scalar builder instead, because some callers append one list at a time — the load-bearing case being SparseArray canonicalization, whose fill loop called append_array_as_list(fill_elements, ctx) once per filled row and would have produced a chunk per row.

That threshold is gone. Deciding on the caller's behalf that its values are cheaper copied than referenced is guessing at a boundary only the caller can see, and a caller that wants them copied already has append_scalar. #9064 converts the looping callers that motivated the threshold — the sparse fill loops, the sparse patch loops, and ListBuilder::extend_from_listview — to bulk appends, so a child is now chunked on exactly the boundaries it was appended on.

A test in encodings/sparse pins that down where it is observable: 100 patches on consecutive rows of a 200-row sparse list array produce an elements child of exactly 2 chunks — one for the patch run, one for the trailing gap. Without the run coalescing in #9064 it is 101.

ChildBuilder::set_validity_unchecked pushes the override into each chunk as a MaskedArray rather than decoding. That path panics if a chunk already contains nulls, since replacing its validity would mean decoding it; it is documented on the method. ExtensionBuilder is the only builder that forwards set_validity into a child — the others keep their own validity buffer — and the fast path (nothing chunked yet) is unchanged.

Tests

22 new tests. Beyond "children survive undecoded" for each nested builder, they cover the arithmetic and validity handling that the chunk path changes:

• empty arrays never become chunks; dtype mismatches are rejected whether or not the array would have become a chunk; scalars, zeros and nulls appended around a chunk keep their order
• set_validity_unchecked slices the override per chunk, covers values still pending in the scalar builder, is dropped for a non-nullable child, and panics on a chunk that already contains nulls
• list offsets are rebased onto the running element count once elements live in chunks — for ListViewArray and ListArray inputs and for ListBuilder::append_array_as_list
• nested builders keep their own validity alongside a chunked child, extension storage receives a validity override through its chunks, and a built array with chunked children still canonicalizes recursively

Each test was checked against a mutated implementation. Dropping the per-chunk validity slice, dropping the pending flush, zeroing the listview offset base, and taking the ListBuilder offset from the appended array instead of the running total each fail the intended tests. Reverting to the old always-materialize behaviour fails 17 of them.

What APIs are changed? Are there any user-facing changes?

No public API changes — ChildBuilder is pub(crate). The user-facing change is the shape of what builders return: ArrayBuilder::finish is now documented as canonical at the top level only, and callers that need a fully decoded tree should ask for one via RecursiveCanonical (arrow export and the DuckDB exporter already recurse per child, so both are unaffected).

Checks

• cargo nextest run --workspace — 7168 passed, 560 skipped
• cargo clippy --all-targets --all-features — clean
• cargo +nightly fmt --all --check — clean
• cargo test --doc -p vortex-array -p vortex-sparse — passed

One existing test needed a fix rather than an update to match: uncompressed_size_in_bytes::list_matches_materialized_size compares the aggregate against builder.finish().nbytes(), and a builder that keeps a dict-encoded child no longer measures anything "materialized". Its helper now finishes the round-trip with RecursiveCanonical before taking nbytes, which is what the name always promised.

---

Generated by Claude Code

Stacked PR Chain: builders-child-stack

PR	Title	Merges Into
#9046	perf(array): stop flattening appended list views in ListViewBuilder [builders-child-stack]	N/A
#8964	feat(array): builders no longer canonicalize their children [builders-child-stack]	#9046
#8965	feat(array): let callers choose a nested builder's chunk threshold [builders-child-stack]	#8964
#8966	perf(array): accumulate nested builder validity without a null buffer [builders-child-stack]	#8965
#8967	refactor(array): canonicalize chunked structs and FSLs through the builder [builders-child-stack]	#8966

---

Generated by Claude Code

[codspeed-hq]

Merging this PR will improve performance by 60.35%

⚡ 4 improved benchmarks
✅ 1837 untouched benchmarks
⏩ 55 skipped benchmarks¹

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
⚡	Simulation	extend_from_array_zctl[(1000, 64)]	728.2 µs	280 µs	×2.6
⚡	Simulation	extend_from_array_zctl[(10000, 8)]	1,185.9 µs	626.6 µs	+89.26%
⚡	Simulation	extend_from_array_zctl[(1000, 8)]	337.9 µs	279.6 µs	+20.85%
⚡	Simulation	extend_from_array_non_zctl_overlapping[(10000, 8)]	705.1 µs	634.4 µs	+11.14%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing claude/builders-canonical-children-9ze0t6 (df75205) with claude/builders-bulk-loop-callers-9ze0t6 (3304731)}

Footnotes

1. 55 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[robert3005]

I am not sure this is real, need to see which benchmarks regress