Skip to content

refactor(array): canonicalize chunked structs and FSLs through the builder [builders-child-stack] - #8967

Open
robert3005 wants to merge 1 commit into
claude/builders-lazy-validity-9ze0t6from
claude/chunked-canonical-via-builder-9ze0t6
Open

refactor(array): canonicalize chunked structs and FSLs through the builder [builders-child-stack]#8967
robert3005 wants to merge 1 commit into
claude/builders-lazy-validity-9ze0t6from
claude/chunked-canonical-via-builder-9ze0t6

Conversation

@robert3005

@robert3005 robert3005 commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

Rationale for this change

Stacked on #8966#8964#9064#9046 — review those first. This is the payoff for the rest of the stack.

ChunkedArray's _canonicalize carried four hand-written special cases plus a generic append_to_builder fallback. Two of them existed purely because append_to_builder used to decode a builder's children; their doc comments say so:

  • pack_struct_chunks — "a single StructArray, where the data for each field is a ChunkedArray"
  • swizzle_fixed_size_list_chunks — "reuse the chunks' elements children directly as the chunks of a combined elements array without copying"

That is what ChildBuilder does now, so both are dead weight — an extra path to keep correct for a result the generic path already produces.

What changes are included in this PR?

DType::Struct and DType::FixedSizeList fall through to the builder, and both helpers are deleted. Nothing has to be configured for this to work: since #8964 a nested builder keeps every appended chunk however short, which is the behaviour the swizzles hand-rolled. An earlier revision of this PR called builder.set_min_chunk_len(0) here to opt out of a length threshold that no longer exists.

I verified the shapes match before deleting anything. A tree_display probe over chunked struct / list / FSL inputs produces byte-identical trees on both paths for chunks of 200 rows — struct field is vortex.chunked(i32, len=400) with two chunks, FSL elements likewise. Two new rstest cases lock that in using chunks of one row.

What is not included

ListArray keeps swizzle_list_chunks. The builder produces the same shape — is_zero_copy_to_list: true and all — but the swizzle also allocates its offsets and sizes through ctx.allocator(), and builders are not allocator-aware: builder_with_capacity_in takes a HostAllocatorRef and discards it (let _allocator = allocator;). list_canonicalize_uses_memory_session_allocator catches exactly this. Making it work means threading an allocator through BufferMut in vortex-buffer, which has none today — a bigger and separate piece of work than this stack, so the list swizzle stays until someone does it.

pack_variant_chunks stays too: builder_with_capacity is unimplemented!() for DType::Variant.

StructArray::try_concat is now unused in-tree. It is pub, and a useful helper in its own right, so I left it — removing public API seemed like a separate call for a reviewer to make rather than something to slip into this PR.

What APIs are changed? Are there any user-facing changes?

No API changes, and no observable difference: the generic builder path preserves every chunk boundary, which is exactly what the swizzles did.

Checks

  • cargo nextest run --workspace — 7168 passed, 560 skipped
  • cargo clippy --all-targets --all-features — clean
  • cargo +nightly fmt --all --check — clean

Generated by Claude Code

Stacked PR Chain: builders-child-stack

PR Title Merges Into
#9046 perf(array): stop flattening appended list views in ListViewBuilder [builders-child-stack] N/A
#8964 feat(array): builders no longer canonicalize their children [builders-child-stack] #9046
#8965 feat(array): let callers choose a nested builder's chunk threshold [builders-child-stack] #8964
#8966 perf(array): accumulate nested builder validity without a null buffer [builders-child-stack] #8965
#8967 refactor(array): canonicalize chunked structs and FSLs through the builder [builders-child-stack] #8966

Generated by Claude Code

@codspeed-hq

codspeed-hq Bot commented Jul 25, 2026

Copy link
Copy Markdown

Merging this PR will degrade performance by 31.27%

❌ 12 regressed benchmarks
✅ 1829 untouched benchmarks
⏩ 55 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation canonicalize_sparse_list[(512, 7, 4)] 2.2 ms 3.6 ms -39.73%
Simulation canonicalize[1024, 32] 23.5 µs 37.8 µs -37.8%
Simulation canonicalize_sparse_list[(1024, 17, 8)] 1.9 ms 3 ms -37.49%
Simulation canonicalize[256, 32] 23.6 µs 37.4 µs -37.02%
Simulation canonicalize[16, 32] 24.5 µs 37.6 µs -34.92%
Simulation canonicalize[1024, 8] 18.8 µs 28.4 µs -33.73%
Simulation canonicalize[256, 8] 18.8 µs 28 µs -32.79%
Simulation canonicalize[1024, 2] 17.9 µs 25.5 µs -30.02%
Simulation canonicalize[16, 8] 20.3 µs 27.9 µs -27.48%
Simulation canonicalize[256, 2] 18 µs 24.5 µs -26.56%
Simulation canonicalize[16, 2] 23.2 µs 29.2 µs -20.46%
Simulation canonicalize_sparse_fixed_size_list[(512, 7, 4)] 622.2 µs 709.9 µs -12.35%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing claude/chunked-canonical-via-builder-9ze0t6 (8f39b06) with claude/builders-lazy-validity-9ze0t6 (8561525)2

Open in CodSpeed

Footnotes

  1. 55 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

  2. No successful run was found on claude/builders-lazy-validity-9ze0t6 (a8ebf30) during the generation of this report, so 9ef739b was used instead as the comparison base. There might be some changes unrelated to this pull request in this report.

@claude claude Bot changed the title refactor(array): canonicalize chunked structs and FSLs through the builder refactor(array): canonicalize chunked structs and FSLs through the builder [builders-child-stack] Jul 25, 2026
@robert3005
robert3005 force-pushed the claude/chunked-canonical-via-builder-9ze0t6 branch 2 times, most recently from afed52f to 79f350a Compare July 29, 2026 07:35
@robert3005
robert3005 force-pushed the claude/chunked-canonical-via-builder-9ze0t6 branch from 79f350a to 1145029 Compare July 29, 2026 13:48
…ilder

`pack_struct_chunks` and `swizzle_fixed_size_list_chunks` existed because
`append_to_builder` used to decode a builder's children: without them,
canonicalizing a `ChunkedArray` would concatenate every chunk's children
instead of reusing them. Their doc comments describe what the builders now
do on their own, so both are dead weight.

Route both dtypes through the generic builder path. `ListArray` keeps its
swizzle for now — that one also allocates its offsets and sizes through the
session allocator, which the builders cannot yet do.

Signed-off-by: Claude <noreply@anthropic.com>
Signed-off-by: Robert Kruszewski <robert@spiraldb.com>

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@robert3005
robert3005 force-pushed the claude/chunked-canonical-via-builder-9ze0t6 branch from 1145029 to 8f39b06 Compare July 29, 2026 14:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants