Skip to content

Split SplitBy::Layout into Layout and LayoutSubSplitting variants - #9043

Open
robert3005 wants to merge 1 commit into
developfrom
claude/pr-8400-splitting-config-ezc21l
Open

Split SplitBy::Layout into Layout and LayoutSubSplitting variants#9043
robert3005 wants to merge 1 commit into
developfrom
claude/pr-8400-splitting-config-ezc21l

Conversation

@robert3005

Copy link
Copy Markdown
Contributor

Rationale for this change

This change separates the concerns of chunk-boundary-aware splitting from automatic sub-division of wide chunk spans. Previously, SplitBy::Layout combined both behaviors: it would split at chunk boundaries and also sub-divide spans wider than MAX_SPLIT_ROWS to enable intra-file decode parallelism.

The new design introduces two distinct variants:

  • SplitBy::Layout: Splits only at chunk boundaries, yielding fewer, larger batches that follow the file's chunking exactly
  • SplitBy::LayoutSubSplitting: The previous default behavior—splits at chunk boundaries and sub-divides wide spans for parallelism

This makes the splitting strategy more explicit and allows users to choose between fine-grained parallelism (default) or coarser-grained batching that respects the file's natural structure.

What changes are included in this PR?

Core changes:

  • Split SplitBy::Layout into two variants in vortex-layout/src/scan/split_by.rs
  • Updated SplitBy::Layout to only split at chunk boundaries without sub-division
  • Made SplitBy::LayoutSubSplitting the new default (with #[default])
  • Modified the split logic to conditionally apply subdivide_large_spans() only for LayoutSubSplitting

API additions:

  • Added ScanBuilder::with_no_sub_splitting() convenience method in vortex-layout/src/scan/scan_builder.rs to explicitly opt into SplitBy::Layout

Test improvements:

  • Refactored test infrastructure: extracted a reusable StubReader struct to replace the inline DupReader
  • Added test_layout_keeps_large_chunk_whole() to verify that SplitBy::Layout keeps wide chunks intact while SplitBy::LayoutSubSplitting sub-divides them
  • Updated existing tests to use the new variants appropriately

Integration updates:

  • Updated vortex-file/src/file.rs to use LayoutSubSplitting for the default splits() method
  • Updated vortex-datafusion/src/persistent/opener.rs to use LayoutSubSplitting for natural split ranges
  • Updated vortex-python/src/dataset.rs to default to LayoutSubSplitting
  • Updated documentation comments to reflect the new variant names

What APIs are changed? Are there any user-facing changes?

Public API changes:

  • SplitBy enum now has two layout-related variants instead of one
  • ScanBuilder::with_no_sub_splitting() is a new convenience method for opting into SplitBy::Layout
  • Default scan behavior remains unchanged (still uses sub-splitting), but is now explicitly SplitBy::LayoutSubSplitting

User-facing impact:

  • Users who want to disable sub-splitting can now call .with_no_sub_splitting() on a scan builder
  • The default behavior is unchanged—scans still sub-divide wide chunks for parallelism
  • File splits reported by VortexFile::splits() now reflect sub-divided boundaries (matching the new default)

https://claude.ai/code/session_01M9d52UfuZRn3cC9eixruzx

@robert3005 robert3005 added the changelog/feature A new feature label Jul 29, 2026
Sub-division of chunk-boundary spans wider than the ideal split size (so a file
with few, large chunks decodes across multiple cores) is now a scan config
option rather than unconditional behaviour.

`SplitBy::Layout` now splits only at the file's own chunk boundaries, and the
new `SplitBy::LayoutSubSplitting` variant — the default — additionally
sub-divides wide spans. `ScanBuilder::with_no_sub_splitting()` is a shorthand
for selecting `SplitBy::Layout`. Callers that relied on the sub-dividing
default (`VortexFile::splits`, the DataFusion repartitioner, Python datasets)
now name `LayoutSubSplitting` explicitly, so their behaviour is unchanged.

Signed-off-by: Robert Kruszewski <robert@spiraldb.com>

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M9d52UfuZRn3cC9eixruzx
@robert3005
robert3005 force-pushed the claude/pr-8400-splitting-config-ezc21l branch from 94e910a to bdd36d1 Compare July 30, 2026 14:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant