Skip to content

[core] Support bucket-first sorting for manifest files - #9792

Draft
jianguotian wants to merge 1 commit into
apache:masterfrom
jianguotian:perf/manifest-bucket-first-sort
Draft

[core] Support bucket-first sorting for manifest files#9792
jianguotian wants to merge 1 commit into
apache:masterfrom
jianguotian:perf/manifest-bucket-first-sort

Conversation

@jianguotian

Copy link
Copy Markdown
Contributor

Purpose

This is the first layout follow-up split from #9784 after review feedback. #9791 has already merged the independent manifest bucket-pruning change. This PR contains only opt-in bucket-first manifest sorting.

Changes

  • Add manifest-sort.bucket-first (default: false).
  • Sort manifest entries by bucket before the configured partition field when enabled.
  • Preserve the existing entry order exactly when the option is disabled.
  • Compare manifest runs by bucket only when every input manifest has bucket bounds; otherwise conservatively use the existing partition-only comparison for the whole pass.
  • Use the same sort key in manifest compaction dry-run.
  • Reject the option for data-evolution / RowID sorting.

Forced manifest rewrites, pruning metadata/filter logic, mixed-count pruning, and other optimizations are intentionally excluded.

Performance

A prior combined experiment reduced scanned manifests from 136 to 1 and median planning time from 10.75 s to 1.50 s for the customer point lookup. That experiment also included forced rewriting and was not based on this clean commit, so it is motivation only and is not claimed as evidence for this PR.

Before marking this PR ready, I will add isolated results comparing the #9791 baseline with #9791 plus this exact commit on the same table, snapshot, predicate, and runtime conditions.

Verification

  • Focused manifest sorting and schema validation tests: 150 run, 0 failures, 0 errors, 1 skipped.
  • paimon-core reactor suite: 5,361 run, 0 failures, 1 environment error, 34 skipped. The only error is PostgresqlCatalogTest because Docker is unavailable locally.
  • Non-fast validation: Checkstyle, Spotless, and Maven Enforcer passed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant