[core] Refactor sort compact to produce COMPACT commits by hbgstc123 · Pull Request #8631 · apache/paimon

hbgstc123 · 2026-07-15T01:54:44Z

Purpose

Follow up on #7595 per this review comment: sort compact should not be modeled as an OVERWRITE commit with extra base-snapshot handling. Instead, it should produce a normal COMPACT commit so the existing commit protocol can perform compact validation and conflict detection.

This PR is core-only, split from the previous ~4.5k-line change per review feedback. Flink/Spark wiring is in a stacked follow-up.

For append tables, this PR adds:

SortCompactCommitMessageRewriter to rewrite write-only append commit messages into CompactIncrement messages.
Base-snapshot deletion-vector metadata in SortCompactPlanMetadata, so later engine commit recovery can still clean up DVs if the base snapshot has expired.
Concurrent DV drift detection at rewrite time, plus compact conflict messages, so concurrent DV writes cannot restore deleted rows.
Abort helpers for written data files and newly persisted DV index files after rewrite/commit failure.

No Flink/Spark topology changes are included here.

Tests

SortCompactCommitMessageRewriterTest
- rewrite to compact messages
- multi-bucket / cross-bucket rewrite
- deletion-vector cleanup, including concurrent DV writes and expired base snapshots
- captured plan metadata serialization / round trip
- compact commit success detection, including expired snapshot gaps and concurrent delete-only compact
- inline compaction rejection
ConflictDetectionTest for compact + DV conflict messages

Merge latest-snapshot deletion-vector metadata at rewrite time so concurrent DV writes are cleaned up, while keeping captured plan metadata for expired snapshots. Align Flink commit failure behavior with Spark by aborting write output when the COMPACT snapshot is not yet visible. Co-authored-by: Cursor <cursoragent@cursor.com>

…sed by subclasses Commit 0f5aecf changed commitUser, state, and write from protected to private, but GlobalFullCompactionSinkWrite and LookupSinkWrite still access them directly. This causes JDK 8 compilation failures. Restore protected visibility to fix the build. Co-authored-by: Cursor <cursoragent@cursor.com>

…pact restricts to append tables SortCompactAction now explicitly rejects primary-key tables and only supports bucket-unaware append tables. The dynamic-bucket test case exercises unsupported behavior and fails on Flink 2.x CI, so remove it. Co-authored-by: Cursor <cursoragent@cursor.com>

Sort compact now commits as COMPACT instead of OVERWRITE. Concurrent merge and sort compact on V2 delta row-level DV append tables can leave duplicate visible rows, so skip that combination and keep coverage on the V1 write path. Co-authored-by: Cursor <cursoragent@cursor.com>

JingsongLi · 2026-07-16T05:46:03Z

There is a data correctness issue with “Sort compact” and concurrent DV updates. Since “Sort compact” generates output based on an old snapshot but deletes the latest DV from the input file upon commit, this can cause deleted data to be restored.

hbgstc123 · 2026-07-17T05:00:39Z

There is a data correctness issue with “Sort compact” and concurrent DV updates. Since “Sort compact” generates output based on an old snapshot but deletes the latest DV from the input file upon commit, this can cause deleted data to be restored.

Right, just pushed a fix

…pend_only

Resolve conflicts in FlinkSink and FlinkSinkBuilder by keeping both sort-compact writeProviderOverride/createAppendTableSink hooks and master's BLOB descriptor reader factory wiring. Co-authored-by: Cursor <cursoragent@cursor.com>

JingsongLi · 2026-07-23T13:09:52Z

+        return this;
+    }
+
+    private void validateSortCompactInput(List<DataSplit> compactInputSplits) {


Remove this method and two options. We don't need to restrict this.

Drop validateSortCompactInput and the warn/max input file options per review feedback. Co-authored-by: Cursor <cursoragent@cursor.com>

…ct_append_only Resolve conflict in BaseAppendDeleteFileMaintainer by keeping upstream javadoc. Co-authored-by: Cursor <cursoragent@cursor.com>

JingsongLi · 2026-07-26T11:49:16Z

dvMaintainer.persist() has already written a new DV file during the rewrite phase, but Flink/Spark’s failure cleanup only calls abort on the original writtenMessages; the new DV exists only in the rewritten compact messages and will not be cleaned up. During a delete-only compact, writtenMessages is even empty, and cleanup returns immediately. Retry attempts following conflicts or commit failures will continue to generate orphaned index files.

Co-authored-by: Cursor <cursoragent@cursor.com>

hbgstc123 · 2026-07-27T07:49:20Z

dvMaintainer.persist() has already written a new DV file during the rewrite phase, but Flink/Spark’s failure cleanup only calls abort on the original writtenMessages; the new DV exists only in the rewritten compact messages and will not be cleaned up. During a delete-only compact, writtenMessages is even empty, and cleanup returns immediately. Retry attempts following conflicts or commit failures will continue to generate orphaned index files.

done

JingsongLi · 2026-07-28T13:48:29Z


    DataFileMeta assignSequenceNumber(long minSequenceNumber, long maxSequenceNumber);

+    default DataFileMeta assignFileSource(FileSource fileSource) {


Why add this method to interface default?

DataFileMeta is @public, so a default method avoids breaking external implementations (same pattern as Table#newHybridSearchBuilder). However the only implementation in Paimon project is PojoDataFileMeta, and it has a real override. Happy to make it abstract

…append_only

After making DataFileMeta#assignFileSource abstract, BinaryDataFileMeta from master must provide the method explicitly.

JingsongLi · 2026-08-14T05:27:42Z

During a "delete-only sort compact" operation, if the commit fails but another compaction process has already deleted the same input files, isBatchCompactCommitSucceeded incorrectly determines that the current operation succeeded. Consequently, it skips the abort step, resulting in a leak of the DV index file generated during the operation. Minimal test case for stable reproduction: expected false, actual true.

… compact Matching success by removed input files can skip abort after a failed delete-only commit when another compact already deleted the same inputs, leaking the DV index files produced by rewrite.

# Conflicts: # paimon-core/src/main/java/org/apache/paimon/io/DataFileMeta.java # paimon-core/src/main/java/org/apache/paimon/io/ProjectedDataFileMeta.java # paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/action/SortCompactAction.java # paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/sink/FlinkSink.java # paimon-flink/paimon-flink-common/src/test/java/org/apache/paimon/flink/source/TestChangelogDataReadWrite.java # paimon-spark/paimon-spark-common/src/main/java/org/apache/paimon/spark/procedure/CompactProcedure.java # paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/commands/PaimonSparkWriter.scala

…rt_compact_append_only

… refactors from master

JingsongLi · 2026-09-10T08:09:17Z

Given that this involves another 4,000 lines of changes, could you split the PR to keep the line count down?

…ct_append_only

Move Flink/Spark wiring to a follow-up PR so the COMPACT-commit protocol can be reviewed independently.

hbgstc123 · 2026-09-13T12:59:03Z

Split per the review comment to keep the line count down.

This PR is now core-only (SortCompactCommitMessageRewriter / plan metadata / compact DV conflict handling / unit tests): about +2.3k lines, 9 files.
Flink + Spark wiring is in stacked follow-up [flink][spark] Wire sort compact to produce COMPACT commits #9790. Engine-only diff: hbgstc123/paimon@fix_sort_compact_append_only...fix_sort_compact_append_engines

hbgstc123 mentioned this pull request Jul 15, 2026

[core] Refactor sort compact to produce COMPACT commits #8571

Closed

hbgstc123 force-pushed the fix_sort_compact_append_only branch from c8a1b81 to de2e634 Compare July 15, 2026 08:04

hbg and others added 5 commits July 16, 2026 09:58

[core] Refactor sort compact to produce COMPACT commits

6d4da26

hbgstc123 force-pushed the fix_sort_compact_append_only branch from de2e634 to d462567 Compare July 16, 2026 02:01

address comment, detect concurrent dv conflict

92e4afa

hbg and others added 3 commits July 17, 2026 14:02

Merge remote-tracking branch 'apache/master' into fix_sort_compact_ap…

dcd3ba7

…pend_only

report correct error message in ConflictDetection

13ba744

Merge branch 'master' into fix_sort_compact_append_only

66235f7

Resolve conflicts in FlinkSink and FlinkSinkBuilder by keeping both sort-compact writeProviderOverride/createAppendTableSink hooks and master's BLOB descriptor reader factory wiring. Co-authored-by: Cursor <cursoragent@cursor.com>

JingsongLi reviewed Jul 23, 2026

View reviewed changes

hbg and others added 2 commits July 24, 2026 00:49

[flink] Remove sort compact input file count limits

626bb3f

Drop validateSortCompactInput and the warn/max input file options per review feedback. Co-authored-by: Cursor <cursoragent@cursor.com>

Merge branch 'master' of github.com:apache/paimon into fix_sort_compa…

0ab8ac0

…ct_append_only Resolve conflict in BaseAppendDeleteFileMaintainer by keeping upstream javadoc. Co-authored-by: Cursor <cursoragent@cursor.com>

hbg and others added 2 commits July 27, 2026 12:06

delete new dv index files when error thrown in rewrite.

1423d4f

Retrigger CI for Flink 1.x Others workflow.

ffc40c2

Co-authored-by: Cursor <cursoragent@cursor.com>

JingsongLi reviewed Jul 28, 2026

View reviewed changes

hbg added 4 commits July 29, 2026 10:22

address comment, make DataFileMeta#assignFileSource abstract

5a39d9c

Merge remote-tracking branch 'upstream/master' into fix_sort_compact_…

0231843

…append_only

[core] Implement assignFileSource for BinaryDataFileMeta

5830cab

After making DataFileMeta#assignFileSource abstract, BinaryDataFileMeta from master must provide the method explicitly.

Trigger CI re-run.

604c6ab

hbg added 3 commits September 8, 2026 22:48

[core] Fix delete-only sort compact success detection with concurrent…

76878c4

… compact Matching success by removed input files can skip abort after a failed delete-only commit when another compact already deleted the same inputs, leaking the DV index files produced by rewrite.

Merge remote-tracking branch 'refs/remotes/apache/master' into fix_so…

68231c2

…rt_compact_append_only

[core] Fix sort compact tests for SchemaManager and ConflictDetection…

68290a5

… refactors from master

hbg added 2 commits September 13, 2026 20:54

Merge branch 'master' of github.com:apache/paimon into fix_sort_compa…

51b6fe3

…ct_append_only

[core] Split sort compact PR: keep core rewriter only

5b5dc50

Move Flink/Spark wiring to a follow-up PR so the COMPACT-commit protocol can be reviewed independently.

hbgstc123 mentioned this pull request Sep 13, 2026

[flink][spark] Wire sort compact to produce COMPACT commits #9790

Open

Navigation Menu

Uh oh!

[core] Refactor sort compact to produce COMPACT commits - #8631

[core] Refactor sort compact to produce COMPACT commits#8631
hbgstc123 wants to merge 23 commits into
apache:masterfrom
hbgstc123:fix_sort_compact_append_only

hbgstc123 commented Jul 15, 2026 •

edited

Loading

Uh oh!

JingsongLi commented Jul 16, 2026

Uh oh!

hbgstc123 commented Jul 17, 2026

Uh oh!

JingsongLi Jul 23, 2026

Uh oh!

hbgstc123 Jul 23, 2026

Uh oh!

JingsongLi commented Jul 26, 2026

Uh oh!

hbgstc123 commented Jul 27, 2026

Uh oh!

JingsongLi Jul 28, 2026

Uh oh!

hbgstc123 Jul 29, 2026

Uh oh!

hbgstc123 Jul 29, 2026

Uh oh!

JingsongLi commented Aug 14, 2026

Uh oh!

JingsongLi commented Sep 10, 2026

Uh oh!

hbgstc123 commented Sep 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants


		DataFileMeta assignSequenceNumber(long minSequenceNumber, long maxSequenceNumber);

		default DataFileMeta assignFileSource(FileSource fileSource) {

Uh oh!

Conversation

hbgstc123 commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Tests

Uh oh!

JingsongLi commented Jul 16, 2026

Uh oh!

hbgstc123 commented Jul 17, 2026

Uh oh!

JingsongLi Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

hbgstc123 Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

JingsongLi commented Jul 26, 2026

Uh oh!

hbgstc123 commented Jul 27, 2026

Uh oh!

JingsongLi Jul 28, 2026

Choose a reason for hiding this comment

Uh oh!

hbgstc123 Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

hbgstc123 Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

JingsongLi commented Aug 14, 2026

Uh oh!

JingsongLi commented Sep 10, 2026

Uh oh!

hbgstc123 commented Sep 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

hbgstc123 commented Jul 15, 2026 •

edited

Loading