[flink][spark] Wire sort compact to produce COMPACT commits - #9790
Open
hbgstc123 wants to merge 24 commits into
Open
[flink][spark] Wire sort compact to produce COMPACT commits#9790hbgstc123 wants to merge 24 commits into
hbgstc123 wants to merge 24 commits into
Conversation
Merge latest-snapshot deletion-vector metadata at rewrite time so concurrent DV writes are cleaned up, while keeping captured plan metadata for expired snapshots. Align Flink commit failure behavior with Spark by aborting write output when the COMPACT snapshot is not yet visible. Co-authored-by: Cursor <cursoragent@cursor.com>
…sed by subclasses Commit 0f5aecf changed commitUser, state, and write from protected to private, but GlobalFullCompactionSinkWrite and LookupSinkWrite still access them directly. This causes JDK 8 compilation failures. Restore protected visibility to fix the build. Co-authored-by: Cursor <cursoragent@cursor.com>
…pact restricts to append tables SortCompactAction now explicitly rejects primary-key tables and only supports bucket-unaware append tables. The dynamic-bucket test case exercises unsupported behavior and fails on Flink 2.x CI, so remove it. Co-authored-by: Cursor <cursoragent@cursor.com>
Sort compact now commits as COMPACT instead of OVERWRITE. Concurrent merge and sort compact on V2 delta row-level DV append tables can leave duplicate visible rows, so skip that combination and keep coverage on the V1 write path. Co-authored-by: Cursor <cursoragent@cursor.com>
Resolve conflicts in FlinkSink and FlinkSinkBuilder by keeping both sort-compact writeProviderOverride/createAppendTableSink hooks and master's BLOB descriptor reader factory wiring. Co-authored-by: Cursor <cursoragent@cursor.com>
Drop validateSortCompactInput and the warn/max input file options per review feedback. Co-authored-by: Cursor <cursoragent@cursor.com>
…ct_append_only Resolve conflict in BaseAppendDeleteFileMaintainer by keeping upstream javadoc. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
After making DataFileMeta#assignFileSource abstract, BinaryDataFileMeta from master must provide the method explicitly.
… compact Matching success by removed input files can skip abort after a failed delete-only commit when another compact already deleted the same inputs, leaking the DV index files produced by rewrite.
# Conflicts: # paimon-core/src/main/java/org/apache/paimon/io/DataFileMeta.java # paimon-core/src/main/java/org/apache/paimon/io/ProjectedDataFileMeta.java # paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/action/SortCompactAction.java # paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/sink/FlinkSink.java # paimon-flink/paimon-flink-common/src/test/java/org/apache/paimon/flink/source/TestChangelogDataReadWrite.java # paimon-spark/paimon-spark-common/src/main/java/org/apache/paimon/spark/procedure/CompactProcedure.java # paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/commands/PaimonSparkWriter.scala
…rt_compact_append_only
… refactors from master
Move Flink/Spark wiring to a follow-up PR so the COMPACT-commit protocol can be reviewed independently.
Use the core rewriter from the stacked core PR so Flink and Spark sort compact produce COMPACT snapshots instead of OVERWRITE.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Stacked follow-up to #8631. Please review this PR after (or together with) the core rewriter.
Until #8631 merges, GitHub's Files changed view includes the core commits. Engine-only diff:
hbgstc123/paimon@fix_sort_compact_append_only...fix_sort_compact_append_engines
This PR wires Flink and Spark sort compact for append tables to the core rewriter:
CompactIncrementmessages.Additional engine hardening:
SortCompactCommittermerge/rewrite/abort and commit recovery.SortCompactActionForDynamicBucketITCasebecause sort compact is append-table only.No new
overwritePartition(baseSnapshotId)APIs or special overwrite conflict logic are introduced.Tests
Flink
SortCompactCommitterTestSortCompactActionForAppendTableITCaseCommitterOperatorTestCompactProcedureITCaseupdated to expect COMPACTSpark
SortCompactSparkCommitTestCompactProcedureTestBaseupdated to expect COMPACT