[spark] Add paimon spark4.2 module - #9265
Open
LuciferYang wants to merge 7 commits into
Open
Conversation
Contributor
Author
|
Let me resolve the conflicts. |
Raises the `spark4` profile baseline from 4.1.2 to 4.2.0, adds a `paimon-spark-4.2` module, and keeps Spark 4.0 and 4.1 working under the new baseline. Same shape as apache#7648, which did this for 4.1. ## Why the bump breaks the older modules `paimon-spark-common` and `paimon-spark4-common` are compiled once, against the newest supported Spark, and the resulting classfiles ship to every 4.x runtime. Raising the baseline therefore changes bytecode that 4.0 and 4.1 have to load, and Spark 4.2 made several source-compatible but binary-incompatible changes: - `CatalogManager` became an interface, so a 4.2-built call site emits `invokeinterface` and dies with `IncompatibleClassChangeError` on 4.0/4.1. - Case classes gained fields (`CatalogStorageFormat`, `AppendData`, `DataSourceV2ScanRelation`), so positional patterns and named-argument `copy` calls no longer compile or link across versions. - `RewriteRowLevelCommand`'s `DELTA_OPERATIONS_WITH_*` constants were renamed, and `V2WriteCommand` gained a `WriteWithSchemaEvolution` supertype. - `DESCRIBE ... PARTITION` moved out of `DescribeRelation` into its own `DescribeTablePartition` plan (SPARK-39660). - SPARK-57058 folded the geo value classes into `BinaryView`: `SpecializedGetters` lost `getGeometry` / `getGeography` in favour of `getBinaryView`, `GeometryVal` / `GeographyVal` were removed, and `STUtils.stAsBinary` / `stSetSrid` were split into `stGeomAsBinary` / `stGeogAsBinary` and `stGeomSetSrid` / `stGeogSetSrid`. ## Four mechanisms, in order of preference 1. **Version-neutral construction.** Match by type with named accessors instead of positional patterns; build placeholders through factory methods (`CatalogStorageFormat.empty`) rather than arity-sensitive constructors. 2. **`SparkShim` methods** where only the arity differs, so each per-version module supplies its own call. 3. **`SparkVersionCompat`** reflective accessors where the *signature* is incompatible. Reflection is immune to the class/interface flip: only invoke opcodes carry that distinction. 4. **Same-FQCN forks** in `paimon-spark-4.0` / `-4.1` where a supertype or a parameter type differs and no accessor can paper over it. Shade writes the module's own classes before the ones it pulls in from `paimon-spark4-common`, so the fork wins. The geospatial support added by apache#9251 needs mechanism 4. `paimon-spark4-common`'s `Spark4ArrayData`, `Spark4InternalRow` and `Spark4Shim` now implement the 4.2 shape (`getBinaryView`, `stGeomAsBinary` / `stGeogAsBinary`, `stGeogSetSrid`), and `paimon-spark-4.1` forks all three to keep the pre-4.2 pair of overrides, which 4.1 still declares abstract. `paimon-spark-4.0` already forked the two data classes for the same reason -- 4.0 has no geo types at all. `paimon-spark-ut-4.0` and `-4.1` recompile the shared test sources against their own baseline; they produce test-jars only and are deliberately left out of publish and release, unlike `paimon-spark-ut`. ## Also fixed here `qualifyIdentifier` has to carry the catalog so Spark 4.2's `SimpleFunctionRegistryBase.normalizeFuncName` sees a 3-part identifier. The same identifier reached the expression builder, which renamed the default output column of an unaliased v1 function call from `db.udf(...)` to `catalog.db.udf(...)` on every version from 3.4 up. The builder name now drops the catalog; the registry key keeps it. The new test asserts the column name -- the existing cases all compare rows with `checkAnswer` and spell out `AS` wherever a name is involved, so none of them could see it. ## Verification `mvn -Pspark4 clean install` over `paimon-spark-common`, `paimon-spark4-common`, the three ut modules and `paimon-spark-4.0` / `-4.1` / `-4.2`: success. `spotless:check` clean on all six touched modules. Targeted suites: `PaimonV1FunctionTest` 13/13 on 4.2, 4.1 and 4.0; `DescribeTableTest` 4/4 on 4.2, 4.1 and 4.0; `SparkVersionCompatTest` 14/14. For the geospatial path, `GeospatialTypeSQLTest` 2/2 and `GeospatialTypeTest` 2/2 on 4.1 (the version that exercises the forked pre-4.2 overrides) and `GeospatialUnsupportedTest` 1/1 on 4.0. The column-name fix was verified in both directions -- reverting it turns the new case red with `ArraySeq("paimon.test.udf_add2(3, 4)") did not equal List("test.udf_add2(3, 4)")`. The full per-module suites have not been run in this branch yet.
LuciferYang
force-pushed
the
spark-4.2-pr1
branch
from
August 17, 2026 06:49
91640c1 to
f37fb20
Compare
All three come from Spark 4.2's storage-partitioned-join rework (SPARK-55535). `EnsureRequirements` now wraps a bucketed scan in `GroupPartitionsExec`, which casts `child.outputPartitioning` to `Partitioning with Expression`. Our AQE prep rule runs after it and disabled the bucketing underneath, which degrades that to `UnknownPartitioning`, so the cast threw at execution time. The traversal now stops at such a node and leaves its whole subtree alone: `EnsureRequirements` wraps whatever satisfied the distribution, which can be a join or an aggregate with several scans below it, and disabling only some of them would instead trip `PartitioningCollection`'s equal-`numPartitions` requirement. `DataSourceRDDPartition`'s payload changed from `Seq[InputPartition]` to `Option[InputPartition]`. `PaimonSparkMicroBatchMetadata` reads that field reflectively and rejected the new shape, and the blanket `catch NonFatal` around it reported that as "metadata absent" instead of failing. It now accepts every shape the field has had: a bare `InputPartition` on 3.2, a `Seq` from 3.3 to 4.1, and an `Option` since 4.2. The accessor is also resolved once per RDD rather than once per partition, since a miss costs a thrown `NoSuchMethodException`. `CREATE TABLE LIKE` with a four-part name is parsed by Spark itself on 4.2 and by Paimon's extension parser below it, so the middle parts arrive split differently. The test now asserts the invariant that holds on every version, which is that the parts survive in order, rather than pinning one version's split.
…LE assertions Spark 4.2 (SPARK-55372) makes `SHOW CREATE TABLE` print the collation on every string-ish column even when the column has none, so that replaying the emitted DDL cannot silently pick up a table- or schema-level `DEFAULT COLLATION` instead. A column declared `STRING` now renders as `STRING COLLATE UTF8_BINARY`, and so do VARCHAR, CHAR and the string leaves of ARRAY / MAP / STRUCT. This is not Paimon-specific: Spark's own golden file expects the same output for a parquet table, and Paimon's string columns reach it through the ordinary `StringType` case object, which Spark reads as "no explicit collation". The 31 failing assertions were all comparing against the pre-4.2 rendering. These tests are compiled once and run against every supported Spark version, so they now route `SHOW CREATE TABLE` through a helper that removes the marker rather than branching on the version. Only ` COLLATE UTF8_BINARY` is removed, so a column carrying a real collation such as `STRING COLLATE UTF8_LCASE` still appears and an assertion cannot be fooled into accepting the wrong collation.
Contributor
Author
|
cc @melin FYI |
Contributor
|
cc @Zouxxyy to take a review. |
Contributor
Author
|
Thank you @JingsongLi |
Contributor
|
cc @Zouxxyy |
JingsongLi
reviewed
Sep 10, 2026
JingsongLi
left a comment
JingsongLi
left a comment
Contributor
There was a problem hiding this comment.
The Spark 4.2 support has concrete user-facing value (#8901), and the compatibility shims/forks address real binary differences in the shared Spark modules. I reviewed the main compatibility paths, old-version test-jar wiring and release inclusion; I did not find an evidence-backed blocking defect in that scope.
The existing Spark 4.x CI run (https://github.com/apache/paimon/actions/runs/32106655843/job/95617365797) actually executes SQL/storage tests across 4.0, 4.1 and 4.2, including reads, writes and schema evolution. Please resolve the current conflicts and rerun that matrix on the resulting head, including dynamic partition overwrite, MERGE, SQL functions and DESCRIBE PARTITION. I did not independently build the full Spark matrix or exhaustively audit every copied class; this is not a merge approval.
Resolve doc conflicts by taking master's restructured Compatibility Matrix and Spark quick-start, and carry the Spark 4.2 entries into the relocated spark/installation.mdx (version list, download tables, build command) and the ecosystem Spark row.
paimon-spark-4.2, paimon-spark-ut-4.0 and paimon-spark-ut-4.1 are new in this PR, so master's 2.2-SNAPSHOT version bump could not reach them and the merge left their paimon-spark parent pinned at 2.1-SNAPSHOT. The reactor then failed to resolve paimon-spark:pom:2.1-SNAPSHOT and could not read these three projects. Align them with the current 2.2-SNAPSHOT. Co-Authored-By: Claude Code <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Raises the
spark4profile baseline from 4.1.2 to 4.2.0, adds apaimon-spark-4.2module, and keeps Spark 4.0 and 4.1 working under the new baseline. Same shape as #7648, which did this for 4.1. Related to #8901.paimon-spark-commonandpaimon-spark4-commonare compiled once, against the newest supported Spark, and the resulting classfiles ship to every 4.x runtime. Raising the baseline therefore changes bytecode that 4.0 and 4.1 have to load, and Spark 4.2 made several source-compatible but binary-incompatible changes:CatalogManagerbecame an interface (a 4.2-built call site emitsinvokeinterfaceand dies withIncompatibleClassChangeErroron 4.0/4.1); case classes gained fields (CatalogStorageFormat,AppendData,DataSourceV2ScanRelation), so positional patterns and named-argumentcopycalls no longer link across versions;RewriteRowLevelCommand'sDELTA_OPERATIONS_WITH_*constants were renamed;V2WriteCommandgained aWriteWithSchemaEvolutionsupertype; andDESCRIBE ... PARTITIONmoved out ofDescribeRelationinto its ownDescribeTablePartitionplan (SPARK-39660).Four mechanisms are used, in order of preference:
CatalogStorageFormat.empty) rather than arity-sensitive constructors.SparkShimmethods where only the arity differs, so each per-version module supplies its own call.SparkVersionCompatreflective accessors where the signature is incompatible. Reflection is immune to the class/interface flip, since only invoke opcodes carry that distinction.paimon-spark-4.0/-4.1where a supertype or a parameter type differs and no accessor can paper over it. Shade writes the module's own classes first, so the fork wins.paimon-spark-ut-4.0and-4.1recompile the shared test sources against their own baseline. They produce test-jars only and are deliberately left out of publish and release, unlikepaimon-spark-ut.One behaviour fix is included because the baseline bump caused it.
qualifyIdentifierhas to carry the catalog so Spark 4.2'sSimpleFunctionRegistryBase.normalizeFuncNamesees a 3-part identifier, but the same identifier reached the expression builder, which renamed the default output column of an unaliased v1 function call fromdb.udf(...)tocatalog.db.udf(...)on every version from 3.4 up. The builder name now drops the catalog; the registry key keeps it.Tests
CI, plus locally:
mvn -Pspark4 clean installoverpaimon-spark-common,paimon-spark4-common,paimon-spark-4.0,-4.1,-4.2and the three ut modules — success.mvn -Pspark3 clean compileoverpaimon-spark-common+paimon-spark3-common— success.spotless:checkclean on all six touched modules.PaimonV1FunctionTest13/13 on 4.2, 4.1 and 4.0;DescribeTableTest4/4 on 4.2, 4.1 and 4.0;SparkVersionCompatTest14/14.ArraySeq("paimon.test.udf_add2(3, 4)") did not equal List("test.udf_add2(3, 4)").The new
PaimonV1FunctionTestBasecase asserts the column name rather than the row values, which is why the existing cases could not catch the regression: they all compare rows withcheckAnswerand spell outASwherever a name is involved.