Skip to content

[spark] Add paimon spark4.2 module #9265

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
LuciferYang wants to merge 11 commits into
base: master
Choose a base branch
from
Open

[spark] Add paimon spark4.2 module #9265

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
f37fb20
[spark] Add paimon spark4.2 module
LuciferYang Aug 13, 2026
a2bad68
[spark] Fix three test failures on the Spark 4.2 baseline
LuciferYang Aug 17, 2026
a7698f0
[spark] Strip the explicit UTF8_BINARY collation from SHOW CREATE TAB…
LuciferYang Aug 18, 2026
e4c3c30
Merge branch 'master' into spark-4.2-pr1 to rerun the Spark 4.x CI ma…
LuciferYang Sep 10, 2026
fd803d9
Merge branch 'master' into spark-4.2-pr1
LuciferYang Sep 11, 2026
e1a64e6
fix: bump spark 4.x module POMs to 2.2-SNAPSHOT
LuciferYang Sep 12, 2026
7df5522
Merge remote-tracking branch 'upstream/master' into HEAD
LuciferYang Sep 12, 2026
3aac2fb
docs: say what the bucketed-scan forks differ by and what the collati…
LuciferYang Sep 13, 2026
e9115b9
Merge remote-tracking branch 'upstream/master' into spark-4.2-pr1
LuciferYang Sep 13, 2026
e15c4dd
ci: run the 4.2 suite in the reusable Java workflow
LuciferYang Sep 13, 2026
b503e3f
fix: align the 4.1 shim with the SparkShim interface
LuciferYang Sep 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/publish-snapshot-spark4.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,8 +67,8 @@ jobs:
echo "<password>$ASF_PASSWORD</password>" >> $tmp_settings
echo "</server></servers></settings>" >> $tmp_settings

mvn --settings $tmp_settings -ntp clean install -Dgpg.skip -Drat.skip -DskipTests -Papache-release,spark4,flink1 -pl org.apache.paimon:paimon-spark-4.0_2.13,org.apache.paimon:paimon-spark-4.1_2.13 -am
mvn --settings $tmp_settings -ntp clean install -Dgpg.skip -Drat.skip -DskipTests -Papache-release,spark4,flink1 -pl org.apache.paimon:paimon-spark-4.0_2.13,org.apache.paimon:paimon-spark-4.1_2.13,org.apache.paimon:paimon-spark-4.2_2.13 -am
# skip deploy paimon-spark-common_2.13 since they are already deployed in publish-snapshot.yml
mvn --settings $tmp_settings -ntp clean deploy -Dgpg.skip -Drat.skip -DskipTests -Papache-release,spark4,flink1 -pl org.apache.paimon:paimon-spark4-common_2.13,org.apache.paimon:paimon-spark-ut_2.13,org.apache.paimon:paimon-spark-4.0_2.13,org.apache.paimon:paimon-spark-4.1_2.13
mvn --settings $tmp_settings -ntp clean deploy -Dgpg.skip -Drat.skip -DskipTests -Papache-release,spark4,flink1 -pl org.apache.paimon:paimon-spark4-common_2.13,org.apache.paimon:paimon-spark-ut_2.13,org.apache.paimon:paimon-spark-4.0_2.13,org.apache.paimon:paimon-spark-4.1_2.13,org.apache.paimon:paimon-spark-4.2_2.13

rm $tmp_settings
8 changes: 4 additions & 4 deletions .github/workflows/release-java.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -121,7 +121,7 @@ jobs:
mvn -ntp -B "${enforcer_goal}" \
-Denforcer.rules="${enforcer_rules}" \
-Papache-release,docs-and-source,spark4 \
-pl org.apache.paimon:paimon-spark-common_2.13,org.apache.paimon:paimon-spark4-common_2.13,org.apache.paimon:paimon-spark-4.0_2.13,org.apache.paimon:paimon-spark-4.1_2.13 \
-pl org.apache.paimon:paimon-spark-common_2.13,org.apache.paimon:paimon-spark4-common_2.13,org.apache.paimon:paimon-spark-4.0_2.13,org.apache.paimon:paimon-spark-4.1_2.13,org.apache.paimon:paimon-spark-4.2_2.13 \
-am
;;
*)
Expand Down Expand Up @@ -186,22 +186,22 @@ jobs:
jdk17)
capture_expected_projects \
-Papache-release,docs-and-source,spark4 \
-pl org.apache.paimon:paimon-spark-common_2.13,org.apache.paimon:paimon-spark4-common_2.13,org.apache.paimon:paimon-spark-4.0_2.13,org.apache.paimon:paimon-spark-4.1_2.13
-pl org.apache.paimon:paimon-spark-common_2.13,org.apache.paimon:paimon-spark4-common_2.13,org.apache.paimon:paimon-spark-4.0_2.13,org.apache.paimon:paimon-spark-4.1_2.13,org.apache.paimon:paimon-spark-4.2_2.13
# This install only supplies reactor dependencies for the deploy
# below. Their Javadocs are built by the owning release lane.
mvn clean install -ntp -B \
-Pdocs-and-source,spark4 \
-DskipTests -Dmaven.javadoc.skip=true \
-Dstyle.color=never \
-pl paimon-spark/paimon-spark-4.0,paimon-spark/paimon-spark-4.1 \
-pl paimon-spark/paimon-spark-4.0,paimon-spark/paimon-spark-4.1,paimon-spark/paimon-spark-4.2 \
-am \
2>&1 | tee "${log}"
mvn deploy -ntp -B \
-Papache-release,docs-and-source,spark4 \
-DskipTests -Dgpg.skip=true -Dstyle.color=never \
-DdeployAtEnd=true \
-DaltDeploymentRepository="${alt_repository}" \
-pl org.apache.paimon:paimon-spark-common_2.13,org.apache.paimon:paimon-spark4-common_2.13,org.apache.paimon:paimon-spark-4.0_2.13,org.apache.paimon:paimon-spark-4.1_2.13 \
-pl org.apache.paimon:paimon-spark-common_2.13,org.apache.paimon:paimon-spark4-common_2.13,org.apache.paimon:paimon-spark-4.0_2.13,org.apache.paimon:paimon-spark-4.1_2.13,org.apache.paimon:paimon-spark-4.2_2.13 \
2>&1 | tee -a "${log}"
;;
*)
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/ecosystem/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ release cycle, embedded Paimon version, and feature limits.
| Integration | Version selection | Access to Paimon tables |
| --- | --- | --- |
| [Flink](../flink/installation) | 1.16–1.20 and 2.0–2.2 | Batch and streaming reads/writes; [DDL](../flink/sql-ddl) and [row changes](../flink/sql-write) have version-specific requirements. |
| [Spark](../spark/quick-start) | 3.2–3.5, 4.0, and 4.1; match the Scala binary version | Batch reads/writes, [DDL](../spark/sql-ddl), and [row changes](../spark/sql-write); [streaming](../spark/structured-streaming) requires Spark 3.3+. |
| [Spark](../spark/quick-start) | 3.2–3.5, 4.0, 4.1, and 4.2; match the Scala binary version | Batch reads/writes, [DDL](../spark/sql-ddl), and [row changes](../spark/sql-write); [streaming](../spark/structured-streaming) requires Spark 3.3+. |
| [Hive](./hive#version) | 2.1, 2.2, 2.3, 3.1, and 2.1-cdh-6.3 | Batch reads, table creation, and `INSERT INTO`; writes require MapReduce. |
| [Trino](./trino#version) | Match the independently released Paimon connector to Trino | Batch reads; supported connectors also provide DDL, inserts, and time travel. See the guide's table-layout limits. |
| [Presto](https://github.com/apache/paimon-presto) | Follow the separate connector's version requirements | See the connector repository for installation and supported operations. |
Expand Down
2 changes: 2 additions & 0 deletions docs/docs/project/download.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,7 @@ signed Apache release.
| Flink 1.17 | [paimon-flink-1.17-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-flink-1.17/@@VERSION@@/) |
| Flink 1.16 | [paimon-flink-1.16-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-flink-1.16/@@VERSION@@/) |
| Flink Action | [paimon-flink-action-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-flink-action/@@VERSION@@/) |
| Spark 4.2 | [paimon-spark-4.2_2.13-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-spark-4.2_2.13/@@VERSION@@/) |
| Spark 4.1 | [paimon-spark-4.1_2.13-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-spark-4.1_2.13/@@VERSION@@/) |
| Spark 4.0 | [paimon-spark-4.0_2.13-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-spark-4.0_2.13/@@VERSION@@/) |
| Spark 3.5 | [paimon-spark-3.5_2.12-@@VERSION@@.jar](https://repository.apache.org/snapshots/org/apache/paimon/paimon-spark-3.5_2.12/@@VERSION@@/) |
Expand Down Expand Up @@ -103,6 +104,7 @@ signed Apache release.
| Flink 1.17 | [paimon-flink-1.17-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-flink-1.17/@@VERSION@@/paimon-flink-1.17-@@VERSION@@.jar) |
| Flink 1.16 | [paimon-flink-1.16-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-flink-1.16/@@VERSION@@/paimon-flink-1.16-@@VERSION@@.jar) |
| Flink Action | [paimon-flink-action-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-flink-action/@@VERSION@@/paimon-flink-action-@@VERSION@@.jar) |
| Spark 4.2 | [paimon-spark-4.2_2.13-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-spark-4.2_2.13/@@VERSION@@/paimon-spark-4.2_2.13-@@VERSION@@.jar) |
| Spark 4.1 | [paimon-spark-4.1_2.13-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-spark-4.1_2.13/@@VERSION@@/paimon-spark-4.1_2.13-@@VERSION@@.jar) |
| Spark 4.0 | [paimon-spark-4.0_2.13-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-spark-4.0_2.13/@@VERSION@@/paimon-spark-4.0_2.13-@@VERSION@@.jar) |
| Spark 3.5 | [paimon-spark-3.5_2.12-@@VERSION@@.jar](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-spark-3.5_2.12/@@VERSION@@/paimon-spark-3.5_2.12-@@VERSION@@.jar) |
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/project/verifying-a-release-candidate.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -264,7 +264,7 @@ the packaging-equivalent build without reproducing the full JDK 17 test lane:
(
cd "paimon-${PAIMON_VERSION}"
mvn -ntp clean install -DskipTests -Pdocs-and-source,spark4 \
-pl paimon-spark/paimon-spark-4.0,paimon-spark/paimon-spark-4.1 \
-pl paimon-spark/paimon-spark-4.0,paimon-spark/paimon-spark-4.1,paimon-spark/paimon-spark-4.2 \
-am
)
```
Expand Down
7 changes: 6 additions & 1 deletion docs/docs/spark/installation.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ Match the connector to the Spark minor version and Scala binary version of your
The Java versions below describe how the connector artifacts are built; your Spark distribution
may require a newer Java runtime. Check its runtime requirements before starting Spark.

- Spark 4.x (including 4.1, 4.0) : Pre-built with Java 17 and Scala 2.13
- Spark 4.x (including 4.2, 4.1, 4.0) : Pre-built with Java 17 and Scala 2.13

- Spark 3.x (including 3.5, 3.4, 3.3, 3.2) : Pre-built with Java 8 and Scala 2.12/2.13

Expand All @@ -42,6 +42,7 @@ Choose the JAR matching your Spark and Scala versions:

| Version | Jar (Scala 2.12) | Jar (Scala 2.13) |
|-----------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Spark 4.2 | - | [JAR](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-spark-4.2_2.13/@@VERSION@@/paimon-spark-4.2_2.13-@@VERSION@@.jar) |
| Spark 4.1 | - | [JAR](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-spark-4.1_2.13/@@VERSION@@/paimon-spark-4.1_2.13-@@VERSION@@.jar) |
| Spark 4.0 | - | [JAR](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-spark-4.0_2.13/@@VERSION@@/paimon-spark-4.0_2.13-@@VERSION@@.jar) |
| Spark 3.5 | [JAR](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-spark-3.5_2.12/@@VERSION@@/paimon-spark-3.5_2.12-@@VERSION@@.jar) | [JAR](https://repo.maven.apache.org/maven2/org/apache/paimon/paimon-spark-3.5_2.13/@@VERSION@@/paimon-spark-3.5_2.13-@@VERSION@@.jar) |
Expand All @@ -55,6 +56,7 @@ Choose the JAR matching your Spark and Scala versions:

| Version | Jar (Scala 2.12) | Jar (Scala 2.13) |
|-----------|-----------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------|
| Spark 4.2 | - | [JAR](https://repository.apache.org/snapshots/org/apache/paimon/paimon-spark-4.2_2.13/@@VERSION@@/) |
| Spark 4.1 | - | [JAR](https://repository.apache.org/snapshots/org/apache/paimon/paimon-spark-4.1_2.13/@@VERSION@@/) |
| Spark 4.0 | - | [JAR](https://repository.apache.org/snapshots/org/apache/paimon/paimon-spark-4.0_2.13/@@VERSION@@/) |
| Spark 3.5 | [JAR](https://repository.apache.org/snapshots/org/apache/paimon/paimon-spark-3.5_2.12/@@VERSION@@/) | [JAR](https://repository.apache.org/snapshots/org/apache/paimon/paimon-spark-3.5_2.13/@@VERSION@@/) |
Expand Down Expand Up @@ -82,6 +84,9 @@ mvn clean package -DskipTests -pl paimon-spark/paimon-spark-4.0 -am -Pspark4

# build paimon spark 4.1
mvn clean package -DskipTests -pl paimon-spark/paimon-spark-4.1 -am -Pspark4

# build paimon spark 4.2
mvn clean package -DskipTests -pl paimon-spark/paimon-spark-4.2 -am -Pspark4
```

For Spark 3.5, you can find the bundled jar in `./paimon-spark/paimon-spark-3.5/target/paimon-spark-3.5_2.12-@@VERSION@@.jar`.
Expand Down
6 changes: 5 additions & 1 deletion paimon-spark/paimon-spark-4.0/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -124,9 +124,13 @@ under the License.

<!-- test -->

<!-- The 4.0-compiled build of `paimon-spark-ut`'s test sources. The baseline build cannot
be used here: Spark 4.2 turned `QueryTest` into a trait (SPARK-55910), so a
4.2-compiled `PaimonSparkTestBase` carries it as an interface and fails to load on a
4.0/4.1 runtime. See `paimon-spark-ut-4.0/pom.xml`. -->
<dependency>
<groupId>org.apache.paimon</groupId>
<artifactId>paimon-spark-ut_${scala.binary.version}</artifactId>
<artifactId>paimon-spark-ut-4.0_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<classifier>tests</classifier>
<scope>test</scope>
Expand Down
Loading
Loading