WIP: Update to arrow/parquet 59.2.0 by alamb · Pull Request #24030 · apache/datafusion

alamb · 2026-07-31T15:19:52Z

Which issue does this PR close?

Related to Release arrow-rs / parquet Minor version 59.2.0 (July 2026) arrow-rs#9879

Rationale for this change

Upgrade to latest arrow / parquet library

What changes are included in this PR?

Update version pins
Update for API deprecation (I will comment inline)

Are these changes tested?

yes, by CI

Are there any user-facing changes?

Not yet

github-actions · 2026-07-31T15:49:30Z

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
error: running 'cargo update' on crate 'datafusion-datasource-parquet' failed with output:
-----
    Updating crates.io index
error: failed to select a version for the requirement `arrow = "^59.2.0"`
candidate versions found which didn't match: 59.1.0, 59.0.0, 58.4.0, ...
location searched: crates.io index
required by package `datafusion-datasource-parquet v54.1.0 (/home/runner/work/datafusion/datafusion/datafusion/datasource-parquet)`
    ... which satisfies path dependency `datafusion-datasource-parquet` of package `placeholder v0.0.0 (/home/runner/work/datafusion/datafusion/target/semver-checks/local-datafusion_datasource_parquet-54_1_0-x86_64_unknown_linux_gnu-709159958f346fac)`

-----
error: failed to update dependencies for crate datafusion-datasource-parquet v54.1.0
note: this is unlikely to be a bug in cargo-semver-checks,
      and is probably an issue with the crate's Cargo.toml
note: the following command can be used to reproduce the compilation error:
      cargo new --lib example &&
          cd example &&
          echo '[workspace]' >> Cargo.toml &&
          cargo add --path /home/runner/work/datafusion/datafusion/datafusion/datasource-parquet --features parquet_encryption &&
          cargo update

error: running 'cargo update' on crate 'datafusion-functions-aggregate' failed with output:
-----
    Updating crates.io index
error: failed to select a version for the requirement `arrow = "^59.2.0"`
candidate versions found which didn't match: 59.1.0, 59.0.0, 58.4.0, ...
location searched: crates.io index
required by package `datafusion-functions-aggregate v54.1.0 (/home/runner/work/datafusion/datafusion/datafusion/functions-aggregate)`
    ... which satisfies path dependency `datafusion-functions-aggregate` of package `placeholder v0.0.0 (/home/runner/work/datafusion/datafusion/target/semver-checks/local-datafusion_functions_aggregate-54_1_0-x86_64_unknown_linux_gnu-12bb28bb2ce9d807)`

-----
error: failed to update dependencies for crate datafusion-functions-aggregate v54.1.0
note: this is unlikely to be a bug in cargo-semver-checks,
      and is probably an issue with the crate's Cargo.toml
note: the following command can be used to reproduce the compilation error:
      cargo new --lib example &&
          cd example &&
          echo '[workspace]' >> Cargo.toml &&
          cargo add --path /home/runner/work/datafusion/datafusion/datafusion/functions-aggregate --features force_hash_collisions &&
          cargo update

error: running 'cargo update' on crate 'datafusion-proto' failed with output:
-----
    Updating crates.io index
error: failed to select a version for the requirement `arrow = "^59.2.0"`
candidate versions found which didn't match: 59.1.0, 59.0.0, 58.4.0, ...
location searched: crates.io index
required by package `datafusion-proto v54.1.0 (/home/runner/work/datafusion/datafusion/datafusion/proto)`
    ... which satisfies path dependency `datafusion-proto` of package `placeholder v0.0.0 (/home/runner/work/datafusion/datafusion/target/semver-checks/local-datafusion_proto-54_1_0-x86_64_unknown_linux_gnu-e8a5173572a151c0)`

-----
error: failed to update dependencies for crate datafusion-proto v54.1.0
note: this is unlikely to be a bug in cargo-semver-checks,
      and is probably an issue with the crate's Cargo.toml
note: the following command can be used to reproduce the compilation error:
      cargo new --lib example &&
          cd example &&
          echo '[workspace]' >> Cargo.toml &&
          cargo add --path /home/runner/work/datafusion/datafusion/datafusion/proto --features avro,datafusion-datasource-avro,datafusion-datasource-parquet,default,json,parquet,serde_json &&
          cargo update

error: aborting due to failure to run 'cargo update' for crate datafusion-datasource-parquet v54.1.0

alamb · 2026-07-31T15:39:18Z

-        if let Some(hint) = metadata_size_hint {
-            inner = inner.with_footer_size_hint(hint)
-        };
+        let location = partitioned_file.object_meta.location;


ParquetObjectReader is deprecated due to

Remove object_store dependency - Extract object store related code arrow-rs#10308

alamb · 2026-07-31T15:40:26Z

    filename: String,
    metadata: Arc<ParquetMetaData>,
-    inner: ParquetObjectReader,
+    object_store: Arc<dyn ObjectStore>,


this is arguable a nicer example now, as it avoids having to wrap the ParquetObjectReader

alamb · 2026-07-31T20:30:24Z

        let to_write = RecordBatch::try_from_iter(data).unwrap();
        let path = object_store::path::Path::from(format!("{file_num}.parquet").as_str());
-        let object_store_writer = ParquetObjectWriter::new(Arc::clone(&store) as _, path);
+        let object_store_writer = BufWriter::new(Arc::clone(&store) as _, path);


ParquetObjectWriter wsas deprecated: apache/arrow-rs#10308

ParquetObjectWriter is a thin wrapper around BufWriter: https://docs.rs/parquet/latest/src/parquet/arrow/async_writer/store.rs.html#72-74

alamb · 2026-07-31T20:32:08Z

    ) -> BoxFuture<'_, datafusion::parquet::errors::Result<Bytes>> {
        println!("get_bytes: {} Reading range {:?}", self.filename, range);
-        self.inner.get_bytes(range)
+        let object_store = Arc::clone(&self.object_store);


This is due to inlining the logic for ParquetObjectReader

It turns out that DataFusion already has its own copy of the logic of the ParquetObjectReader between ParquetFileReader and CachedParquetFileReader. I have a PR that will unify them:

refactor: unify ParquetFileReader and CachedParquetFileReader #24036

alamb · 2026-07-31T20:33:44Z

        options: Option<&'a ArrowReaderOptions>,
    ) -> BoxFuture<'a, parquet::errors::Result<Arc<ParquetMetaData>>> {
-        self.inner.get_metadata(options)
+        let object_meta = self.partitioned_file.object_meta.clone();


THe need for this replication is removed by unifying the structures, as I propose to do do in

refactor: unify ParquetFileReader and CachedParquetFileReader #24036

alamb · 2026-07-31T20:34:17Z

+
+**Migration guide:**
+
+If your [`AsyncFileReader`] implementation exists mainly to read from an


This will need to get updated if we merge this one first

refactor: unify ParquetFileReader and CachedParquetFileReader #24036

alamb · 2026-07-31T21:01:12Z

        acc1 = merge(acc1, acc2)?;

-        assert_eq!(acc1.size(), 282);
+        assert_eq!(acc1.size(), 290);


Due to

fix off by one error for slice accounting arrow-rs#10406

@Jefffrey

… arrow spec (#10297)" (#10506) # Which issue does this PR close? N/A cc @Jefffrey and @rluvaton # Rationale for this change #10297 changed the default map field names from `keys`/`values` to `key`/`value` to match the Arrow spec. This is a good change to align with the spec, but it is a breaking change: it broke the DataFusion upgrade (apache/datafusion#24030) because data produced elsewhere (e.g. by other Arrow implementations) uses the old `keys`/`values` field names, causing schema mismatches such as: ``` InvalidArgumentError("Incorrect datatype for StructArray field \"metadata\", expected Map(\"entries\": non-null Struct(\"keys\": non-null Utf8, \"values\": Utf8), unsorted) got Map(\"entries\": non-null Struct(\"key\": non-null Utf8, \"value\": Utf8), unsorted)") ``` See discussion on #10297: #10297 (comment) We should hold the field name change for the next breaking release to minimize downstream churn on a minor release, rather than ship it in a minor release. # What changes are included in this PR? - Reverts b963ecf (#10297), restoring the default map field names to `keys`/`values` (plural). - Updates a test added by #10475 after #10297 merged # Are these changes tested? Existing tests. # Are there any user-facing changes? Yes: this reverts the default `MapFieldNames` back to `keys`/`values` (as it was prior to #10297), rather than `key`/`value`. The intent is to reapply #10297 as part of the next breaking release.

…rrow-rs 59.2.0 Bumping arrow-rs changed the underlying buffer memory footprint reported by ArrayData::get_slice_memory_size() for this scenario, shifting the accumulator's accounted size from 282 to 290 bytes. This is an expected size increase from the dependency bump, not an accounting regression: the merge behavior and both sibling tests (does_not_over_account_memory_distinct/_ordered) are unaffected.

arrow-rs 59.2.0 added validation that rejects constructing a MapArray whose key field is nullable (apache/arrow-rs#10272), which is required by the Arrow spec. The Map test cases in round_trip_scalar_values_and_data_types built their key field with nullable=true, so decoding the IPC-encoded ScalarValue now fails with "Map key field must not be nullable".

alamb · 2026-08-02T21:35:28Z

                    "entries",
                    DataType::Struct(Fields::from(vec![
-                        Field::new("key", DataType::Int32, true),
+                        Field::new("key", DataType::Int32, false),


this is necessary due to MapArray becoming more strict in

fix(arrow-array): disallow creating MapArray with nullable key field arrow-rs#10272

codecov-commenter · 2026-08-03T09:48:33Z

Codecov Report

❌ Patch coverage is 88.00000% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.86%. Comparing base (f3a895f) to head (3465b7d).
⚠️ Report is 8 commits behind head on main.

Files with missing lines	Patch %	Lines
datafusion/datasource-parquet/src/reader.rs	88.88%	1 Missing and 4 partials ⚠️
benchmarks/src/cancellation.rs	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #24030      +/-   ##
==========================================
- Coverage   80.86%   80.86%   -0.01%     
==========================================
  Files        1101     1101              
  Lines      375446   375464      +18     
  Branches   375446   375464      +18     
==========================================
+ Hits       303596   303606      +10     
- Misses      53757    53759       +2     
- Partials    18093    18099       +6

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

alamb mentioned this pull request Jul 31, 2026

Release arrow-rs / parquet Minor version 59.2.0 (July 2026) apache/arrow-rs#9879

Open

9 tasks

github-actions Bot added the datasource Changes to the datasource crate label Jul 31, 2026

github-actions Bot added auto detected api change Auto detected API change documentation Improvements or additions to documentation labels Jul 31, 2026

alamb mentioned this pull request Jul 31, 2026

chore: formalize the default map field names to match default arrow spec apache/arrow-rs#10297

Merged

alamb commented Jul 31, 2026

View reviewed changes

alamb mentioned this pull request Jul 31, 2026

refactor: unify ParquetFileReader and CachedParquetFileReader #24036

Open

github-actions Bot added common Related to common crate functions Changes to functions implementation labels Jul 31, 2026

alamb commented Jul 31, 2026

View reviewed changes

alamb mentioned this pull request Jul 31, 2026

Revert "chore: formalize the default map field names to match default arrow spec (#10297)" apache/arrow-rs#10506

Merged

alamb commented Jul 31, 2026

View reviewed changes

alamb added 3 commits August 2, 2026 05:19

TEMP: Update to arrow/parquet 59.2.0

1633c2b

chore: Update for deprecated APIs

0933d1b

alamb force-pushed the alamb/update_arrow_59.2.0 branch from e0fe024 to f19d7ce Compare August 2, 2026 09:19

github-actions Bot removed the common Related to common crate label Aug 2, 2026

alamb added 2 commits August 2, 2026 05:32

rettier

42749cf

github-actions Bot added the proto Related to proto crate label Aug 2, 2026

alamb commented Aug 2, 2026

View reviewed changes

Navigation Menu

Uh oh!

WIP: Update to arrow/parquet 59.2.0 - #24030

WIP: Update to arrow/parquet 59.2.0#24030
alamb wants to merge 5 commits into
apache:mainfrom
alamb:alamb/update_arrow_59.2.0

alamb commented Jul 31, 2026

Uh oh!

github-actions Bot commented Jul 31, 2026 •

edited

Loading

Uh oh!

alamb Jul 31, 2026

Uh oh!

alamb Jul 31, 2026

Uh oh!

alamb Jul 31, 2026

Uh oh!

alamb Jul 31, 2026

Uh oh!

Uh oh!

alamb Jul 31, 2026

Uh oh!

alamb Jul 31, 2026

Uh oh!

alamb Jul 31, 2026 •

edited

Loading

Uh oh!

alamb Aug 2, 2026

Uh oh!

codecov-commenter commented Aug 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants


		Migration guide:

		If your [`AsyncFileReader`] implementation exists mainly to read from an

Uh oh!

Conversation

alamb commented Jul 31, 2026

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions Bot commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alamb Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

alamb Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

alamb Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

alamb Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

alamb Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

alamb Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

alamb Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb Aug 2, 2026

Choose a reason for hiding this comment

Uh oh!

codecov-commenter commented Aug 3, 2026

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented Jul 31, 2026 •

edited

Loading

alamb Jul 31, 2026 •

edited

Loading