Skip to content

chore(deps): upgrade to DataFusion 55 and arrow 59 - #34

Merged
zfarrell merged 4 commits into
mainfrom
chore/datafusion-55
Aug 28, 2026
Merged

zfarrell merged 4 commits into
mainfrom
chore/datafusion-55

Conversation

@zfarrell

Copy link
Copy Markdown
Contributor

Bumps DataFusion 54→55 and the arrow/parquet family 58→59, which forces the MSRV pin to 1.94.0 and requires adapting to the DF 55 planner/ExecutionPlan API changes (&dyn Session, PhysicalPlanningContext, the now-required apply_expressions) and replacing the deprecated ParquetObjectReader with a direct AsyncFileReader over ObjectStore.

CI now builds, lints, and tests with --all-features so the feature-gated parquet/sqlite/feather providers are actually covered.

@zfarrell
zfarrell requested a review from a team as a code owner August 28, 2026 02:36
@zfarrell
zfarrell requested review from anoop-narang and removed request for a team August 28, 2026 02:36
Comment thread .github/workflows/ci.yml
restore-keys: |
${{ runner.os }}-cargo-registry-
- run: cargo check --all-targets
- run: cargo check --all-targets --all-features

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: switching all three jobs from bare to --all-features gains feature coverage but drops the default build (not blocking).

The crate has no default features and every feature-gated test file is #![cfg(feature = "...")]-gated (e.g. tests/parquet_provider_test.rs:1), so the previous cargo check --all-targets was the only thing verifying that the crate compiles with no features enabled. After this change nothing does — a missing #[cfg(feature = ...)] on an item that references a feature-gated dep would build green here and break for anyone depending on the crate with default features.

Cheap fix is to keep both, e.g. on the check job:

      - run: cargo check --all-targets
      - run: cargo check --all-targets --all-features

Separately, CLAUDE.md:17 still documents the pre-push command as cargo test --features sqlite-provider; worth updating to --all-features so local runs match CI.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 90b0aa5 — you were right that this was a real coverage regression, not just a style point. The crate has no default features and the provider test files are all #![cfg(feature = ...)]-gated, so the bare cargo check --all-targets really was the only thing proving the no-feature build compiles. Restored it alongside the all-features one exactly as you suggested; verified cargo check --all-targets passes locally.

Also updated CLAUDE.md:17 to cargo test --all-features in 187a0e7 so the documented pre-push command matches CI.

Comment thread Cargo.toml
[dependencies]
tracing = "0.1"
datafusion = "54"
datafusion = "55"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

super nit: README.md:7 advertises **DataFusion:** 53 (not blocking).

It was already stale at 54, and this bump makes it two majors behind. Since this PR is the version bump, it's the natural place to fix it to 55.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 187a0e7 — README now says **DataFusion:** 55. Agreed this is the natural place for it since this PR is the version bump.

claude[bot]
claude Bot previously approved these changes Aug 28, 2026

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the DF 55 / arrow 59 adaptation. The API migrations look faithful:

  • ObjectStoreReader matches the semantics of the ParquetObjectReader it replaces — the old reader also had file_size: None and took the load_via_suffix_and_finish path with no page-index preload, so footer round-trips and page-index behaviour are unchanged.
  • All three ExecutionPlan impls (USearchExec, SqliteFullScanExec, BatchExec) got apply_expressions, and the no-op is correct for each: none holds a root PhysicalExpr, and USearchExec's pre-scan predicate lives in the FilterExec that children() already exposes.
  • &dyn Session / planning_ctx threading in planner.rs is complete — no other SessionState or create_physical_expr call sites remain.
  • Cargo.lock resolves single versions of arrow 59.2.0, parquet 59.2.0 and object_store 0.13.2, so no duplicate-crate type mismatches.

Two non-blocking nits inline. Note that CI was still pending when this review ran, so I have not seen build/clippy/test results.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Prior nits addressed: default-feature cargo check --all-targets restored alongside --all-features, CLAUDE.md pre-push command updated, and README.md DataFusion version bumped to 55.

Checked the DF 55 adaptation:

  • apply_expressions is implemented on all three ExecutionPlan impls in the crate (src/planner.rs:316, src/sqlite_provider.rs:639, src/udtf.rs:240) — none missed. All three own no root physical exprs; USearchExec's pre-scan predicate lives in the FilterExec under provider_scan, which children() returns, so the expression walk still reaches it.
  • ObjectStoreReader follows the upstream parquet suffix-fetch pattern, and CachedMetaReader still short-circuits get_metadata, so the per-query footer round trips remain eliminated.
  • Cargo.lock resolves a single object_store 0.13.2, so the direct 0.13 pin and DF 55's transitive dependency don't split into two incompatible copies.

No new findings. Approving on code review; CI checks were still pending when this review ran, so this is not a statement that the build/tests passed.

@zfarrell
zfarrell merged commit a4d4a65 into main Aug 28, 2026
6 checks passed
@zfarrell
zfarrell deleted the chore/datafusion-55 branch August 28, 2026 21:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant