feat: Implement FFI_QueryPlanner by timsaucer · Pull Request #24028 · apache/datafusion

timsaucer · 2026-07-31T12:50:06Z

Which issue does this PR close?

Unblocks Integrate datafusion-distributed with datafusion-python datafusion-python#1612

Rationale for this change

This is the last in a series of PRs that would enable FFI Session to support a QueryPlanner. The prior work was in

With those changes in place we now have the dependencies correct that we can expose a FFI_QueryPlanner on a FFI_Session. With this we can enable foreign libraries such as datafusion-distributed and ballista to provide a query planner in Python and connect it directly to a datafusion-python's SessionContext.

What changes are included in this PR?

Addition only. Adds these functions to FFI_Session and their supporting structures:

query_planner()
optimize()
physical_optimizers()

Are these changes tested?

Unit and integration tests are provided.

Are there any user-facing changes?

This is addition, but it does break the FFI ABI, which is already evolving in DF55.

…ncryption feature is not enabled

Add FFI_QueryPlanner and ForeignQueryPlanner with logical and physical plan codec support. Forward query planners, logical optimization, and physical optimizer rules through foreign sessions, with unit and cross-library coverage.\n\nAI Disclosure: This code was written in part by an AI agent.

timsaucer · 2026-07-31T12:50:57Z


 #[cfg(not(feature = "parquet_encryption"))]
 #[expect(dead_code)]
+#[expect(clippy::unused_async)]


Drive by fix to when you run clippy and don't have the parquet_encryption feature enabled.

timsaucer · 2026-07-31T12:51:04Z

 }

 #[cfg(not(feature = "parquet_encryption"))]
+#[expect(clippy::unused_async)]


Drive by fix to when you run clippy and don't have the parquet_encryption feature enabled.

timsaucer · 2026-07-31T12:51:08Z

 }

 #[cfg(not(feature = "parquet_encryption"))]
+#[expect(clippy::unused_async)]


Drive by fix to when you run clippy and don't have the parquet_encryption feature enabled.

github-actions · 2026-07-31T12:59:09Z

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
    Building datafusion-datasource-parquet v54.1.0 (current)
       Built [  46.672s] (current)
     Parsing datafusion-datasource-parquet v54.1.0 (current)
      Parsed [   0.033s] (current)
    Building datafusion-datasource-parquet v54.1.0 (baseline)
       Built [  47.520s] (baseline)
     Parsing datafusion-datasource-parquet v54.1.0 (baseline)
      Parsed [   0.032s] (baseline)
    Checking datafusion-datasource-parquet v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.152s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [  95.821s] datafusion-datasource-parquet
    Building datafusion-ffi v54.1.0 (current)
       Built [  65.106s] (current)
     Parsing datafusion-ffi v54.1.0 (current)
      Parsed [   0.064s] (current)
    Building datafusion-ffi v54.1.0 (baseline)
       Built [  64.435s] (baseline)
     Parsing datafusion-ffi v54.1.0 (baseline)
      Parsed [   0.064s] (baseline)
    Checking datafusion-ffi v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.233s] 223 checks: 221 pass, 1 fail, 1 warn, 30 skip

--- failure constructible_struct_adds_field: externally-constructible struct adds field ---

Description:
A pub struct constructible with a struct literal has a new pub field. Existing struct literals must be updated to include the new field.
        ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.49.0/src/lints/constructible_struct_adds_field.ron

Failed in:
  field ForeignLibraryModule.create_query_planner in /home/runner/work/datafusion/datafusion/datafusion/ffi/src/tests/mod.rs:122

--- warning repr_c_plain_struct_fields_reordered: struct fields reordered in repr(C) struct ---

Description:
A public repr(C) struct had its fields reordered. This can change the struct's memory layout, possibly breaking FFI use cases that depend on field position and order.
        ref: https://doc.rust-lang.org/reference/type-layout.html#reprc-structs
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.49.0/src/lints/repr_c_plain_struct_fields_reordered.ron

Failed in:
  ForeignLibraryModule.version moved from position 19 to 20, in /home/runner/work/datafusion/datafusion/datafusion/ffi/src/tests/mod.rs:127

     Summary semver requires new major version: 1 major and 0 minor checks failed
     Warning produced 1 major and 0 minor level warnings
    Finished [ 132.361s] datafusion-ffi

Keep the standard FFI_SessionRef constructor focused on the required logical codec and derive a default physical codec. Add an explicit constructor for callers that already own matching logical and physical codecs.\n\nAI Disclosure: This code was written in part by an AI agent.

codecov-commenter · 2026-07-31T13:22:51Z

Codecov Report

❌ Patch coverage is 93.56061% with 17 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.87%. Comparing base (dbcb5c0) to head (b18d41d).

Files with missing lines	Patch %	Lines
datafusion/ffi/src/query_planner.rs	93.67%	1 Missing and 10 partials ⚠️
datafusion/ffi/src/session/mod.rs	93.33%	0 Missing and 6 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #24028      +/-   ##
==========================================
+ Coverage   80.85%   80.87%   +0.01%     
==========================================
  Files        1101     1102       +1     
  Lines      374933   375192     +259     
  Branches   374933   375192     +259     
==========================================
+ Hits       303166   303435     +269     
+ Misses      53671    53644      -27     
- Partials    18096    18113      +17

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Document the query planner serialization boundary and exercise an A/B/C ownership model with independently loaded cdylib images. Reuse the existing FFI table provider and verify foreign plans are reconstructed as local nodes through A's codecs. AI Disclosure: This code was written in part by an AI agent.

Use one test query planner for both the basic round trip and the three-library table scan scenario. Remove the redundant library C constructor from the integration-test module. AI Disclosure: This code was written in part by an AI agent.

Render the private FFI_SessionRef type as code so public query planner documentation passes rustdoc's private intra-doc link checks. AI Disclosure: This code was written in part by an AI agent.

paleolimbot

I will try to circle back to take a closer look at the details, but I did take two passes and didn't spot anything out of place here. You've documented the motivation nicely here and conceptually I think this is a great approach that allows quite a lot of flexibility for multiple datafusion-based libraries to interact. The FFI pattern you've established from previous PRs works well here.

I believe this will allow our (SedonaDB) spatial join to be used in datafusion-python plans (injected via a combination of logical optimizer rules and the query planner). Exciting!

milenkovicm

Hey @timsaucer first of all appologies for late review and thank you for sticking with this. Can't wait for this release to finally get py ballista support.

I have fiew minor questions, more for my understanding than blockers

milenkovicm · 2026-08-01T16:46:08Z

+
+    /// Library A's logical codec stores library B's provider while the logical
+    /// plan crosses into library C. A real application would encode enough
+    /// metadata to reconstruct or locate the provider instead.


just for my reference, which metadata is needed? will need some help to integrate this with py ballista

milenkovicm · 2026-08-01T17:06:09Z

+        &Self,
+        logical_plan_serialized: SVec<u8>,
+        session: FFI_SessionRef,
+    ) -> FfiFuture<FFI_Result<SVec<u8>>>,


One question, why FFI_Result<FFI_ExecutionPlan> not used instad of serialized plan?

timsaucer added 2 commits July 31, 2026 08:31

Add clippy exception so that no warnings are generated when parquet_e…

ba0a3dd

…ncryption feature is not enabled

timsaucer added api change Changes the API exposed to users of the crate ffi Changes to the ffi crate labels Jul 31, 2026

timsaucer commented Jul 31, 2026

View reviewed changes

github-actions Bot added the auto detected api change Auto detected API change label Jul 31, 2026

github-actions Bot added the datasource Changes to the datasource crate label Jul 31, 2026

timsaucer changed the title ~~Feat/ffi query planner main~~ feat: Implement FFI_QueryPlanner Jul 31, 2026

timsaucer added 2 commits July 31, 2026 10:35

test: consolidate FFI query planner fixtures

9ca40b3

Use one test query planner for both the basic round trip and the three-library table scan scenario. Remove the redundant library C constructor from the integration-test module. AI Disclosure: This code was written in part by an AI agent.

docs: fix private query planner links

b18d41d

Render the private FFI_SessionRef type as code so public query planner documentation passes rustdoc's private intra-doc link checks. AI Disclosure: This code was written in part by an AI agent.

timsaucer marked this pull request as ready for review July 31, 2026 14:53

This was referenced Jul 31, 2026

FFI Query Planner on 54.0.0 #23581

Closed

Rerun PRs rerun-io/opensource#2

Open

paleolimbot approved these changes Jul 31, 2026

View reviewed changes

milenkovicm approved these changes Aug 1, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: Implement FFI_QueryPlanner - #24028

feat: Implement FFI_QueryPlanner#24028
timsaucer wants to merge 6 commits into
apache:mainfrom
timsaucer:feat/ffi-query-planner-main

timsaucer commented Jul 31, 2026 •

edited

Loading

Uh oh!

timsaucer Jul 31, 2026

Uh oh!

timsaucer Jul 31, 2026

Uh oh!

timsaucer Jul 31, 2026

Uh oh!

github-actions Bot commented Jul 31, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented Jul 31, 2026 •

edited

Loading

Uh oh!

paleolimbot left a comment

Uh oh!

milenkovicm left a comment

Uh oh!

milenkovicm Aug 1, 2026

Uh oh!

milenkovicm Aug 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

timsaucer commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

timsaucer Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

timsaucer Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

timsaucer Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

paleolimbot left a comment

Choose a reason for hiding this comment

Uh oh!

milenkovicm left a comment

Choose a reason for hiding this comment

Uh oh!

milenkovicm Aug 1, 2026

Choose a reason for hiding this comment

Uh oh!

milenkovicm Aug 1, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

timsaucer commented Jul 31, 2026 •

edited

Loading

github-actions Bot commented Jul 31, 2026 •

edited

Loading

codecov-commenter commented Jul 31, 2026 •

edited

Loading