fix: preserve MERGE target qualifier bindings

[wirybeaver]

Which issue does this PR close?

• Part of Support MERGE INTO #20746.
• Follow-up to feat: Add SQL planner, physical planner, and TableProvider hook for MERGE INTO #22988.

Rationale for this change

#22988 deliberately rejected two valid MERGE forms to avoid silently changing expression meaning.

Limitation 1: target-correlated subquery with an aliased target

MERGE INTO target AS t
USING source AS s
ON EXISTS (SELECT 1 FROM source AS x WHERE x.id = t.id)
WHEN MATCHED THEN DELETE;

t.id inside the subquery becomes outer_ref(t.id). The old top-level canonicalizer only rewrote Expr::Column(t.id) to target.id, leaving the outer reference inconsistent with the schema later rebuilt from DmlStatement.table_name.

Limitation 2: source qualifier equals the target table name

MERGE INTO target AS t
USING source AS target
ON t.id = target.id
WHEN MATCHED THEN DELETE;

Canonicalizing target t.id to target.id collapses both operands onto the source qualifier and can turn the condition into target.id = target.id.

A recursive string rewrite is not safe: qualifiers are scope-local, so an inner relation can legally shadow t. It also cannot solve the second limitation because both relations would still have the same qualifier after rewriting.

What changes are included in this PR?

Solution

Keep provider identity and the SQL-visible target qualifier as separate plan state:

SQL: MERGE INTO target AS t USING source AS target

                     SQL binding
                +-------------------+
                |                   |
        target table           source relation
        provider: target       qualifier: target
        qualifier: t
                |                   |
                +---------+---------+
                          |
              MERGE expression schema
              +-----------+-----------+
              | t.*       | target.*  |
              | index 0.. | index N.. |
              +-----------+-----------+
                          |
                 TableProvider::merge_into

Logical / protobuf representation

  DmlStatement.table_name / DmlNode.table_name
      = target                 (provider identity)

  MergeIntoOp.target_qualifier /
  MergeIntoOpNode.target_qualifier
      = t                      (SQL-visible binding)

Planning now proceeds as follows:

1. Resolve target to the target provider, while retaining alias t as the visible qualifier.
2. Store t on MergeIntoOp; do not canonicalize target columns or recursively rewrite subquery plans.
3. Build one target-first expression schema (t.*, then source fields) and use it consistently in SQL planning, analyzer/optimizer rules, physical planning, and programmatic plans.
4. Reject only a source qualifier that collides with the visible target qualifier in the outer MERGE scope. A source qualifier matching the real table name remains valid when the target is aliased.
5. Pass the preserved schema and expressions to TableProvider::merge_into, where target and source columns resolve to distinct physical indices.

Protobuf change and 55.0/55.1 compatibility

MergeIntoOpNode gains optional target_qualifier field 3. This field is necessary because DmlNode.table_name contains provider identity and cannot also represent alias t; without it, a proto round trip loses the binding needed to resolve MERGE expressions.

Compatibility is directional:

• 55.0 payload -> 55.1 reader: supported. The field is absent, so the new reader falls back to DmlNode.table_name, matching 55.0's canonicalized representation.
• 55.1 payload -> 55.0 reader: unsupported. A 55.0 reader ignores the unknown field but alias-preserving expressions still require it, so expressions can fail resolution or be misbound.

Rust API compatibility

DataFusion 55.0 released MergeIntoOp with public struct-literal construction: MergeIntoOp { on, clauses }. This PR makes the struct non-exhaustive, adds private target-qualifier state, and requires MergeIntoOp::new(target_qualifier, on, clauses). Therefore existing 55.0 downstream struct literals will not compile unchanged against 55.1. This compatibility exception should be considered explicitly for the 55.1 release.

The PR also:

• removes alias canonicalization and the recursive target-correlation guard;
• documents that providers may receive residual subqueries; and
• adds coverage for direct correlations, nested shadowing, lateral and LIMIT scopes, qualifier collisions, quoted/qualified identifiers, proto fallback/round trips, and physical column indices.

Are these changes tested?

• cargo fmt --all
• cargo clippy --all-targets --all-features -- -D warnings
• ./ci/scripts/doc_prettier_check.sh --write --allow-dirty
• RUST_BACKTRACE=1 cargo test --profile ci --exclude datafusion-examples --exclude datafusion-benchmarks --exclude datafusion-cli --workspace --lib --tests --bins --features avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption

Are there any user-facing changes?

Yes. Both valid MERGE forms above now plan successfully and reach TableProvider::merge_into. The Rust and protobuf compatibility constraints for upgrading from 55.0 to 55.1 are documented above.

[github-actions]

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
    Building datafusion v55.0.0 (current)
       Built [  59.069s] (current)
     Parsing datafusion v55.0.0 (current)
      Parsed [   0.035s] (current)
    Building datafusion v55.0.0 (baseline)
       Built [  58.605s] (baseline)
     Parsing datafusion v55.0.0 (baseline)
      Parsed [   0.034s] (baseline)
    Checking datafusion v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   0.579s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [ 119.842s] datafusion
    Building datafusion-expr v55.0.0 (current)
       Built [  29.727s] (current)
     Parsing datafusion-expr v55.0.0 (current)
      Parsed [   0.076s] (current)
    Building datafusion-expr v55.0.0 (baseline)
       Built [  29.238s] (baseline)
     Parsing datafusion-expr v55.0.0 (baseline)
      Parsed [   0.077s] (baseline)
    Checking datafusion-expr v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   1.174s] 223 checks: 221 pass, 1 fail, 1 warn, 31 skip

--- failure struct_marked_non_exhaustive: struct marked #[non_exhaustive] ---

Description:
A public struct has been marked #[non_exhaustive], which will prevent it from being constructed using a struct literal outside of its crate. It previously had no private fields, so a struct literal could be used to construct it outside its crate.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#attr-adding-non-exhaustive
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/struct_marked_non_exhaustive.ron

Failed in:
  struct MergeIntoOp in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/dml.rs:320
  struct MergeIntoOp in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/dml.rs:320
  struct MergeIntoOp in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/dml.rs:320
  struct MergeIntoOp in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/dml.rs:320

--- warning partial_ord_struct_fields_reordered: struct fields reordered in #[derive(PartialOrd)] struct ---

Description:
A public struct that derives PartialOrd had its fields reordered. #[derive(PartialOrd)] uses the field order to set the struct's ordering behavior, so this change may break downstream code that relies on the previous order.
        ref: https://doc.rust-lang.org/std/cmp/trait.PartialOrd.html#derivable
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/partial_ord_struct_fields_reordered.ron

Failed in:
  MergeIntoOp.on moved from position 1 to 2, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/dml.rs:324
  MergeIntoOp.clauses moved from position 2 to 3, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/dml.rs:326
  MergeIntoOp.on moved from position 1 to 2, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/dml.rs:324
  MergeIntoOp.clauses moved from position 2 to 3, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/dml.rs:326
  MergeIntoOp.on moved from position 1 to 2, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/dml.rs:324
  MergeIntoOp.clauses moved from position 2 to 3, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/dml.rs:326
  MergeIntoOp.on moved from position 1 to 2, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/dml.rs:324
  MergeIntoOp.clauses moved from position 2 to 3, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/dml.rs:326

     Summary semver requires new major version: 1 major and 0 minor checks failed
     Warning produced 1 major and 0 minor level warnings
    Finished [  61.228s] datafusion-expr
    Building datafusion-optimizer v55.0.0 (current)
       Built [  27.281s] (current)
     Parsing datafusion-optimizer v55.0.0 (current)
      Parsed [   0.030s] (current)
    Building datafusion-optimizer v55.0.0 (baseline)
       Built [  27.764s] (baseline)
     Parsing datafusion-optimizer v55.0.0 (baseline)
      Parsed [   0.032s] (baseline)
    Checking datafusion-optimizer v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   0.151s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [  55.983s] datafusion-optimizer
    Building datafusion-proto v55.0.0 (current)
       Built [  55.077s] (current)
     Parsing datafusion-proto v55.0.0 (current)
      Parsed [   0.017s] (current)
    Building datafusion-proto v55.0.0 (baseline)
       Built [  55.759s] (baseline)
     Parsing datafusion-proto v55.0.0 (baseline)
      Parsed [   0.018s] (baseline)
    Checking datafusion-proto v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   0.105s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [ 112.677s] datafusion-proto
    Building datafusion-proto-models v55.0.0 (current)
       Built [  25.543s] (current)
     Parsing datafusion-proto-models v55.0.0 (current)
      Parsed [   0.129s] (current)
    Building datafusion-proto-models v55.0.0 (baseline)
       Built [  25.145s] (baseline)
     Parsing datafusion-proto-models v55.0.0 (baseline)
      Parsed [   0.134s] (baseline)
    Checking datafusion-proto-models v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   1.623s] 223 checks: 222 pass, 1 fail, 0 warn, 31 skip

--- failure constructible_struct_adds_field: struct exhaustively constructible through public API adds field ---

Description:
A pub struct that could be exhaustively constructed with a literal using only public API has a new pub field, breaking existing exhaustive literals.
        ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/constructible_struct_adds_field.ron

Failed in:
  field MergeIntoOpNode.target_qualifier in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:540
  field MergeIntoOpNode.target_qualifier in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:540

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  54.008s] datafusion-proto-models
    Building datafusion-session v55.0.0 (current)
       Built [  38.459s] (current)
     Parsing datafusion-session v55.0.0 (current)
      Parsed [   0.011s] (current)
    Building datafusion-session v55.0.0 (baseline)
       Built [  37.716s] (baseline)
     Parsing datafusion-session v55.0.0 (baseline)
      Parsed [   0.011s] (baseline)
    Checking datafusion-session v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   0.177s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [  77.333s] datafusion-session
    Building datafusion-sql v55.0.0 (current)
       Built [  43.486s] (current)
     Parsing datafusion-sql v55.0.0 (current)
      Parsed [   0.031s] (current)
    Building datafusion-sql v55.0.0 (baseline)
       Built [  44.427s] (baseline)
     Parsing datafusion-sql v55.0.0 (baseline)
      Parsed [   0.032s] (baseline)
    Checking datafusion-sql v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   0.227s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [  89.209s] datafusion-sql
    Building datafusion-sqllogictest v55.0.0 (current)
       Built [ 103.595s] (current)
     Parsing datafusion-sqllogictest v55.0.0 (current)
      Parsed [   0.023s] (current)
    Building datafusion-sqllogictest v55.0.0 (baseline)
       Built [ 104.555s] (baseline)
     Parsing datafusion-sqllogictest v55.0.0 (baseline)
      Parsed [   0.022s] (baseline)
    Checking datafusion-sqllogictest v55.0.0 -> v55.0.0 (no change; assume patch)
     Checked [   0.098s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [ 210.905s] datafusion-sqllogictest

[codecov-commenter]

Codecov Report

❌ Patch coverage is 79.85075% with 27 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.34%. Comparing base (5f0ba13) to head (3d19096).

Files with missing lines	Patch %	Lines
datafusion/proto-models/src/generated/pbjson.rs	0.00%	13 Missing ⚠️
datafusion/expr/src/logical_plan/dml.rs	93.15%	1 Missing and 4 partials ⚠️
datafusion/core/src/physical_planner.rs	50.00%	0 Missing and 3 partials ⚠️
...afusion/optimizer/src/analyzer/function_rewrite.rs	0.00%	2 Missing ⚠️
datafusion/optimizer/src/analyzer/type_coercion.rs	75.00%	0 Missing and 1 partial ⚠️
datafusion/optimizer/src/rewrite_set_comparison.rs	50.00%	0 Missing and 1 partial ⚠️
...timizer/src/simplify_expressions/simplify_exprs.rs	50.00%	0 Missing and 1 partial ⚠️
datafusion/proto/src/logical_plan/from_proto.rs	93.33%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #24429      +/-   ##
==========================================
- Coverage   81.34%   81.34%   -0.01%     
==========================================
  Files        1117     1117              
  Lines      397528   397525       -3     
  Branches   397528   397525       -3     
==========================================
- Hits       323385   323376       -9     
- Misses      55225    55229       +4     
- Partials    18918    18920       +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[wirybeaver]

@alamb @kosiew @timsaucer Could you take a look of this PR before the Datafusion 55.0.0 release. I think adding the target table's qualifer into the protobuf is a more elegant solution to make code succinct and recursively rewriting subquery.

[kosiew]

@wirybeaver,

Thanks for working on this. This looks good to me.

I like the approach of keeping MERGE's SQL-visible target qualifier separate from the target provider identity. Using that qualifier consistently when rebuilding the MERGE expression schema across the analyzer, optimizer, and physical planner avoids the alias canonicalization issues while preserving the correct SQL scoping semantics.

The protobuf handling also looks good. Persisting the qualifier while falling back to DmlNode.table_name for older payloads keeps the change backward compatible.

The added coverage for aliased targets, source-name collisions, correlated subqueries, qualifier shadowing, and the basic protobuf round trip gives me good confidence in the change.

One note on protobuf coverage: I think keeping the MERGE round-trip test's ON expression flat is appropriate here. A target-correlated subquery cannot currently be serialized because datafusion/proto/src/logical_plan/to_proto.rs does not support OuterReferenceColumn, Exists, or InSubquery. A MERGE round-trip test for that case would fit better with future work adding broader protobuf support for correlated subquery expressions.

Thanks again!

[alamb]

Thanks for this @wirybeaver and @kosiew

The code looks good -- I would just like to request we migrate some of these tests to use slt rather than more rust code

[wirybeaver]

@alamb The testing code has moved to SLT. Thanks for the guide