Skip to content

fix: Emit LIKE the way the Substrait extension defines it - #25442

Open
namanjain24-sudo wants to merge 1 commit into
apache:mainfrom
namanjain24-sudo:fix-substrait-like-conformance
Open

namanjain24-sudo wants to merge 1 commit into
apache:mainfrom
namanjain24-sudo:fix-substrait-like-conformance

Conversation

@namanjain24-sudo

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

functions_string.yaml defines one function for this, with two arguments and an option:

    name: like
    impls:
      - args:
          - value: "varchar<L1>"
            name: "input"
          - value: "varchar<L2>"
            name: "match"
        options:
          case_sensitivity:
            values: [ CASE_SENSITIVE, CASE_INSENSITIVE, CASE_INSENSITIVE_ASCII ]
        return: "boolean"

The producer emitted three arguments, registered ilike for the case insensitive form, and never set the option. Neither ilike nor an escape argument appears in any extension file, in the pinned crate or upstream.

What changes are included in this PR?

  • make_substrait_like_expr registers like, emits the two arguments the definition takes, and sets case_sensitivity to CASE_INSENSITIVE for ILIKE. A plain LIKE sets no option, which leaves the default.
  • LIKE ... ESCAPE is now not_impl_err. There is no escape in the definition, so the third argument only produced a call a consumer cannot bind.
  • The consumer reads case_sensitivity, taking the first value it supports and refusing the ones it does not, as Substrait requires. It still accepts the ilike name and the three argument form, so plans written by an older DataFusion still load.

What is the testing strategy for this PR?

  • New producer tests: LIKE emits like with two arguments and no option, ILIKE emits the case_sensitivity option, and an escape character is rejected.
  • roundtrip_like and roundtrip_ilike still pass, which is what covers the consumer: with the option reading removed, roundtrip_ilike fails because ILIKE comes back as LIKE. Removing the producer half likewise fails the new producer test.
  • cargo test -p datafusion-substrait passes and ./ci/scripts/rust_clippy.sh is clean.

Measured against substrait-java 0.103.0 and Spark 3.5.4, over t(s) holding abc and ABC, with the #11545 URN patched and output_type filled (#25049):

plan before after
s LIKE 'a%' AnalysisException: WRONG_NUM_ARGS ... requires 2 parameters but the actual number is 3 runs, returns abc
s ILIKE 'a%' IllegalArgumentException: Unexpected scalar function with key ilike:str_str runs, returns abc

The second row is a trade-off worth stating. ILIKE used to fail loudly there, and now it is accepted but answered case sensitively, because substrait-spark does not read the option: case_sensitivity appears nowhere in its sources. Substrait says a consumer that does not recognise an option must reject the call, so that looks like a gap on that side, and I am happy to report it there. If you would rather not emit something a known consumer mishandles, the alternative is to reject ILIKE as well until a consumer honours the option, and I will change it.

Are there any user-facing changes?

LIKE and ILIKE are emitted in the form the extension defines, so consumers that bind arguments by the declaration can now read them. LIKE ... ESCAPE is rejected instead of being emitted in a shape no consumer can bind; DataFusion round trips it today only because its own consumer reads the same private convention.

This touches make_substrait_like_expr, which #25367 also edits, so whichever lands second needs a small rebase.

The extension defines one function, `like`, taking two arguments and
carrying case sensitivity as the `case_sensitivity` option. The producer
emitted three arguments, registered `ilike` for the case insensitive
form, which no extension defines, and never set the option. substrait-java
resolves `ilike` to nothing, and Spark rejects the three argument call
with WRONG_NUM_ARGS.

The producer now registers `like`, emits the two arguments, and sets the
option for ILIKE. An escape character has no place in that definition, so
LIKE ... ESCAPE is rejected rather than emitted as a third argument.

The consumer reads the option, taking the first value it supports, and
still accepts the `ilike` name and the three argument form so that plans
from an older DataFusion keep loading.
@github-actions github-actions Bot added the substrait Changes to the substrait crate label Sep 17, 2026
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 90.90909% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.33%. Comparing base (3ab72e6) to head (e003924).
⚠️ Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
.../src/logical_plan/consumer/expr/scalar_function.rs 75.00% 3 Missing and 1 partial ⚠️
.../src/logical_plan/producer/expr/scalar_function.rs 94.44% 2 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #25442      +/-   ##
==========================================
- Coverage   82.34%   82.33%   -0.01%     
==========================================
  Files        1137     1137              
  Lines      432364   432574     +210     
  Branches   432364   432574     +210     
==========================================
+ Hits       356020   356178     +158     
- Misses      54830    54853      +23     
- Partials    21514    21543      +29     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

substrait Changes to the substrait crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Substrait producer emits LIKE with three arguments and an undefined ilike function

2 participants