fix: Emit LIKE the way the Substrait extension defines it - #25442
Open
namanjain24-sudo wants to merge 1 commit into
Open
namanjain24-sudo wants to merge 1 commit into
namanjain24-sudo wants to merge 1 commit into
Conversation
The extension defines one function, `like`, taking two arguments and carrying case sensitivity as the `case_sensitivity` option. The producer emitted three arguments, registered `ilike` for the case insensitive form, which no extension defines, and never set the option. substrait-java resolves `ilike` to nothing, and Spark rejects the three argument call with WRONG_NUM_ARGS. The producer now registers `like`, emits the two arguments, and sets the option for ILIKE. An escape character has no place in that definition, so LIKE ... ESCAPE is rejected rather than emitted as a third argument. The consumer reads the option, taking the first value it supports, and still accepts the `ilike` name and the three argument form so that plans from an older DataFusion keep loading.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #25442 +/- ##
==========================================
- Coverage 82.34% 82.33% -0.01%
==========================================
Files 1137 1137
Lines 432364 432574 +210
Branches 432364 432574 +210
==========================================
+ Hits 356020 356178 +158
- Misses 54830 54853 +23
- Partials 21514 21543 +29 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
functions_string.yamldefines one function for this, with two arguments and an option:The producer emitted three arguments, registered
ilikefor the case insensitive form, and never set the option. Neitherilikenor an escape argument appears in any extension file, in the pinned crate or upstream.What changes are included in this PR?
make_substrait_like_exprregisterslike, emits the two arguments the definition takes, and setscase_sensitivitytoCASE_INSENSITIVEforILIKE. A plainLIKEsets no option, which leaves the default.LIKE ... ESCAPEis nownot_impl_err. There is no escape in the definition, so the third argument only produced a call a consumer cannot bind.case_sensitivity, taking the first value it supports and refusing the ones it does not, as Substrait requires. It still accepts theilikename and the three argument form, so plans written by an older DataFusion still load.What is the testing strategy for this PR?
LIKEemitslikewith two arguments and no option,ILIKEemits thecase_sensitivityoption, and an escape character is rejected.roundtrip_likeandroundtrip_ilikestill pass, which is what covers the consumer: with the option reading removed,roundtrip_ilikefails becauseILIKEcomes back asLIKE. Removing the producer half likewise fails the new producer test.cargo test -p datafusion-substraitpasses and./ci/scripts/rust_clippy.shis clean.Measured against substrait-java 0.103.0 and Spark 3.5.4, over
t(s)holdingabcandABC, with the #11545 URN patched andoutput_typefilled (#25049):s LIKE 'a%'AnalysisException: WRONG_NUM_ARGS ... requires 2 parameters but the actual number is 3abcs ILIKE 'a%'IllegalArgumentException: Unexpected scalar function with key ilike:str_strabcThe second row is a trade-off worth stating.
ILIKEused to fail loudly there, and now it is accepted but answered case sensitively, because substrait-spark does not read the option:case_sensitivityappears nowhere in its sources. Substrait says a consumer that does not recognise an option must reject the call, so that looks like a gap on that side, and I am happy to report it there. If you would rather not emit something a known consumer mishandles, the alternative is to rejectILIKEas well until a consumer honours the option, and I will change it.Are there any user-facing changes?
LIKEandILIKEare emitted in the form the extension defines, so consumers that bind arguments by the declaration can now read them.LIKE ... ESCAPEis rejected instead of being emitted in a shape no consumer can bind; DataFusion round trips it today only because its own consumer reads the same private convention.This touches
make_substrait_like_expr, which #25367 also edits, so whichever lands second needs a small rebase.