fix: Set Substrait output_type on window functions and LIKE - #25367
Open
namanjain24-sudo wants to merge 1 commit into
Open
namanjain24-sudo wants to merge 1 commit into
namanjain24-sudo wants to merge 1 commit into
Conversation
Substrait documents output_type on both ScalarFunction and WindowFunction as the return type of the function, and a consumer that reads it rejects a call that leaves it unset: substrait-java refuses a window function plan and a LIKE plan with "Type is not set". Both types are derived from the expression itself, so they match what DataFusion derives: from_window_function from Expr::WindowFunction, and make_substrait_like_expr from Expr::Like, which also covers the not() wrapper a negated LIKE emits. make_substrait_window_function had a single caller and its body now sits in from_window_function, and make_substrait_like_expr takes the Like rather than its five fields, so neither grows an extra parameter. A DataFusion round trip cannot catch this, because the consumer reads output_type only when it converts a cast.
This was referenced Sep 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
The producer left
output_typeunset on window function calls and onLIKE/ILIKE, including thenotthat wraps a negated one. Substrait documents bothExpression.WindowFunction.output_typeandExpression.ScalarFunction.output_typeas:A consumer that reads the field rejects such a call. substrait-java 0.103.0 refuses both plans, from
ProtoTypeConverter.from:125:SELECT sum(i) OVER (ORDER BY i) FROM tUnsupportedOperationException: Type is not setatProtoExpressionConverter.fromWindowFunction:406SELECT i FROM t WHERE CAST(i AS VARCHAR) LIKE '1%'UnsupportedOperationException: Type is not setatProtoExpressionConverter:201, viaProtoRelConverter.newFilterA DataFusion round trip cannot catch this: the consumer reads
output_typeonly inconsumer/expr/cast.rs, so producer and consumer agree on the omission.#15831 and #20597 set this field for binary and unary expressions,
from_functionand higher order functions, and #25049 and #25090 coverAggregateFunction. These were the remaining expression kinds.What changes are included in this PR?
Both types come from the expression itself, so they match what DataFusion derives rather than being restated in the producer:
from_window_functionderives the type fromExpr::WindowFunction(..).to_field(schema)and writes it on the call it builds.make_substrait_window_functionhad one caller and eight parameters once the type was added, so its body now sits infrom_window_functionand the helper is gone.make_substrait_like_exprtakes theLikeitself rather than its five fields, derives the type fromExpr::Like(..).to_field(schema), and sets it on thelikecall and on thenotwrapper of a negated one, which yields that same type.What is the testing strategy for this PR?
binary_expr_output_type:window_function_output_typeassertssum(i)over a nullablei64carries a nullablei64, andlike_output_typeasserts aLIKEover a nullable input carries a nullable boolean, on thelikecall, on thenotof a negated one, and on thelikenested inside thatnot.window_function_output_typefails, without theLIKEchange onlylike_output_typefails.cargo test -p datafusion-substraitpasses (60 unit, 210 integration with the 6 that were already ignored, 3 doc tests), and./ci/scripts/rust_clippy.sh, the workspace clippy CI runs, is clean.sumoutput_typeunsetlikenotaround a negatedlikeType is not set. Both now get throughProtoPlanConverter, while the plans built frommainstill fail there. Each then stops further along for reasons that have nothing to do with this field: substrait-spark has no visitor for a window function that appears in a project expression, and Spark'sliketakes two arguments while the call we emit passes three, the third being the escape character. I am looking at that second one separately.Are there any user-facing changes?
No API changes. Plans produced for window functions and
LIKEnow carry the expression's type, which consumers that require it will accept.