Skip to content

fix: preserve nulls when unnesting structs - #25389

Open
1fanwang wants to merge 1 commit into
apache:mainfrom
1fanwang:1fannnw/fix-null-struct-unnest
Open

1fanwang wants to merge 1 commit into
apache:mainfrom
1fanwang:1fannnw/fix-null-struct-unnest

Conversation

@1fanwang

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #25293.

Rationale for this change

Unnesting a NULL struct can expose its stored child values. A non-nullable child field can also make a valid query fail. Both cases now produce NULL values.

What changes are included in this PR?

Output fields include the parent's nullability. The executor applies the parent mask through Arrow and reuses the original child arrays when the parent has no nulls.

What is the testing strategy for this PR?

SQL and native-array regressions cover schema nullability and child validity, including nested and encoded children.

Run these commands from the repository root:

cargo run --locked --profile ci -p datafusion-cli -- --quiet --format json \
  -c "CREATE TABLE base AS SELECT named_struct('a', 1, 'b', 'x') AS s; CREATE TABLE masked AS SELECT nullif(s, s) AS s FROM base; SELECT unnest(s) FROM masked;"
cargo run --locked --profile ci -p datafusion-cli -- --quiet --format json \
  -c "SELECT unnest(arrow_cast(NULL, 'Struct(\"a\": non-null Int32)'));"

The JSON writer omits NULL-valued fields.

Before, the first query exposes values and the second fails:

[{"masked.s.a":1,"masked.s.b":"x"}]
Error: Arrow error: Invalid argument error: Column '__unnest_placeholder(arrow_cast(NULL,Utf8("Struct("a": non-null Int32)"))).a' is declared as non-nullable but contains null values

After the same two commands:

[{}]
[{}]
Raw schema diagnostic

The SQL regression also catches the incorrect output nullability before the fix:

cargo test --locked --profile ci --test sqllogictests -- unnest_struct_nulls.slt

Raw before-fix excerpt:

3. query result mismatch:
[SQL] SELECT is_nullable FROM information_schema.columns WHERE table_name = 'nullable_fields';
[Diff] (-expected|+actual)
-   YES
+   NO

Are there any user-facing changes?

Yes. NULL structs now unnest into NULL fields rather than values or an error.

Signed-off-by: 1fanwang <1fannnw@gmail.com>
@github-actions github-actions Bot added logical-expr Logical plan and expressions sqllogictest SQL Logic Tests (.slt) physical-plan Changes to the physical-plan crate labels Sep 16, 2026
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 89.70588% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.93%. Comparing base (140c7c5) to head (6e16f70).

Files with missing lines Patch % Lines
datafusion/physical-plan/src/unnest.rs 88.13% 4 Missing and 3 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #25389      +/-   ##
==========================================
- Coverage   81.93%   81.93%   -0.01%     
==========================================
  Files        1136     1136              
  Lines      429152   429215      +63     
  Branches   429152   429215      +63     
==========================================
+ Hits       351633   351679      +46     
- Misses      56475    56487      +12     
- Partials    21044    21049       +5     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

logical-expr Logical plan and expressions physical-plan Changes to the physical-plan crate sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

unnest of a struct ignores the struct's null mask

2 participants