Skip to content

feat: recognize more lossless casts for statistics and ordering - #25500

Open
haohuaijin wants to merge 2 commits into
apache:mainfrom
haohuaijin:fix/lossless-unsigned-to-signed-casts
Open

haohuaijin wants to merge 2 commits into
apache:mainfrom
haohuaijin:fix/lossless-unsigned-to-signed-casts

Conversation

@haohuaijin

@haohuaijin haohuaijin commented Sep 19, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Follow-up to #25227 and its discussion.

Rationale for this change

Some lossless casts unnecessarily discard column statistics or fail to preserve strict ordering information.

For example, UInt32 → Int64 currently requires both bounds to be exact to propagate statistics, even though every UInt32 value fits in Int64. Recognizing this conversion as globally safe preserves available statistics even when one bound is absent or bounds are inexact.

What changes are included in this PR?

  • Recognize unsigned-to-signed widening casts: UInt8 → Int16 / Int32 / Int64, UInt16 → Int32 / Int64, and UInt32 → Int64.
  • Recognize Utf8 → Utf8View and Binary → LargeBinary / BinaryView.
  • Extend the existing check_bigger_cast / is_bigger_cast predicates to recognize these lossless conversions.

What is the testing strategy for this PR?

  • Test values, nulls, integer extrema, and strict ordering properties.
  • Test empty values, Unicode, long strings, and non-UTF8 binary values.
  • Test statistics propagation with exact, inexact, and missing bounds.
  • Verify unsafe reverse conversions remain excluded.

Validation on the latest revision: 25 relevant unit tests passed; one existing test is ignored. Formatting, all-target/all-feature Clippy, and the standard rust_lint.sh suite passed.

Are there any user-facing changes?

More lossless casts retain statistics and strict ordering information, enabling existing optimizer rules.

@github-actions github-actions Bot added physical-expr Changes to the physical-expr crates auto detected api change Auto detected API change labels Sep 19, 2026
@github-actions github-actions Bot removed the auto detected api change Auto detected API change label Sep 19, 2026
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.46465% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.38%. Comparing base (ccfe704) to head (28ede8a).

Files with missing lines Patch % Lines
datafusion/physical-expr/src/expressions/cast.rs 95.62% 1 Missing and 5 partials ⚠️
datafusion/physical-expr/src/projection.rs 98.36% 1 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff            @@
##             main   #25500    +/-   ##
========================================
  Coverage   82.38%   82.38%            
========================================
  Files        1138     1138            
  Lines      433733   433927   +194     
  Branches   433733   433927   +194     
========================================
+ Hits       357331   357509   +178     
- Misses      54848    54854     +6     
- Partials    21554    21564    +10     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

physical-expr Changes to the physical-expr crates

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants