Skip to content

fix: preserve total_byte_size in calculate_total_byte_size when num_r… - #24027

Open
bert-beyondloops wants to merge 1 commit into
apache:mainfrom
bert-beyondloops:stats_total_byte_size_calculation
Open

fix: preserve total_byte_size in calculate_total_byte_size when num_r…#24027
bert-beyondloops wants to merge 1 commit into
apache:mainfrom
bert-beyondloops:stats_total_byte_size_calculation

Conversation

@bert-beyondloops

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

Statistics::calculate_total_byte_size is meant to derive total_byte_size from num_rows and the schema's fixed-width columns. When all columns have a primitive width but num_rows is Precision::Absent, the old code computed
self.num_rows.multiply(&Precision::Exact(size)), and Precision::multiply returns Precision::Absent whenever either operand is Absent. This silently overwrote any previously known total_byte_size (exact or inexact) with
Absent, even though the non-primitive-width branch already handled this situation correctly by downgrading the existing value to inexact instead of discarding it.

What changes are included in this PR?

  • In Statistics::calculate_total_byte_size, when the schema is all fixed-width but num_rows is Precision::Absent, keep the existing total_byte_size and downgrade it to inexact via to_inexact(), instead of overwriting it with
    Absent.
  • Updated the doc comment on calculate_total_byte_size to describe this behavior.
  • Added test_calculate_total_byte_size covering: an all-primitive schema with known row count (exact size), an all-primitive schema with unknown row count (preserved but downgraded to inexact), and a non-primitive schema
    (always downgraded to inexact regardless of row count).

Are these changes tested?

Yes — added `stats::tests::test_calculate_total_byte_size, exercising all three branches of the updated match.

Are there any user-facing changes?

No public API changes. Statistics propagation is more accurate (previously known total_byte_size estimates are no longer dropped to Absent when num_rows is unknown), which may result in slightly better cost-based planning
decisions in some cases.

@github-actions github-actions Bot added the common Related to common crate label Jul 31, 2026
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.85%. Comparing base (dbcb5c0) to head (5496845).

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #24027      +/-   ##
==========================================
- Coverage   80.85%   80.85%   -0.01%     
==========================================
  Files        1101     1101              
  Lines      374933   374951      +18     
  Branches   374933   374951      +18     
==========================================
+ Hits       303166   303177      +11     
- Misses      53671    53675       +4     
- Partials    18096    18099       +3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

match row_size {
None => {
match (row_size, &self.num_rows) {
(None, _) | (Some(_), Precision::Absent) => {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change makes sense to me since it avoids downgrading Exact total_byte_size to absent when num_rows is Absent.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Intuitively this makes sense to me too, hopefully we will soon be able to have a more data-driven approach thanks to #23975.

@asolimando asolimando left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

common Related to common crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Statistics::calculate_total_byte_size discards a known total_byte_size when num_rows is unknown

4 participants