Skip to content

Fix metric() showing an extra significant digit on rounding carry#359

Open
vidigoat wants to merge 1 commit into
python-humanize:mainfrom
vidigoat:fix-metric-significant-digits
Open

Fix metric() showing an extra significant digit on rounding carry#359
vidigoat wants to merge 1 commit into
python-humanize:mainfrom
vidigoat:fix-metric-significant-digits

Conversation

@vidigoat

Copy link
Copy Markdown

Summary

metric() can show one significant figure too many when rounding carries the mantissa up a power of ten.

>>> import humanize
>>> humanize.metric(9999)
'10.00 k'      # 4 significant figures at the default precision=3
>>> humanize.metric(99999)
'100.0 k'      # 4 significant figures
>>> humanize.metric(9.99999)
'10.00'

The neighbours prove the inconsistency — same magnitude, correct 3 significant figures:

>>> humanize.metric(10000)
'10.0 k'
>>> humanize.metric(100000)
'100 k'

Cause

The number of decimal places is derived from the mantissa's position in its SI bucket (digits = precision - exponent % 3 - 1), which assumes the mantissa keeps its integer-digit count. When rounding to digits places carries the mantissa up a power of ten (9.999 → 10.0, 99.99 → 100), it gains an integer digit and therefore shows an extra significant figure.

The existing guard only handled the mantissa rounding all the way to 1000 (a full bucket crossing, e.g. 999.9 → 1.00 k); it never handled the → 10 and → 100 crossings inside a bucket.

Fix

Compare the rounded mantissa against the next power of ten (10 ** (exponent % 3 + 1)) rather than the hard-coded 1000, and bump the exponent by one to absorb the carry. When the mantissa reaches 1000 this still crosses into the next SI bucket exactly as before, so all existing outputs (including metric(999.9, "V") == "1.00 kV") are unchanged.

This matches the documented contract that the prefix is chosen "so that non-significant zero digits are required" and that precision is "the number of digits the output should contain."

Test cases for the within-bucket carry are added to test_metric; the full suite passes.

metric() derives the number of decimal places from the mantissa's
exponent, assuming the mantissa keeps its digit count. When rounding
carries it up a power of ten (9.999 -> 10.0, 99.99 -> 100) it gains an
integer digit and shows one significant figure too many, e.g.
metric(9999) returned '10.00 k' instead of '10.0 k'. The existing guard
only handled the mantissa reaching 1000 (a full SI-bucket crossing).

Detect the carry against the next power of ten and bump the exponent by
one, which recomputes the decimal places and, when the mantissa reaches
1000, still crosses into the next bucket exactly as before.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant