Fix metric() showing an extra significant digit on rounding carry#359
Open
vidigoat wants to merge 1 commit into
Open
Fix metric() showing an extra significant digit on rounding carry#359vidigoat wants to merge 1 commit into
vidigoat wants to merge 1 commit into
Conversation
metric() derives the number of decimal places from the mantissa's exponent, assuming the mantissa keeps its digit count. When rounding carries it up a power of ten (9.999 -> 10.0, 99.99 -> 100) it gains an integer digit and shows one significant figure too many, e.g. metric(9999) returned '10.00 k' instead of '10.0 k'. The existing guard only handled the mantissa reaching 1000 (a full SI-bucket crossing). Detect the carry against the next power of ten and bump the exponent by one, which recomputes the decimal places and, when the mantissa reaches 1000, still crosses into the next bucket exactly as before.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
metric()can show one significant figure too many when rounding carries the mantissa up a power of ten.The neighbours prove the inconsistency — same magnitude, correct 3 significant figures:
Cause
The number of decimal places is derived from the mantissa's position in its SI bucket (
digits = precision - exponent % 3 - 1), which assumes the mantissa keeps its integer-digit count. When rounding todigitsplaces carries the mantissa up a power of ten (9.999 → 10.0,99.99 → 100), it gains an integer digit and therefore shows an extra significant figure.The existing guard only handled the mantissa rounding all the way to
1000(a full bucket crossing, e.g.999.9 → 1.00 k); it never handled the→ 10and→ 100crossings inside a bucket.Fix
Compare the rounded mantissa against the next power of ten (
10 ** (exponent % 3 + 1)) rather than the hard-coded1000, and bump the exponent by one to absorb the carry. When the mantissa reaches1000this still crosses into the next SI bucket exactly as before, so all existing outputs (includingmetric(999.9, "V") == "1.00 kV") are unchanged.This matches the documented contract that the prefix is chosen "so that non-significant zero digits are required" and that
precisionis "the number of digits the output should contain."Test cases for the within-bucket carry are added to
test_metric; the full suite passes.