feat: add WelfordAlgorithm, online mean and variance in one pass - #7602
Open
alxkm wants to merge 1 commit into
Open
feat: add WelfordAlgorithm, online mean and variance in one pass#7602alxkm wants to merge 1 commit into
alxkm wants to merge 1 commit into
Conversation
Welford's recurrence keeps a running mean and the sum of squared deviations from it, so it never forms the large nearly equal intermediate values that make the textbook variance formula lose its significant digits, and it needs O(1) time per sample and O(1) memory regardless of the stream length. Beyond the plain accumulation it supports removal, which runs the recurrence backwards and turns the accumulator into the statistics of a sliding window, and a static merge implementing Chan's parallel update so partial results from different shards combine exactly. Signed-off-by: alxkm <19151554+alxkm@users.noreply.github.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #7602 +/- ##
============================================
+ Coverage 81.01% 81.03% +0.02%
- Complexity 7740 7768 +28
============================================
Files 825 826 +1
Lines 24594 24645 +51
Branches 4810 4814 +4
============================================
+ Hits 19925 19972 +47
- Misses 3906 3907 +1
- Partials 763 766 +3 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds Welford's algorithm: the mean and the variance of a stream in a single pass, in O(1) time per sample and O(1) memory however long the stream is.
The textbook formula
Var = (sum(x^2) - n * mean^2) / (n - 1)is fast but numerically treacherous --sum(x^2)andn * mean^2can be huge and nearly equal, so their difference loses most of its significant digits and may even come out negative. Welford's recurrence never forms those intermediate values; it carries the running mean andM2, the sum of squared deviations from that running mean:Two things come with the accumulator:
remove(double)runs the recurrence backwards, which turns it into the statistics of a sliding window: hand the incoming sample toaddand the sample that just left the window toremove. The Javadoc is explicit that this is the one operation that can degrade over a very long run, since the removed value no longer matches the mean it was added to, and that the accumulator should be recreated periodically when that matters.merge(WelfordAlgorithm, WelfordAlgorithm)is Chan's parallel update, so partial summaries computed on different shards combine exactly rather than approximately.The class exposes both the sample and the population variance and standard deviation, plus the standard error,
sumandsumOfSquaredDeviations, and it refuses non-finite samples rather than silently poisoning the state with NaN.WelfordAlgorithmTestcovers 20 cases. Among them: the results are compared against a two-pass computation for several sample counts, a stream offset by 1e9 stays accurate where the naive sum of squares collapses,removeexactly undoesadd, a sliding window is checked against recomputation from scratch, removing a sample far from the mean cannot push the variance below zero,mergeis checked against the same data accumulated in one go and with empty operands, and an empty accumulator reports NaN instead of dividing by zero.Checklist
clang-format -i --style=file path/to/your/file.java