Fix one-tailed Wilcoxon p-values in compute_pvals_wilcoxon - #1177
Open
azrabano23 wants to merge 1 commit into
Open
Fix one-tailed Wilcoxon p-values in compute_pvals_wilcoxon#1177azrabano23 wants to merge 1 commit into
azrabano23 wants to merge 1 commit into
Conversation
compute_pvals_wilcoxon derived the one-tailed p-value by halving the two-sided signed-rank p-value and choosing the tail from the sign of the mean paired difference. The signed-rank test does not test the mean: its direction is given by the rank sums W+/W-, which can disagree with the sign of the mean when a few large differences point one way and many small ones the other. In that case both entries of the p-value matrix are on the wrong side of 0.5, e.g. for differences (+0.02 x6, -0.20) the function reported p = 0.82 for pipeline 1 being better while the one-sided signed-rank test gives p = 0.18. Even when the tails agree, 1 - p/2 differs from the exact one-sided p-value by the point mass at the observed statistic. Take the one-sided p-value directly from scipy.stats.wilcoxon(..., alternative="greater"), available since SciPy 1.5 (moabb requires >= 1.9.3). The all-zero-differences guard and the (0, 1) clipping are unchanged. Closes NeuroTechX#1176 Signed-off-by: Azra Bano <azrabano.work@gmail.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
azrabano23
force-pushed
the
fix-1176-wilcoxon-one-sided
branch
from
September 4, 2026 20:14
5da7f02 to
41e3f0b
Compare
Collaborator
|
@qbarthelemy, i was wondering, can you please review once you have time please 🙏🏽 |
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
The change directly aligns implementation with SciPy’s one-sided Wilcoxon API and is backed by a focused regression test reproducing the reported failure mode.
Pull request overview
This PR fixes the computation of one-tailed Wilcoxon signed-rank p-values in compute_pvals_wilcoxon by using SciPy’s native one-sided test (alternative="greater") instead of deriving a one-sided value from the two-sided p-value and the mean paired difference (which can disagree with the signed-rank statistic’s direction).
Changes:
- Update
compute_pvals_wilcoxonto callscipy.stats.wilcoxon(..., alternative="greater")directly for the “pipe1 > pipe2” direction. - Add a regression test covering the case where mean difference and signed-rank direction disagree (issue #1176 reproducer).
- Add a changelog entry documenting the bug fix.
File summaries
| File | Description |
|---|---|
moabb/tests/test_analysis.py |
Adds a regression test validating that one-tailed p-values follow the signed-rank statistic direction (via SciPy alternatives), not the mean difference. |
moabb/analysis/meta_analysis.py |
Fixes one-tailed Wilcoxon p-value computation by delegating tail selection to SciPy’s alternative="greater" logic. |
docs/source/whats_new.rst |
Documents the Wilcoxon one-tailed p-value fix in the Bugs section. |
Review details
- Files reviewed: 3/3 changed files
- Comments generated: 0
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1176
What was wrong
compute_pvals_wilcoxonbuilt the one-tailed p-value from SciPy's two-sided signed-rank p-value: halve it, then put it on the "pipe1 > pipe2" side if the mean paired difference is positive, else on the other side (1 - p/2). The signed-rank test does not test the mean; its direction comes from the rank sumsW+/W-, which disagree with the sign of the mean whenever a few large differences point one way and many small ones the other.Example, 7 subjects, pipeline 1 wins on 6 by 0.02 and loses on one by 0.20 (SciPy 1.15.3):
developscipy.stats.wilcoxon(x, y, alternative=...)pvals[pipeline_1, pipeline_2]"greater": 0.1797pvals[pipeline_2, pipeline_1]"less": 0.9453So the matrix said pipeline 2 was better (p = 0.18) when the signed-rank test says the opposite (p = 0.18 for pipeline 1). The values feed
compute_dataset_statistics(every dataset with>= perm_cutoffsubjects),find_significant_differencesand the summary / meta-analysis plots. When mean and rank sums agree, the halved value still differs from the exact one-sided p by the point mass at the observed statistic (0.82 vs 0.95 above).The fix
Ask SciPy for the one-sided test directly:
alternativeexists since SciPy 1.5; moabb requiresscipy>=1.9.3. The all-zero-differences guard (0.5) and the(0, 1)clipping from #1134 are unchanged. In the common case where mean and rank sums agree and the exact method is used, results change only by the point-mass term; with the normal approximation used for largern, the two agree whenever the rank sums and the mean point the same way.Verification (CPU only, Python 3.12, macOS arm64)
Also added a "Bugs" entry to
docs/source/whats_new.rst.🤖 Generated with Claude Code