Skip to content

Fix one-tailed Wilcoxon p-values in compute_pvals_wilcoxon - #1177

Open
azrabano23 wants to merge 1 commit into
NeuroTechX:developfrom
azrabano23:fix-1176-wilcoxon-one-sided
Open

Fix one-tailed Wilcoxon p-values in compute_pvals_wilcoxon#1177
azrabano23 wants to merge 1 commit into
NeuroTechX:developfrom
azrabano23:fix-1176-wilcoxon-one-sided

Conversation

@azrabano23

Copy link
Copy Markdown
Contributor

Fixes #1176

What was wrong

compute_pvals_wilcoxon built the one-tailed p-value from SciPy's two-sided signed-rank p-value: halve it, then put it on the "pipe1 > pipe2" side if the mean paired difference is positive, else on the other side (1 - p/2). The signed-rank test does not test the mean; its direction comes from the rank sums W+/W-, which disagree with the sign of the mean whenever a few large differences point one way and many small ones the other.

Example, 7 subjects, pipeline 1 wins on 6 by 0.02 and loses on one by 0.20 (SciPy 1.15.3):

current develop scipy.stats.wilcoxon(x, y, alternative=...)
pvals[pipeline_1, pipeline_2] 0.8203 "greater": 0.1797
pvals[pipeline_2, pipeline_1] 0.1797 "less": 0.9453

So the matrix said pipeline 2 was better (p = 0.18) when the signed-rank test says the opposite (p = 0.18 for pipeline 1). The values feed compute_dataset_statistics (every dataset with >= perm_cutoff subjects), find_significant_differences and the summary / meta-analysis plots. When mean and rank sums agree, the halved value still differs from the exact one-sided p by the point mass at the observed statistic (0.82 vs 0.95 above).

The fix

Ask SciPy for the one-sided test directly:

p = stats.wilcoxon(df.loc[:, pipe1], df.loc[:, pipe2], alternative="greater")[1]

alternative exists since SciPy 1.5; moabb requires scipy>=1.9.3. The all-zero-differences guard (0.5) and the (0, 1) clipping from #1134 are unchanged. In the common case where mean and rank sums agree and the exact method is used, results change only by the point-mass term; with the normal approximation used for larger n, the two agree whenever the rank sums and the mean point the same way.

Verification (CPU only, Python 3.12, macOS arm64)

python -m pytest moabb/tests/test_analysis.py -k "TestStats or Corrected"   # 36 passed
# the new test test_wilcoxon_tail_follows_rank_sums_not_mean fails on develop:
#   assert np.isclose(pvals[0, 1], expected_greater)  ->  0.8203 vs 0.1797
ruff check / ruff format --check on the two changed files                  # clean

Also added a "Bugs" entry to docs/source/whats_new.rst.

🤖 Generated with Claude Code

compute_pvals_wilcoxon derived the one-tailed p-value by halving the
two-sided signed-rank p-value and choosing the tail from the sign of
the mean paired difference. The signed-rank test does not test the
mean: its direction is given by the rank sums W+/W-, which can
disagree with the sign of the mean when a few large differences point
one way and many small ones the other. In that case both entries of
the p-value matrix are on the wrong side of 0.5, e.g. for differences
(+0.02 x6, -0.20) the function reported p = 0.82 for pipeline 1 being
better while the one-sided signed-rank test gives p = 0.18. Even when
the tails agree, 1 - p/2 differs from the exact one-sided p-value by
the point mass at the observed statistic.

Take the one-sided p-value directly from
scipy.stats.wilcoxon(..., alternative="greater"), available since
SciPy 1.5 (moabb requires >= 1.9.3). The all-zero-differences guard
and the (0, 1) clipping are unchanged.

Closes NeuroTechX#1176

Signed-off-by: Azra Bano <azrabano.work@gmail.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@azrabano23
azrabano23 force-pushed the fix-1176-wilcoxon-one-sided branch from 5da7f02 to 41e3f0b Compare September 4, 2026 20:14
@bruAristimunha
bruAristimunha requested a lite review from Copilot September 5, 2026 08:17
@bruAristimunha

Copy link
Copy Markdown
Collaborator

@qbarthelemy, i was wondering, can you please review once you have time please 🙏🏽

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The change directly aligns implementation with SciPy’s one-sided Wilcoxon API and is backed by a focused regression test reproducing the reported failure mode.

Pull request overview

This PR fixes the computation of one-tailed Wilcoxon signed-rank p-values in compute_pvals_wilcoxon by using SciPy’s native one-sided test (alternative="greater") instead of deriving a one-sided value from the two-sided p-value and the mean paired difference (which can disagree with the signed-rank statistic’s direction).

Changes:

  • Update compute_pvals_wilcoxon to call scipy.stats.wilcoxon(..., alternative="greater") directly for the “pipe1 > pipe2” direction.
  • Add a regression test covering the case where mean difference and signed-rank direction disagree (issue #1176 reproducer).
  • Add a changelog entry documenting the bug fix.
File summaries
File Description
moabb/tests/test_analysis.py Adds a regression test validating that one-tailed p-values follow the signed-rank statistic direction (via SciPy alternatives), not the mean difference.
moabb/analysis/meta_analysis.py Fixes one-tailed Wilcoxon p-value computation by delegating tail selection to SciPy’s alternative="greater" logic.
docs/source/whats_new.rst Documents the Wilcoxon one-tailed p-value fix in the Bugs section.
Review details
  • Files reviewed: 3/3 changed files
  • Comments generated: 0
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

compute_pvals_wilcoxon picks the one-sided tail from the mean difference, not from the signed-rank statistic

3 participants