fix(mutation): reject zero-test survivor evidence - #799
Conversation
Stryker 9.6.1's Vitest runner can complete a per-test mutant run with zero selected tests when a test-name filter resolves empty. The aggregator previously accepted that result as a Survived mutant, which could inflate mutation evidence and hide broken test selection. Validate raw file reports before trusting metrics and require every Survived mutant to report at least one completed test. Keep metrics-only fixtures valid for the aggregate test contract; existing status and metric relationship validation remain unchanged. This is a bounded qualification guard. It does not upgrade Stryker, change mutation thresholds, alter workflow mode, or change production test selection. Validation: - pnpm exec vitest run tests/unit/tooling/strykerAggregation.test.ts tests/unit/tooling/strykerWorkflowPolicy.test.ts - pnpm exec biome check scripts/aggregate-stryker-reports.mjs tests/unit/tooling/strykerAggregation.test.ts - git diff --check - pnpm run signing:doctor - pnpm run hooks:install
📝 WalkthroughWalkthroughChangesMutation aggregation now validates survived-mutant execution evidence before reading or deriving metrics. Tests cover valid and zero-test cases. The changelog documents the fix. README metrics and Vitest references now reflect the updated repository values. Mutation evidence validation
Repository test metrics
Priority: ➖ Normal Severity of issue fixed: Medium Merge Risk: 🔵 Low · up to The current README remains valid, but future grouped test counts could become stale without detection. This is a bounded, straightforward documentation-check fix. Comment |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Reviewer's GuideThis draft reserves PR #764 while introducing a guard in Stryker report aggregation that prevents survived mutants without completed-test evidence from being treated as valid mutation results, with corresponding unit-test coverage and README test-count updates. Flow diagram for Stryker mutation-evidence validationflowchart TD
A[Read Stryker report] --> B[validateMutationExecutionEvidence]
B --> C{Survived mutant has testsCompleted >= 1?}
C -->|Yes| D[readReportMetrics]
C -->|No| E[Throw validation error]
D --> F[Aggregate mutation metrics]
File-Level Changes
Possibly linked issues
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
🏁 CodeAnt Quality Gate ResultsCommit: ✅ Overall Status: PASSEDQuality Gate Details
|
![codescene-access[bot]](https://avatars.githubusercontent.com/in/3755875?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
Record the actual GitHub PR number in the Unreleased mutation-tooling note so the PR CHANGELOG Reference Guard can validate the release note. This bootstrap correction intentionally changes CHANGELOG.md only; no source, test, workflow, or governance behavior is included. Validation: - git diff --check - PR budget prospective check against origin/main
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
qnbs
changed the title
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
🤖 CodeAnt AI — Review Status
|
Thanks for using CodeAnt! 🎉We're free for open-source projects. if you're enjoying it, help us grow by sharing. Share on X · |
codeant-ai
Bot
added
the
size:M
CodeAnt Nitpicks1 code suggestion1. The README claims Vitest 4.x, but package.json declares Vitest ^5.0.0; the metrics sync script also preserves this incorrect version.Inconsistent naming · |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
CodeScene identified nested conditional logic in the new mutation-evidence validator, while CodeAnt identified stale Vitest 4.x documentation that conflicted with the repository's Vitest 5 toolchain. Split report, file, and mutant validation into bounded helpers without changing fail-closed semantics. Update the README and both source-derived metric parsers to recognize and report Vitest 5.x. The zero-test survivor guard, existing metric invariants, mutation thresholds, Stryker version, and workflow policy remain unchanged. This is the single bounded correction wave for the exact current review findings. Validation:\n- focused Vitest aggregation and workflow-policy tests: 11 passed\n- docs:check: OK\n- Biome: clean\n- git diff --check: clean\n- PR budget: 4 files, 60 meaningful lines, 2 commits; NORMAL
There was a problem hiding this comment.
Gates Passed
See analysis details in CodeScene
Absence of Expected Change Pattern
- WorldScript-Studio/scripts/check-doc-metrics.mjs is usually changed with: WorldScript-Studio/tests/unit/checkDocMetrics.test.ts
Quality Gate Profile: The Bare Minimum
Install CodeScene MCP: safeguard and uplift AI-generated code. Catch issues early with our IDE extension and CLI tool.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository: qnbs/WorldScript-Studio/.coderabbit.yaml
Review profile: CHILL
Plan: Essentials
Run ID: 9800b1dd-634e-4955-96cd-cbbd3a0ea215
📒 Files selected for processing (4)
README.mdscripts/aggregate-stryker-reports.mjsscripts/check-doc-metrics.mjsscripts/sync-readme-metrics.mjs
🚧 Files skipped from review as they are similar to previous changes (1)
- README.md
Included review availability: 3 reviews are currently available. Your included PR review attempts over the past 7 days set your current allowance at 5 reviews per hour.
|
CodeAnt Nitpick validated and addressed in corrective commit |
1 participant
User description
Bootstrap draft for the bounded #764 Stryker/Vitest 5 mutation-evidence qualification. The PR number is reserved so CHANGELOG.md can receive its real release-note reference before governance checks and review.
Summary by Sourcery
Fail closed on zero-test survived mutation evidence during Stryker report aggregation.
Bug Fixes:
Documentation:
Tests:
Chores:
Summary by cubic
Rejects zero-test survived mutants in Stryker aggregation so empty Vitest filter results no longer count as valid mutation evidence.
Written for commit 66d7254. Summary will update on new commits.
CodeAnt-AI Description
Reject invalid zero-test mutation survivors
What Changed
Impact
✅ Prevented false mutation results✅ Clearer mutation report failures✅ Updated test coverage documentation💡 Usage Guide
Checking Your Pull Request
Every time you make a pull request, our system automatically looks through it. We check for security issues, mistakes in how you're setting up your infrastructure, and common code problems. We do this to make sure your changes are solid and won't cause any trouble later.
Talking to CodeAnt AI
Got a question or need a hand with something in your pull request? You can easily get in touch with CodeAnt AI right here. Just type the following in a comment on your pull request, and replace "Your question here" with whatever you want to ask:
This lets you have a chat with CodeAnt AI about your pull request, making it easier to understand and improve your code.
Example
Preserve Org Learnings with CodeAnt
You can record team preferences so CodeAnt AI applies them in future reviews. Reply directly to the specific CodeAnt AI suggestion (in the same thread) and replace "Your feedback here" with your input:
This helps CodeAnt AI learn and adapt to your team's coding style and standards.
Example
Retrigger review
Ask CodeAnt AI to review the PR again, by typing:
Check Your Repository Health
To analyze the health of your code repository, visit our dashboard at https://app.codeant.ai. This tool helps you identify potential issues and areas for improvement in your codebase, ensuring your repository maintains high standards of code health.
Summary by CodeRabbit
Bug Fixes
Documentation