Add JUnit report generation support to test runner - #72
swissspidy wants to merge 2 commits into
Conversation
The progress output prints one character per step, which says how many scenarios failed but not which ones. That makes it hard to compare two runs, such as a run against a WordPress build under test and a run against the build it is based on, in order to tell which failures the change under test actually caused. Setting `WP_CLI_TEST_JUNIT_DIR` now additionally writes JUnit reports to the given directory. Those name every scenario along with the file and line it is defined on. Failed scenarios are run a second time. Reports of that rerun go to a `rerun` subdirectory, as they only cover the scenarios that failed the first time and would otherwise overwrite the full report of the package. Keeping both makes it possible to tell a flaky scenario, which fails once and then passes, from one that consistently fails. The progress output is unchanged, and nothing happens unless the environment variable is set. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CE81GsxUY597AdaMP1NXzk
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
🚧 Files skipped from review as they are similar to previous changes (2)
Included review availability: Your plan includes up to 1 review per rolling hour; 0 remain after this review. 📝 WalkthroughWalkthroughThe test script now supports optional JUnit reports for initial runs and failed-scenario reruns. It removes stale reports, creates separate report locations, and logs the configured destination. The README documents configuration, naming, storage, and flaky-scenario reporting. ChangesJUnit reporting
Estimated code review effort: 2 (Simple) | ~10 minutes Merge Risk: ⚪ Minimal · up to The change adds optional JUnit report output while preserving existing test-runner behavior when the setting is absent; no actionable merge-blocking risk remains beyond normal checks. 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Pull request overview
Adds optional JUnit XML report generation for Behat-based test runs to improve CI/CD integration and result tracking, while keeping existing progress output behavior.
Changes:
- Documented a new
WP_CLI_TEST_JUNIT_DIRenvironment variable and the report/rerun directory structure inREADME.md. - Updated
bin/test-sourceto emit JUnit reports (including rerun reports under arerun/subdirectory) whenWP_CLI_TEST_JUNIT_DIRis set.
Reviewed changes
Copilot reviewed 1 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| README.md | Adds usage docs and explains how initial vs rerun JUnit reports are organized. |
| bin/test-source | Adds conditional Behat CLI arguments and directory creation to generate JUnit XML reports (plus rerun reports). |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@bin/test-source`:
- Around line 55-61: Update the JUnit setup around JUNIT_ARGS and
JUNIT_RERUN_ARGS to remove stale XML reports owned by this runner before the
test loop begins, including prior files under the rerun directory. Preserve the
existing output paths and avoid deleting unrelated files in
WP_CLI_TEST_JUNIT_DIR.
- Around line 17-21: The flaky-detection documentation must use testcase status
transitions: in bin/test-source lines 17-21, rewrite the shell comments to
compare matching testcase statuses, identifying failed-then-passed as flaky; in
README.md line 49, document failed-to-passed as flaky and failed-to-failed as
consistently failing.
In `@README.md`:
- Line 44: Update the report-path code block in README.md to declare the text
language by changing its opening fence to ```text, while preserving the listed
paths and closing fence.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: e6e97d94-1ff3-4a58-9fa6-e43cc4cf342c
📒 Files selected for processing (2)
README.mdbin/test-source
The rerun report holds a result for every scenario it ran, including the ones that passed the second time, rather than omitting them as the documentation claimed. A flaky scenario is therefore one reported as failed in the report of the package and as passed in the rerun report, not one that is missing from the latter. Reports of an earlier run are now removed before the tests start. A package that passes produces no rerun report at all, so a report left behind by an earlier run would have been read as belonging to the current one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CE81GsxUY597AdaMP1NXzk
|
@coderabbitai review |
✅ Action performedReview finished.
|
schlessera
left a comment
schlessera
left a comment
There was a problem hiding this comment.
Approving. Checked against the installed Behat: --format/--out pair by index (Testwork OutputController::configureOutputs), std keeps progress on stdout, the JUnit file is <dir>/<sanitised suite name>.xml, and the --rerun cache key ignores --format/--out, so the rerun still picks up exactly the scenarios the first run recorded. With the variable unset the two behat command lines are unchanged.
Two small things, neither blocking:
bin/test-source:58removes every*.xmlin the directory and inrerun/, not only reports. The README says "reports left behind by an earlier run are removed"; say that all XML files there are deleted, so the directory should be dedicated to these reports.--rerunruns everything when the first run recorded no failed scenario (Behat aborting before the suite runs, for example). In that case thererun/report covers the whole package and the flaky/failing comparison does not apply. Half a sentence in the README if you want to cover it.
No test exercises the new path; CI never sets WP_CLI_TEST_JUNIT_DIR. Follow-up worth doing: set it in the "Run Behat" step of reusable-testing.yml and upload the directory as an artifact. That runs the code on every job and gives the reports a consumer. bin/test-phar:103-105 has the same two invocations and gets no reports, and the use-phar: true half of the matrix runs through it; same follow-up.
Unrelated, pre-existing: line 40 reads wp-cli-/wp-cli, which is not a suite in behat.yml. Separate PR.
Merge when you like; fix the README wording in this PR or after, your call.
Summary
This change adds support for generating JUnit XML reports from Behat test runs, enabling better integration with CI/CD systems and test result tracking.
Key Changes
$WP_CLI_TEST_JUNIT_DIRenvironment variable support to optionally write JUnit reports to a specified directoryrerunsubdirectory to distinguish between flaky and truly failing testsImplementation Details
$WP_CLI_TEST_JUNIT_DIRis set, the script creates the directory structure (includingrerunsubdirectory) and constructs appropriate Behat format arguments--format junit --out std --out <dir>pattern is used to maintain progress output on stdout while writing JUnit XML to the specified directoryhttps://claude.ai/code/session_01CE81GsxUY597AdaMP1NXzk
Summary by CodeRabbit
New Features
Documentation