From 82eb20e6e3b96854a80036924648f49ea314dc0d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jakub=20Jare=C5=A1?= Date: Sat, 29 Aug 2026 09:29:24 +0200 Subject: [PATCH] Say how the exit code behaves in the CI we actually build The test script page teaches that the exit code is the number of failures and proves it with pwsh -File. The GitHub Actions step on the next page runs pwsh -command, which normalizes any non-zero exit to 1, so the count does not survive into the pipeline the tutorial builds. The build still goes red, but we were explaining it with the wrong mechanism. Say that the coverage target is not a gate. Coming in under CoveragePercentTarget does not fail the run and does not change the exit code, and "the target you are being measured against" reads like a gate to anybody setting up a pipeline. Create the .gitignore. Three pages said the artifacts belong in it and none of them created it, while the checklist claimed both files were ignored. Following the tutorial literally committed testResults.xml and coverage.xml. Mention that Pester already writes GitHub annotations through Output.CIFormat before recommending an action that does something similar. Add culture to the cross-platform note. It is the bug Planetarium actually has, the report reads "Mercury 0,387 AU" on a machine with a comma decimal separator, and the suite stays green because the only asserted line has no decimal part. Fix #462 --- tutorial/2-testing-a-module/5-output.mdx | 2 +- tutorial/6-code-coverage/1-measuring.mdx | 2 ++ tutorial/6-code-coverage/2-closing-the-gaps.mdx | 2 +- tutorial/7-ci/1-test-script.mdx | 11 +++++++++-- tutorial/7-ci/2-github-actions.mdx | 10 +++++++++- 5 files changed, 22 insertions(+), 5 deletions(-) diff --git a/tutorial/2-testing-a-module/5-output.mdx b/tutorial/2-testing-a-module/5-output.mdx index 83a0217..7a4af59 100644 --- a/tutorial/2-testing-a-module/5-output.mdx +++ b/tutorial/2-testing-a-module/5-output.mdx @@ -136,7 +136,7 @@ Invoke-Pester -Configuration $config The default format is `NUnitXml`, which nearly every CI system understands. `NUnit2.5`, `NUnit3` and `JUnitXml` are the other values for `TestResult.OutputFormat`. See [Test results](../../docs/usage/test-results) for which to pick. -The file is a build artifact — something a run produces, not source. If you keep your projects under version control, add it to your `.gitignore`: +The file is a build artifact — something a run produces, not source. If you keep your projects under version control it belongs in `.gitignore`. The CI module creates that file, for now just know the line it will need: ``` testResults.xml diff --git a/tutorial/6-code-coverage/1-measuring.mdx b/tutorial/6-code-coverage/1-measuring.mdx index bf2cd76..6a3eead 100644 --- a/tutorial/6-code-coverage/1-measuring.mdx +++ b/tutorial/6-code-coverage/1-measuring.mdx @@ -40,6 +40,8 @@ Covered 100% / 75%. 36 analyzed Commands in 7 Files. Read that last line carefully — it trips people up. **`100%` is what you achieved; `75%` is the target you are being measured against.** The target is `CodeCoverage.CoveragePercentTarget`, which defaults to 75. +It is a number to compare against, not a gate. Coming in under it does not fail the run and does not change the exit code, so a pipeline will not go red on coverage alone. If you want that, compare the number yourself from the result object. + :::warning Make sure `CodeCoverage.Path` points to your code `Run.Path` says which *tests* to run; `CodeCoverage.Path` says which *code* to measure. It will default to `Run.Path` which works in this tutorial, but might return 0% coverage if you used a dedicated `tests` folder. Set both options explicitly to avoid surprises. ::: diff --git a/tutorial/6-code-coverage/2-closing-the-gaps.mdx b/tutorial/6-code-coverage/2-closing-the-gaps.mdx index aa29e00..f31689e 100644 --- a/tutorial/6-code-coverage/2-closing-the-gaps.mdx +++ b/tutorial/6-code-coverage/2-closing-the-gaps.mdx @@ -141,7 +141,7 @@ Invoke-Pester -Configuration $config The default format is **JaCoCo**, which most coverage services understand. `Cobertura` is the other option, via `CodeCoverage.OutputFormat`. -Like the test results file, it is a build artifact — if your project is under version control, `.gitignore` it alongside `testResults.xml`. +Like the test results file, it is a build artifact, and it goes into `.gitignore` alongside `testResults.xml`. The [CI module](../ci/test-script) creates that file. ## Using coverage well diff --git a/tutorial/7-ci/1-test-script.mdx b/tutorial/7-ci/1-test-script.mdx index eb12498..0af45f3 100644 --- a/tutorial/7-ci/1-test-script.mdx +++ b/tutorial/7-ci/1-test-script.mdx @@ -54,6 +54,13 @@ Exit code 1. *That* is what makes CI red. Put the `8` back. The code is in fact the **number of failures** — one failing test exits with `1`, five with `5`. Handy at a glance, though the results file is where the detail lives. +That last part depends on how the script is started. `pwsh -File` passes the code through as it is. `pwsh -command`, which is what the GitHub Actions step on the next page uses, normalizes any non-zero exit to `1`. The build still goes red either way, you just do not get the count: + +``` +pwsh -NoProfile -File ./test.ps1 # 3 failures, exit code 3 +pwsh -NoProfile -command ". './test.ps1'" # 3 failures, exit code 1 +``` + :::note `Invoke-Pester -CI` is the shorthand — but not for this Pester has a `-CI` switch that sets exactly two things: @@ -85,9 +92,9 @@ Two files come out of every run: - `testResults.xml` — NUnit format, per-test results - `coverage.xml` — JaCoCo format, coverage data -Both are build output, so keep them out of version control: +Both are build output, so keep them out of version control. Create a `.gitignore` next to `test.ps1`: -``` +```text title=".gitignore" testResults.xml coverage.xml ``` diff --git a/tutorial/7-ci/2-github-actions.mdx b/tutorial/7-ci/2-github-actions.mdx index d2e1961..5fd2015 100644 --- a/tutorial/7-ci/2-github-actions.mdx +++ b/tutorial/7-ci/2-github-actions.mdx @@ -129,13 +129,21 @@ The artifact name now includes both matrix values. Jobs uploading to the same ar :::note This is where the cross-platform bugs show up `Join-Path` and `$TestDrive` have been quietly protecting you. Hard-coded `\` separators, case-sensitivity assumptions, and `C:\temp` paths all work on your machine and fail on Linux. A matrix is how you find them, and it is the main reason this module is worth doing even for a small module. + +Culture belongs on that list, and Planetarium has the bug right now. `'{0,-8} {1} AU' -f ...` formats with the current culture, so on a machine that uses a comma as the decimal separator the report reads `Mercury 0,387 AU`. The suite stays green anyway, because the only line it asserts on is `Earth 1 AU`, which has no decimal part. An operating system matrix does not find this one, the runners are all set to the same culture. If your code formats numbers or dates, test it under a culture that is not yours. ::: ## Where to go from here The workflow above is a complete, working setup. Natural next steps, in rough order of value: -- **Publish the test results** as a check run so failures annotate the pull request directly, using something like `dorny/test-reporter`, which reads the NUnit file you are already producing. +- **Read the inline annotations you already get.** `Output.CIFormat` defaults to `Auto`, so on GitHub Actions Pester writes failures as workflow commands and they show up on the job without any extra step: + + ``` + ::error::[-] Returns all eight planets 26ms + ``` + +- **Publish the test results** as a check run if you want failures attached to the changed lines in the pull request, using something like `dorny/test-reporter`, which reads the NUnit file you are already producing. - **Send `coverage.xml` to a coverage service** — the JaCoCo format is widely supported. - **Require the check** in branch protection, so a red build actually blocks the merge. Until you do this, CI is advisory. - **Cache the Pester install** if the install step becomes a meaningful share of the run time.