Skip to content

Enforced a coverage floor on the merged report - #667

Merged
fdesbiens merged 1 commit into
eclipse-threadx:devfrom
fdesbiens:feature/coverage-floor
Aug 26, 2026
Merged

Enforced a coverage floor on the merged report#667
fdesbiens merged 1 commit into
eclipse-threadx:devfrom
fdesbiens:feature/coverage-floor

Conversation

@fdesbiens

Copy link
Copy Markdown
Contributor

The coverage summary has always reported a percentage and never been able to fail. irongut/CodeCoverageSummary takes thresholds and fail_below_min; neither was set, so coverage could fall from 99.97% to anything at all and every check stayed green — against an AGENTS.md that asks for 100% test coverage. A stated requirement measured with a gauge that has no failure mode.

Both are set now, through a new coverage_thresholds input on regression_template.yml, because the two suites do not sit at the same figure.

suite merged report floor
ThreadX 4503 / 4503 lines — 100.00% 99
SMP 5114 / 5178 lines — 98.76% 98

Figures from the dev push run of #666, run 32967526931, read off the uploaded artifacts.

What was probed before picking those numbers

The action pins its own repository to a SHA but runs docker://ghcr.io/irongut/codecoveragesummary:v1.3.0 — a mutable tag — so reading the source at that SHA is not proof of what executes. All of the below was measured on a runner against the two real merged.xml files above, plus one edited to 4502/4503 to stand in for the flap.

The floor compares the line rate and nothing else. This was the open question, and it mattered: branch coverage is ~78% in both suites while line coverage is 98.8–100%, so a floor aimed at the line figure would have been an immediate red wall if it tested branches, or the lower of the two. The ThreadX report at 100.00% lines and 77.67% branches clears a floor of 99.

The thresholds are whole numbers. '99.9 100' — the value this change was meant to carry — is rejected with System.ArgumentException - Threshold parameter set incorrectly., and the step fails whether or not fail_below_min is set. So the choice is 99 or 100, with nothing between them.

100 would fail on a race. tx_thread_system_resume.c:529 is reached by timing rather than by construction and flaps between runs of the same green tree — it is the one line #666 could not reach deliberately, and two local runs of the same tree gave 100.00% and 99.98%. 4502/4503 fails a floor of 100 and clears one of 99. A coverage gate that goes red on a coin toss is how coverage gates get switched off again.

A shared floor would not have worked. SMP at 98.76% fails a floor of 99 and clears 98. #666 closed the uncovered lines in common/src only; common_smp/src still carries the same gaps in tx_block_allocate.c, tx_byte_allocate.c, tx_thread_create.c and tx_trace_object_register.c, plus a family of its own in the *_delete.c teardown paths and tx_byte_pool_search.c — 64 lines across 11 files. Closing those and raising the SMP floor is follow-up work, not this PR.

One limit recorded rather than fixed

An empty report reads as 100%. gcovr writes line-rate="1.0" beside lines-valid="0" when it finds no data, and the action prints Line Rate = 100% (0 / 0) and passes any floor — probed. No threshold can catch that, so the floor does not defend the failure mode it most looks like it defends. The check for it is the emptiness assertion #664 added to each suite's coverage.sh. Said so in a comment next to the step, because the next reader will otherwise assume the floor covers it.

Also in here

Two stale filenames in deploy_code_coverage's comment: since #665 each coverage artifact carries merged.xml, not default_build_coverage.xml, and the per-suite directory is renamed from merged. Verified against the artifacts of the run above. Folded in rather than spending a PR on it.

Verification

The floor is exercised by this PR's own tx and smp runs — both jobs read a real merged report against a real threshold, so a wrong number here shows up as a red check rather than as a surprise on dev.

The coverage summary reported a percentage and could not fail. Coverage
could fall from 99.97% to anything at all and every check stayed green,
against an AGENTS.md that asks for 100% test coverage -- a stated
requirement measured with a gauge that had no failure mode.

CodeCoverageSummary already takes thresholds and fail_below_min; neither
was set. Both are now, through a new coverage_thresholds input on the
template, because the two suites do not sit at the same figure: ThreadX
99, SMP 98.

Three things were probed against the pinned action on a runner before
picking those numbers, using the real merged.xml files from the dev push
run of eclipse-threadx#666.

The floor compares the line rate and nothing else. That mattered because
branch coverage is around 78% in both suites while line coverage is
98.8-100%, so a floor aimed at the line figure would have been an
immediate red wall had it tested branches or the lower of the two. The
ThreadX report at 100.00% lines and 77.67% branches clears a floor of 99.

The thresholds are whole numbers. '99.9 100' -- the value this was meant
to be -- is rejected with 'System.ArgumentException - Threshold parameter
set incorrectly.', and the step fails whether or not fail_below_min is
set. So the choice is 99 or 100 with nothing between.

100 would fail on a race. tx_thread_system_resume.c:529 is reached by
timing rather than by construction and flaps between runs of the same
green tree, which is why eclipse-threadx#666 left it; 4502/4503 fails a floor of 100 and
clears one of 99. A coverage gate that goes red on a coin toss is how
coverage gates get switched off.

SMP is 5114/5178 lines, 98.76%, with 64 uncovered lines across 11 files
of common_smp/src -- eclipse-threadx#666 closed the equivalent gaps in common/src only.
A shared floor of 99 would have failed that job on every run while
ThreadX passed.

One limit is recorded in the file rather than fixed: an empty report
reads as 100%. gcovr writes line-rate="1.0" beside lines-valid="0" when
it finds no data, and the action prints 'Line Rate = 100% (0 / 0)' and
passes any floor. The check for that is the emptiness assertion eclipse-threadx#664 put
in each suite's coverage.sh, not this one.

Also corrected two stale filenames in the deploy job's comment: since
eclipse-threadx#665 each coverage artifact carries merged.xml, not
default_build_coverage.xml. Verified on the runner.

Assisted-by: Claude Opus 5 <noreply@anthropic.com>
@fdesbiens
fdesbiens merged commit 147754c into eclipse-threadx:dev Aug 26, 2026
8 checks passed
@fdesbiens
fdesbiens deleted the feature/coverage-floor branch August 26, 2026 13:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant