Skip to content

tests: drop the xfail marker and its runtests support #4544

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
BsAtHome merged 1 commit into from
Sep 15, 2026
Merged

tests: drop the xfail marker and its runtests support #4544

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 0 additions & 3 deletions docs/src/code/writing-tests.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,9 +68,6 @@ checkresult::
The exit code of this program controls the result of the test.
If both `expected` and `checkresult` exist, only `checkresult` is consulted to validate the test output.

xfail::
If this file exist, a test failure is expected and does not cause runtests to return an exit code signaling an error.

skip::
If this file exist, the test is skipped and not executed at all.

Expand Down
29 changes: 8 additions & 21 deletions scripts/runtests.in
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,6 @@ export RUNTESTS

NUM=0
FAIL=0; FAIL_NAMES=""
XFAIL=0
SHMERR=0
SKIP=0
VERBOSE=0
Expand Down Expand Up @@ -286,35 +285,23 @@ run_tests () {
fi
fi
if [ "$exitcode" -ne 0 ]; then
echo "*** $testdir: XFAIL: $reason"
echo "*** $testdir: FAIL: $reason"
if test $PRINT = 1; then
echo "************** result:"
tail -500 "$testdir/result" | sed 's/^/ /'
echo "************** stderr:"
tail -500 "$testdir/stderr" | sed 's/^/ /'
echo "**************"
fi
if [ -f "$testdir/xfail" ]; then
XFAIL=$((XFAIL + 1))
if [ $NOCLEAN -eq 0 ]; then
rm -f "$testdir/stderr" "$testdir/result" \
"$testdir"/*.var "$testdir"/*.var.bak
fi
else
FAIL=$((FAIL + 1))
FAIL_NAMES="$FAIL_NAMES"$'\n'"$testdir"
fi
FAIL=$((FAIL + 1))
FAIL_NAMES="$FAIL_NAMES"$'\n'"$testdir"
if test $STOP = 1; then
break
fi
else
if [ -f "$testdir/xfail" ]; then
echo "*** $testdir: XPASS: Passed, but was expected to fail"
else
if [ $NOCLEAN -eq 0 ]; then
rm -f "$testdir/stderr" "$testdir/result" \
"$testdir"/*.var "$testdir"/*.var.bak
fi
if [ $NOCLEAN -eq 0 ]; then
rm -f "$testdir/stderr" "$testdir/result" \
"$testdir"/*.var "$testdir"/*.var.bak
fi
fi

Expand All @@ -327,8 +314,8 @@ run_tests () {
fi
done < "$TMPDIR/alltests"

SUCC=$((NUM-FAIL-XFAIL))
echo "Runtest: $NUM tests run, $SUCC successful, $FAIL failed + $XFAIL expected, $SKIP skipped, $SHMERR shmem errors"
SUCC=$((NUM-FAIL))
echo "Runtest: $NUM tests run, $SUCC successful, $FAIL failed, $SKIP skipped, $SHMERR shmem errors"
if [ "$FAIL" -ne 0 ]; then
echo "Failed: $FAIL_NAMES"
exit 1;
Expand Down
Empty file removed tests/interp/m98m99/12-M99-endless-main-program/xfail
View file
Open in desktop
Empty file.
4 changes: 2 additions & 2 deletions tests/overrun/README
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
This test makes sure that the "overrun" detection works. It is an XFAIL
because the testing infrastructure will believe it is a failure.
This test makes sure that the "overrun" detection works. The inner runtests
run is expected to fail, so test.sh negates its exit code.
2 changes: 1 addition & 1 deletion tests/overrun/checkresult
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,4 @@
set -e
grep -q "overrun detected in sampler" "$1"
grep -q "10 overruns detected, giving up" "$1"
grep -q "1 tests run, 0 successful, 1 failed . 0 expected" "$1"
grep -q "1 tests run, 0 successful, 1 failed," "$1"
Loading