Add recycle_on_failure and retries options to runtests

Project.toml

@@ -1,7 +1,7 @@
 name = "ParallelTestRunner"
 uuid = "d3525ed8-44d0-4b2c-a655-542cee43accc"
 authors = ["Valentin Churavy <v.churavy@gmail.com>"]
-version = "2.7.0"
+version = "2.8.0"
 
 [deps]
 Dates = "ade2ca70-3891-5945-98fb-dc099432e06a"

docs/src/advanced.md

@@ -174,6 +174,55 @@ duration, longest first) and their results appear in the same overall summary.
     If the user filters tests via positional arguments (e.g. `julia test/runtests.jl unit`),
     any serial test names that were filtered out are silently removed from the serial list.
 
+## Failure Handling
+
+Both options described in this section are opt-in and default to off.
+
+### Recycling Workers after a Failure
+
+Workers are reused across tests, so a test that corrupts process-wide state — a wedged GPU driver whose every subsequent allocation fails, a global left in an inconsistent state, a library put in an unusable configuration — can make every later test scheduled on that same worker fail too.
+
+Setting `recycle_on_failure=true` stops the worker after any test that did not pass, so the next test gets a fresh process:
+
+```julia
+runtests(MyPackage, ARGS; recycle_on_failure=true)
+```
+
+This complements the existing recycling of workers exceeding `max_worker_rss` and of workers that crashed outright.
+The cost is worker start-up time (plus re-running `init_worker_code`) after each failure, which is why it is off by default: for a suite whose failures are self-contained it is pure overhead.
+
+### Retrying Failed Tests
+
+When several workers compete for a limited resource — GPU memory, RAM, a device that only allows so many contexts — a failure can mean "lost the race for the resource" rather than "the code is broken".
+Such a test typically passes when run on its own.
+
+The `retries` keyword argument re-runs tests that did not pass, up to `N` times, after the main run has completed:
+
+```julia
+runtests(MyPackage, ARGS; retries=1)
+```
+
+The retry environment is deliberately quiesced: all parallel workers have been stopped by then, and the retried tests run **sequentially on a single fresh worker**, so a test that failed only because of concurrent resource pressure gets an otherwise-idle system.
+If a test fails again, its worker is stopped before the next retry, so one failure cannot contaminate the following one.
+
+Only the final attempt of each test is recorded in the results, so a test that passes on retry is reported as passing and a persistently broken test is reported as failing exactly once.
+Retries are visible in the output, so flakiness is surfaced rather than hidden:
+
+```
+Retrying 2 failed test(s) on a fresh worker...
+  gpu/memory passed on retry
+  broken_test failed again
+```
+
+!!! note
+    Retries are skipped when the run was interrupted (e.g. `Ctrl+C`) or when `--quickfail` is
+    in effect, since in both cases the run stopped early on purpose.
+
+!!! tip
+    `recycle_on_failure` and `retries` address different halves of the same problem and work
+    well together: recycling keeps one bad test from cascading onto its worker during the run,
+    while retries give the tests that did fail a contention-free second chance.
+
 ## Custom Workers
 
 For tests that require specific environment variables or Julia flags, you can use the `test_worker` keyword argument to [`runtests`](@ref) to assign tests to custom workers:
@@ -303,3 +352,5 @@ function jltest {
 1. **Use custom workers sparingly**: Custom workers add overhead. Only use them when tests genuinely require different configurations.
 
 1. **Use `serial` for resource-intensive tests**: If a test allocates significant memory or uses exclusive hardware resources, mark it as serial rather than reducing `--jobs` globally. This keeps the rest of your suite running in parallel.
+
+1. **Don't paper over real failures with `retries`**: Retries are meant for failures caused by contention between concurrent workers, not for tests that are genuinely broken. Persistent failures still fail after their retries, and retried tests are reported as such, so keep an eye on which tests keep needing a second attempt.

docs/src/index.md

@@ -114,6 +114,15 @@ The `serial` keyword argument to [`runtests`](@ref) lets you designate specific
 for sequential execution, either before or after the parallel batch.
 See [Serial Tests](@ref) in the advanced usage guide for details.
 
+### Failure Recycling and Retries
+
+Workers are recycled when they crash or exceed the memory threshold.
+Additionally, `recycle_on_failure=true` recycles a worker after any failed test, so a test
+that corrupts process-wide state cannot poison later tests, and `retries=N` re-runs failed
+tests on an otherwise-idle system, to tell tests broken by resource contention apart from
+genuinely broken ones.
+See [Failure Handling](@ref) in the advanced usage guide for details.
+
 ### Real-time Progress
 
 The test runner provides real-time output showing:

src/ParallelTestRunner.jl

@@ -833,7 +833,9 @@ end
              stderr = Base.stderr,
              max_worker_rss = get_max_worker_rss(),
              serial = String[],
-             serial_position::Symbol = :before)
+             serial_position::Symbol = :before,
+             recycle_on_failure::Bool = false,
+             retries::Integer = 0)
     runtests(mod::Module, ARGS; ...)
 
 Run Julia tests in parallel across multiple worker processes.
@@ -880,6 +882,10 @@ Several keyword arguments are also supported:
   instead of in parallel. An `ArgumentError` is thrown if any name is not found in the testsuite.
 - `serial_position`: When to run serial tests relative to the parallel batch.
   Must be `:before` (default) or `:after`.
+- `recycle_on_failure`: Whether to recycle a worker after any test that did not pass
+  (default: `false`). See the Failure Handling section below.
+- `retries`: How many times to re-run tests that did not pass after the main run completes
+  (default: `0`). See the Failure Handling section below.
 
 ## Command Line Options
 
@@ -959,6 +965,17 @@ runtests(MyPackage, ARGS; serial=["big_alloc_test", "huge_matrix"])
 
 Workers are automatically recycled when they exceed memory limits to prevent out-of-memory
 issues during long test runs. The memory limit is set based on system architecture.
+
+## Failure Handling
+
+With `recycle_on_failure = true`, a worker is recycled after any test that did not pass, so
+a test that corrupts process-wide state (e.g. wedges a GPU driver) cannot poison subsequent
+tests on the same worker.
+
+With `retries = N` (default 0), tests that did not pass are re-run up to `N` times after
+the main run completes — sequentially, on a single fresh worker, with all other workers
+stopped — so tests that failed due to resource pressure from concurrent workers get an
+otherwise-idle system. Only the final attempt of each test is reported.
 """
 function runtests(mod::Module, args::ParsedArgs;
                   testsuite::Dict{String,Expr} = find_tests(pwd()),
@@ -973,6 +990,8 @@ function runtests(mod::Module, args::ParsedArgs;
                   stdout = Base.stdout,
                   stderr = Base.stderr,
                   max_worker_rss = get_max_worker_rss(),
+                  recycle_on_failure::Bool = false,
+                  retries::Integer = 0,
                   )
     #
     # set-up
@@ -1023,6 +1042,8 @@ function runtests(mod::Module, args::ParsedArgs;
         stdout,
         stderr,
         max_worker_rss,
+        recycle_on_failure,
+        retries,
     )
 end
 
@@ -1045,6 +1066,8 @@ function _runtests(mod::Module, args::ParsedArgs;
                    stdout = Base.stdout,
                    stderr = Base.stderr,
                    max_worker_rss = get_max_worker_rss(),
+                   recycle_on_failure::Bool = false,
+                   retries::Integer = 0,
                    )
 
     # partition into serial and parallel groups
@@ -1274,6 +1297,7 @@ function _runtests(mod::Module, args::ParsedArgs;
     #
 
     tests_to_start = Threads.Atomic{Int}(length(tests))
+    interrupted = false
     # After parallel-before-serial: stop extra workers so only one process is alive for
     # serial tests, but keep one parallel worker so we do not add a third addworker (ID_COUNTER).
     function drain_pool_leaving_one_worker!(pool, njobs)
@@ -1374,6 +1398,11 @@ function _runtests(mod::Module, args::ParsedArgs;
                                       # the worker has reached the max-rss limit, recycle it
                                       # so future tests start with a smaller working set
                                       Malt.stop(wrkr)
+                                  elseif recycle_on_failure && anynonpass(result[])
+                                      # a failing test may have left the worker in a bad state
+                                      # (e.g. a wedged GPU driver whose every later allocation
+                                      # fails); recycle it so future tests get a fresh process
+                                      Malt.stop(wrkr)
                                   end
                               else
                                   # One of Malt.TerminatedWorkerException, Malt.RemoteException, or ErrorException
@@ -1430,6 +1459,7 @@ function _runtests(mod::Module, args::ParsedArgs;
             end
         end
     catch err
+        interrupted = true
         if !(err isa InterruptException)
             println(io_ctx.stderr, "\nCaught an error, stopping...")
         end
@@ -1467,6 +1497,73 @@ function _runtests(mod::Module, args::ParsedArgs;
         end
     end
 
+    # retry failed tests, if requested: sequentially, on a single fresh worker, with every
+    # other worker gone — tests that failed due to resource pressure (e.g. GPU memory
+    # oversubscription from concurrent workers) reliably pass on an otherwise-idle system.
+    # only the retried result is reported; persistent failures fail again and are reported
+    # exactly once.
+    if retries > 0 && !interrupted && args.quickfail === nothing
+        local retry_wrkr = nothing
+        for round in 1:retries
+            retryable = [r.test for r in results.value
+                         if r.result isa Exception || anynonpass(r.result[])]
+            isempty(retryable) && break
+            println(io_ctx.stdout)
+            printstyled(io_ctx.stdout,
+                        "Retrying $(length(retryable)) failed test(s) on a fresh worker...\n";
+                        color = :yellow)
+            for test in retryable
+                # pass in init_worker_code to custom worker function if defined
+                wrkr = if init_worker_code == :()
+                    test_worker(test)
+                else
+                    test_worker(test, init_worker_code)
+                end
+                if wrkr !== nothing && !Malt.isrunning(wrkr)
+                    wrkr = nothing
+                end
+                custom = wrkr !== nothing
+                if !custom
+                    if retry_wrkr === nothing || !Malt.isrunning(retry_wrkr)
+                        retry_wrkr = addworker(; init_worker_code, io_ctx.color, exename,
+                                               exeflags, env)
+                    end
+                    wrkr = retry_wrkr
+                end
+                test_t0 = time()
+                result = try
+                    Malt.remote_call_fetch(invokelatest, wrkr.w, runtest,
+                                           RecordType, testsuite[test], test,
+                                           init_code, test_t0, custom_args)
+                catch ex
+                    isa(ex, InterruptException) && rethrow()
+                    ex
+                end
+                test_t1 = time()
+                output = @lock wrkr.io String(take!(wrkr.io[]))
+                filter!(r -> r.test != test, results.value)
+                push!(results.value, (; test, result, output, test_t0, test_t1))
+                if result isa AbstractTestRecord && !anynonpass(result[])
+                    printstyled(io_ctx.stdout, "  $test passed on retry\n"; color = :green)
+                else
+                    printstyled(io_ctx.stdout, "  $test failed again\n"; color = :red)
+                    if !custom
+                        # don't let a failure contaminate the next retry
+                        Malt.stop(retry_wrkr)
+                        retry_wrkr = nothing
+                    end
+                end
+                # get rid of the custom worker
+                if custom && Malt.isrunning(wrkr)
+                    Malt.stop(wrkr)
+                end
+            end
+        end
+        if retry_wrkr !== nothing && Malt.isrunning(retry_wrkr)
+            Malt.stop(retry_wrkr)
+        end
+    end
+
     # print the output generated by each testset
     # (`@sync` above joined all writers, so `results` is quiescent from here on)
     for (testname, result, output, _start, _stop) in results.value