test_asyncio: socket test harness cannot fail a test from the client/server thread

[matthiasgoergens]

test_asyncio's socket-test harness aborts a test by calling self.fail() from the server or client thread:

# Lib/test/test_asyncio/functional.py
def _abort_socket_test(self, ex):
    try:
        self.loop.stop()
    finally:
        self.fail(ex)

self.fail() raises AssertionError in the calling thread. Both call sites are inside a Thread.run(), so the exception escapes the thread without ever reaching TestCase.run() — the test still reports ok. An error in the server half of these tests does not fail them.

I ran into this chasing an unrelated red CI job and thought it was worth reporting separately.

It has never worked

The function arrived in f111b3d (bpo-23749, 2017-12-30) and those four lines are byte-identical today. git log -L on the function returns no later commits, and both call sites have always been inside Thread.run(), so there has never been a path on which it reaches the main thread.

Rather than rely on that, I ran a minimal equivalent under every release the code has shipped in:

class T(unittest.TestCase):
    def test_abort_from_worker_thread(self):
        t = threading.Thread(target=lambda: self.fail("aborted"))
        t.start(); t.join()

Python	3.6.15	3.7.17	3.8.20	3.9.25	3.10.20	3.11.15	3.12.12	3.13.12	3.14.2
failures	0	0	0	0	0	0	0	0	0
wasSuccessful()	True	True	True	True	True	True	True	True	True

Whether anything reports it is a race

Automatic reporting of uncaught thread exceptions arrived in b136b1a (bpo-43843), first released in 3.10.0 and not backported. Before that the only mechanism was the opt-in threading_helper.catch_threading_exception, which this harness has never used. So on 3.7–3.9 an aborted socket test was completely silent.

Since 3.10 there is a Warning -- Uncaught thread exception and ENV_CHANGED, but it does not always win the race. Injecting a failure into test_shutdown_corrupted_ssl_sends_close_notify so that it aborts on every run:

invocation	reported SUCCESS	reported FAILURE
-m test test_asyncio.test_sslproto -m <test>	6 / 12	6 / 12
the same, with -j1	3 / 8	5 / 8

The warning text is printed every time; whether it reaches support.environment_altered before regrtest reads it is timing. 9 of those 20 runs exited 0.

It has already hidden a real failure

On a Windows CI job the server's sock.unwrap() raised

ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host

test_shutdown_corrupted_ssl_sends_close_notify exists to check that a peer sees "a clean TLS EOF instead of a connection reset" (gh-98078). Its server catches only ssl.SSLError:

    sock.unwrap()
except ssl.SSLError as exc:
    server_err = exc
...
self.assertIsNone(server_err)

so a ConnectionResetError — the condition the test exists to detect — leaves server_err as None, the assertion passes, and the error disappears into the thread. The test reported ok.

A second problem in the same three lines

self.loop.stop() is called from the worker thread; event loops are not thread-safe, so this should be loop.call_soon_threadsafe(loop.stop). If the stop does not take effect the main thread keeps waiting, which resembles some long-standing reports of test_start_tls_server_1 timing out on ARMv7 and Fedora — though I have not verified that connection and am not claiming it.

Proposed fix

Record the exception in the worker thread and re-raise it from tearDown(), and stop the loop thread-safely. PR to follow.

Please treat the PR as potentially disruptive

I want to flag the risk clearly rather than bury it, because the change is small but its effect is not local.

These failures currently do not fail tests. Making the abort work means any latent failure in the client or server half of a socket test will start failing — including the Windows ConnectionResetError above, which I expect to go red. The mixin is used by test_sslproto, test_ssl, test_events, test_server, test_streams, test_buffered_proto, test_sock_lowlevel and test_unix_events, so the blast radius is most of test_asyncio's network tests across every platform and buildbot.

I have no way to predict from here how many buildbots this lights up; platform-specific socket behaviour is exactly what a Linux dev box cannot tell you, and eight years of accumulated silence is a lot of surface. It is possible the honest sequence is to land the diagnosis first, survey what actually turns red, and fix those before enabling the abort — or to land it early in a release cycle rather than near a beta. I am happy to split it that way, to gate it behind a flag, or to drop it entirely if the churn is not judged worth it.

Environment

Linux, in-tree build of main (3.16.0a0). The version table was produced with uv run --python <v> for 3.8–3.14 and the python:3.6/python:3.7 images for the two EOL releases; the detection endpoints were checked by building v3.10.0a1 and v3.10.0 from source.

Linked PRs

• gh-155027: Make test_asyncio's socket harness able to fail a test #155028