Conversation
run_tests_debug adds `3>&1`, duplicating the job's stdout onto fd 3. proc_open rewires only fds 0/1/2, so every server the harness spawns inherits fd 3. nginx runs with `daemon off; worker_processes 1`, and `stop(0)` SIGKILLs the master ~1ms after SIGTERM, before it can reap its worker -- the orphan keeps the job's log pipe from reaching EOF, `tee` blocks forever, and the job burns an hour until the runner kills it. Measured: nginx master, worker and php-cgi all hold the runner's fd-1 pipe inode; closing fd 3 once in phpunit leaves none holding it. Fleet rate was 58 hangs / 14,318 attempts across 10 pipelines, spread over 13 frameworks. Also add the readiness gate PhpCgi was the only isFastCgi() SAPI missing. php-cgi binds its port only after MINIT, and nginx turns an unbound upstream into a *successful* transfer carrying a 502, which the request retry loop cannot tell from a real response -- so the 502s surfaced as assertion failures with every log line reading `attempt #0`. Verified: the fd-3 repro goes from exit=124 (hung 25s) to exit=0 (0.9s) with the Makefile change alone; a 3s bind delay yields 10/10 502s before the gate and a 200 on attempt #0 after. test_web_custom over cgi-fcgi is unchanged.
❌ ErrorsYour PR has failed checks. Please review the issues below and take necessary action before merging. 🚦 7 Pipeline jobs failed
ℹ️ InfoNo other issues found (see more)🧪 All tests passed 🎯 Code Coverage (details) Useful? React with 👍 / 👎 This comment will be updated automatically if new data arrives.🔗 Commit SHA: ed222b6 | Docs | View more details | Give us feedback! |
…op flaking The runner process is itself traced but had no agent configured, so it used the compiled-in default localhost:8126 where nothing listens. Remote config polled it every 5s and each failure was logged at error level, alongside agent_info, send_with_retry and trace_flusher. The runner and the web server share one sidecar, whose log sink is a fan-out writer that appends every line to every log file any session registered -- so once a post-fork or new-ZTS-thread connection registered the app's dd_php_error.log, the runner's errors landed there and WebServer::checkErrors() failed the class. That is why swoole (forks workers) and frankenphp (ZTS) were hit at ~50% while test_web_* jobs emitting far more of these errors stayed green: they connect once at MINIT, before the error log exists. Placed on .cli_integration_test, not .base_test: the latter reaches 1256 jobs, 89 of which have no test-agent service (unresolvable host), and also reaches the .phpt jobs, where request_replayer.inc derives the replayer's own endpoint from DD_AGENT_HOST/DD_TRACE_AGENT_PORT and would break two of them. This hits the 104 PHPUnit targets only, all of which already run the service. Mirrors what Makefile:1339-1341 already does for test_integration. Verified locally on swoole: 14/15 red before, 15/15 green after, with zero [error|warn] lines in dd_php_error.log on every iteration and the localhost:8126 sessions gone. The runner emits no trace payloads (510 agent requests, all /v0.7/config and telemetry), so snapshot sessions cannot be contaminated, and check_test_agent.sh returns 200 before and after.
Leiyks
changed the title
.cli_integration_test sets KUBERNETES_CPU_REQUEST: 2 with no KUBERNETES_CPU_LIMIT, so automaxprocs finds no cgroup quota, logs "Leaving GOMAXPROCS=87: CPU quota undefined" and FrankenPHP starts 2*GOMAXPROCS+1 threads -- measured at 175 on an 87-core node and 153 on a 76-core node, inside a 2-CPU/4GiB pod. The thread count therefore depended on which node the job landed on, which is why test_integrations_frankenphp hit the 1h job timeout in ~40% of attempts with PHPUnit still running and no summary printed. Pinning num_threads with max_threads unset also disables autoscaling, so the count is now constant and independent of GOMAXPROCS. num_threads must stay above the worker num or FrankenPHP refuses to start outright -- both are set together for that reason. max_wait_time was 0, meaning a request with no free thread waited on a nil channel forever, which is what turned a stuck server into a job timeout rather than a failed test. It bounds only the queue wait, never execution (measured: a 30s request still returns 200), so a slow ASAN request cannot be rejected. FrankenphpServer also had no readiness wait -- one of only two SAPIs missing one -- and Caddy logs "server running" both before and after FrankenPHP's own startup line, so the probe gates on that line and the port together. Verified against the pinned v1.12.7 binary with the real rendered Caddyfile: starts with num_threads=4/max_threads=4, /simple 200, /error 500, 5 sequential 200s. Capacity measured at 2 against a suite demand of 1 (all requests sequential, one PHPUnit process). The PHPUnit suite itself was not run locally: CI uses debug-zts-asan and the local container is NTS, so it self-skips.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why FastCGI web jobs hang for an hour after the tests pass
run_tests_debugadds3>&1, which duplicates the job's stdout pipe onto fd 3.proc_openonly rewires fds 0/1/2, so every server the harness spawns inherits fd 3 — a live write end of the runner's log pipe. nginx runsdaemon off; worker_processes 1, andstop(0)SIGKILLs the master ~1ms after SIGTERM, before it can reap its worker. The orphaned worker survives the job, keeps the log pipe from reaching EOF,teenever returns, and the job burns its full hour until the runner kills it.Measured with
lsofon the pipe inode: nginx master, nginx worker and php-cgi all hold the runner's fd-1 pipe. Closing fd 3 once in phpunit leaves none of them holding it — so the fix is one line in the Makefile, and the nginx/php-cgistop()grace bumps here are purely defence-in-depth (the orphan is still alive, it just no longer holds the job's pipe).Fleet rate: 58 hangs / 14,318 attempts across 10 pipelines, spread over 13 frameworks.
PhpCgi readiness gate
PhpCgiwas the onlyisFastCgi()SAPI without a readiness gate. php-cgi binds its port only after MINIT, and nginx turns an unbound upstream into a successful transfer carrying a 502 — indistinguishable, to the existing request retry loop, from a real response. So startup races surfaced as assertion failures with every log line readingattempt #0, rather than as retries.Why swoole/frankenphp PHPUnit jobs flake at ~50%
Second, unrelated fix in the same area. The PHPUnit runner process is itself traced, but no agent was configured for it, so it used the compiled-in default
localhost:8126where nothing listens. Remote config polled it every 5s and logged each failure at error level, alongsideagent_info,send_with_retryandtrace_flusher. The runner and the web server share one sidecar, and the sidecar's log sink is a fan-out writer that appends every line to every log file any session has registered — so once a post-fork or new-ZTS-thread connection registered the app'sdd_php_error.log, the runner's own errors landed in it andWebServer::checkErrors()failed the class. That is why swoole (forks workers) and frankenphp (ZTS) were hit at ~50% whiletest_web_*jobs emitting far more of these errors stayed green: they connect once at MINIT, before the error log exists.Fix: set
DD_AGENT_HOST/DD_TRACE_AGENT_PORTon.cli_integration_test, mirroring whatMakefile:1339-1341already does fortest_integration. Not on.base_test: that reaches 1256 jobs, 89 of which have notest-agentservice (the host would not resolve), and also reaches the.phptjobs, whererequest_replayer.incderives the replayer's own endpoint from those two variables and two tests would break..cli_integration_testcovers the 104 PHPUnit targets only, all of which already run the service.Verified locally on swoole: 14/15 iterations red before, 15/15 green after, with zero
[error|warn]lines indd_php_error.logon every iteration and thelocalhost:8126sessions gone. The runner emits no trace payloads (510 agent requests, all/v0.7/configand telemetry), so snapshot sessions cannot be contaminated, andcheck_test_agent.shreturns 200 before and after. frankenphp was not verified locally — CI runs it ondebug-zts-asanand the local container is NTS, so the suite self-skips; confirmation is expected from CI.Why the frankenphp job hangs for its full hour
Third fix, again independent of the two above.
.cli_integration_testsetsKUBERNETES_CPU_REQUEST: 2with noKUBERNETES_CPU_LIMIT, so automaxprocs finds no cgroup quota, logsLeaving GOMAXPROCS=87: CPU quota undefined, and FrankenPHP starts2*GOMAXPROCS+1threads — measured at 175 on an 87-core node and 153 on a 76-core node, inside a 2-CPU/4GiB pod. The thread count therefore depended on which node the job landed on, which is whytest_integrations_frankenphphit the 1h job timeout in ~40% of attempts with PHPUnit still running and no summary printed. Pinningnum_threadswithmax_threadsunset also disables autoscaling, so the count is now constant and independent ofGOMAXPROCS;num_threadsmust stay above the workernumor FrankenPHP refuses to start outright, which is why both are set together.max_wait_timewas0, meaning a request with no free thread waited on a nil channel forever — that is what turned a stuck server into a job timeout rather than a failed test. It bounds only the queue wait, never execution (measured: a 30s request still returns 200), so a slow ASAN request cannot be rejected.FrankenphpServeralso had no readiness wait, one of only two SAPIs missing one. Caddy logsserver runningboth before and after FrankenPHP's own startup line, so the probe gates on that line and the port together.Verified against the pinned v1.12.7 binary with the real rendered Caddyfile: starts with
num_threads=4/max_threads=4,/simple200,/error500, 5 sequential 200s. Capacity measured at 2 against a suite demand of 1 (all requests are sequential, one PHPUnit process). The PHPUnit frankenphp suite itself was not run locally — CI runs it ondebug-zts-asanand the local container is NTS, so the suite self-skips.Caveat: pinning to 2 worker threads deliberately gives up the incidental multi-thread stress coverage that 153 threads provided.
Verification (fd-3 leak + readiness gate)
exit=124(hung 25s) →exit=0(0.9s) with the Makefile change alone.attempt #0after.test_web_customovercgi-fcgiunchanged.Not fixed here
Pre-existing issues, distinct from the three fixes above (each of the first two observed across at least 4 SHAs with no transition point — no commit caused them):
GracefulShutdownTestpassing on a SIGSEGV (exit 139).System Tests/PHP Language Tests, which do not run throughrun_tests_debugand so are unexplained by this fix.Follow-ups, not fixed here:
sendRequest()still has noCURLOPT_TIMEOUT, so a wedged server stalls a test rather than failing it.install-frankenphp.shruns an unbounded coldgo buildinside the first test that needs it.