fix: fail fast on endpoint response stalls

[roborluo]

Summary

• add an opt-in no-progress deadline for in-flight endpoint requests
• expose it as settings.timeouts.no_progress_timeout_s plus a --no-progress-timeout CLI alias
• arm the guard only while work is in flight; retire it as soon as the phase drains

When this is useful

Use this for automated runs where a request can be accepted but the endpoint then becomes silent — for example, a TensorRT-LLM disaggregated executor or KV-transfer stall that never reaches the normal terminal-error path. It is engine-agnostic: it catches the same client-visible silent failure through vLLM, a frontend, or transport. Without it the benchmark blocks until its outer wall-time limit, because the phase drain waits on responses that never arrive and drain_timeout defaults to unlimited.

Disabled by default. It starts after work is issued, runs only while requests are in flight, and resets on an observed stream chunk or final result. It does not diagnose or restart the backend — it makes the benchmark fail with a clear error. For non-streaming endpoints, configure it above the full expected request latency. For TensorRT-LLM disaggregated serving the documented starting value is 300 s, matching the executor hang_detection_timeout.

Changes since the last review

Rebased onto latest main and moved the setting into the Timeouts model added by #409 (@viraatc).

Review point	Change
[P1] completed_uuids retained every completed UUID	Only register_skipped adds tombstones now, so it no longer grows O(total requests) alongside uuid_to_index
[P2] disabled path did work per response	Gated on the feature being enabled, and the receiver reuses the arrival timestamp it already takes instead of reading the clock twice
[P2] task + timer per 0→1 transition	Single self-rearming loop.call_later TimerHandle — no task, no Event, no per-iteration wait_for
[P2] full-timeout sleep could detect silence ~2x late	Remaining deadline is computed from the last progress stamp
cohort wording	Removed from the docs and from the local variable in issue()
help/description drift	Schema description is the short form; docs/config/DESIGN.md holds the tuning detail

Measured cost when disabled (A/B against the base commit): +3.5 ns per response, +9.4 ns per request. With stream_all_chunks: false (the default) the main process sees 2 messages per request, not one per token.

Measured cost of the arming change under the production eager task factory: arm+cancel per request went from 2205 ns to 535 ns (4.1x).

Also fixed while here

A receiver transport error could overwrite an earlier NoProgressError. A stalled endpoint often drops its connection too, so the real diagnosis was being masked by a generic "receiver failed" message. First error now wins, with a regression test.

Not adopted

The suggestion to use one phase/session-lifetime watchdog driven by an activity event. The guard is still armed on the 0→1 in-flight transition and retired on drain — now with a TimerHandle rather than a task. This keeps the timer heap clear once a phase drains, and measured 4.1x cheaper than the reviewed version. Happy to switch to a resident watchdog if that is preferred.

Validation

• pytest tests/unit/config/test_schema.py tests/unit/commands/test_benchmark.py tests/unit/load_generator/test_async_session.py — 425 passed
• pre-commit run --all-files — all hooks pass except mypy, which reports 3 pre-existing os.sched_*affinity errors on macOS in cpu_affinity.py / token_metrics.py; neither file is touched by this PR and the symbols exist on Linux
• python scripts/regenerate_templates.py --check
• AGA disaggregated held-response canary, job 578946: failed as intended with Endpoint made no response progress for 10.0s with 1 request(s) in flight, 10.002 s after the phase started against a 10 s deadline. The 10 s value is fault-injection coverage, not the deployment recommendation.

Timer lifecycle is covered in both directions: the perf issue cap (stop_current_phase) deliberately leaves the guard armed so a stall during drain is still caught, while drain completion, stop(), a phase change, and session teardown each retire it.

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[codecov-commenter]

Codecov Report

❌ Patch coverage is 98.76543% with 1 line in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (main@ccf6929). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
src/inference_endpoint/load_generator/session.py	98.75%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #462   +/-   ##
=======================================
  Coverage        ?   81.31%           
=======================================
  Files           ?      151           
  Lines           ?    20528           
  Branches        ?        0           
=======================================
  Hits            ?    16693           
  Misses          ?     3835           
  Partials        ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[arekay-nv]

Lets not use cohort here.

[arekay-nv]

Lets keep a single copy in the help/description so there isn't drift over time.

[arekay-nv]

The opt-in liveness guard is useful, but this implementation adds avoidable work and state to the load generator's hot paths. The most serious issue is retaining every completed UUID in a second phase-lifetime container, which is an O(total requests) memory regression at the repository's 50k+ QPS target. The disabled path also performs work for every response, and low-concurrency workloads can create/cancel a watchdog task per request. Please keep the disabled path inert and use bounded active-request state plus a single event/deadline-driven watchdog.

[roborluo]

Thanks for the comments @viraatc and @arekay-nv . All issues resovled other than one.

For the per transition watchdog, I choosed a different approche than your suggested ,so flagging it here.

the guard is still armed on the 0→1 in-flight transition and retired on drain — but it's now a single loop.call_later TimerHandle instead of an asyncio task. No task, no Event, no per-iteration `wait_for.

Also measured arm+cancel per request, under the eager task factory this repo uses in production:

Task (reviewed version): 2205 ns
Timer (now): 535 ns → 4.1x

It also leaves the timer heap clear once a phase drains, which covers the "stale timer work" part of your comment.

let me know if that make sense to you, Thanks a lot!