Skip to content

fix: resolve result_future on inference error instead of tearing down the model - #157

Open
AmirF194 wants to merge 1 commit into
SearchSavior:mainfrom
AmirF194:fix/153-worker-error-hang-and-unload-crash
Open

fix: resolve result_future on inference error instead of tearing down the model#157
AmirF194 wants to merge 1 commit into
SearchSavior:mainfrom
AmirF194:fix/153-worker-error-hang-and-unload-crash

Conversation

@AmirF194

Copy link
Copy Markdown

Root cause

QueueWorker.queue_worker_llm and the identical loop repeated in every other
worker (vlm, whisper, qwen3_asr, kokoro, qwen3_tts, emb, rr)
broke out of its while True loop on an inference error before ever calling
packet.result_future.set_result(...). Non-streaming callers such as
WorkerRegistry.generate() await that future, so a failed generation hung
the caller forever. The same branch also fired an unconditional
registry.register_unload(model_name); that unload's pipeline destructor can
touch a corrupted device context after certain errors and abort the whole
process, taking every other loaded model down with it. This matches the issue
exactly for the LLM worker; I confirmed by reading each of the other seven
worker functions that they share the identical structural bug (grep/ast
enumeration in src/server/worker_registry.py), not only the ones named in
the issue as "likely" affected.

emb and rr trigger on a different condition (not completed_packet.response,
since InferWorker.infer_emb/infer_rerank already tag a raised exception as
a truthy "Error: ..." string, so that branch is actually reached only when
the pipeline returns no data with no exception), but the same break-before-resolve
bug is there too.

Fix

Each worker now reports the failure through result_future.set_exception(...)
and continues its loop instead of breaking, so a failed request surfaces as
an error to its caller and the worker/model stay alive to serve the next
request, per the issue's suggested design.

Verification

  • New tests/unit/test_worker_registry_error_handling.py drives the real
    (unmocked) queue_worker_* loop for all 8 workers with a failing request
    followed by a healthy one. Confirmed fails on unmodified main (each
    test times out waiting on result_future, matching the reported hang) and
    passes on this branch, in a clean python:3.12-slim container.
  • Full uv run pytest -W ignore tests/unit: 119 passed (was 111), no
    regressions.
  • coverage run --source=src.server.worker_registry: every line this diff
    changes is exercised by the new tests.
  • Not checked: the reported SIGABRT itself, since that needs a real Intel
    GPU/OpenCL context corrupted by a genuine device error, which this session
    doesn't have. This PR removes the register_unload call on the error path
    entirely (the mechanism the issue names as the crash trigger), but I can't
    reproduce the abort to confirm the crash is gone, only that the code no
    longer calls the destructor on that path.

Fixes #153

… the model

QueueWorker.queue_worker_llm (and the identical loop in every other worker:
vlm, whisper, qwen3_asr, kokoro, qwen3_tts, emb, rr) broke out of its
processing loop on an inference error before ever calling
packet.result_future.set_result(...). Non-streaming callers such as
WorkerRegistry.generate() await that future, so a failed generation hung
the HTTP caller forever. The same branch also fired an unconditional
registry.register_unload(model_name), whose destructor can touch a
corrupted device context after certain errors and SIGABRT the whole
process, taking every other loaded model down with it.

Each worker now resolves result_future with the error via
set_exception(RuntimeError(...)) and continues its loop instead of
breaking, so a failed request surfaces as an error to its caller and the
model/worker stay alive for the next request. The EMB and RR workers use a
different failure condition (an empty response, since InferWorker.infer_emb
tags a raised exception as a truthy "Error: ..." string) but had the exact
same break-before-resolve bug.

Fixes SearchSavior#153
@SearchSavior

Copy link
Copy Markdown
Owner

@AmirF194 Thanks for the PR!

RE:

Not checked: the reported SIGABRT itself, since that needs a real Intel
GPU/OpenCL context corrupted by a genuine device error, which this session
doesn't have.

Does this mean the proposed fix has not been tested on hardware?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

LLM inference failure hangs the caller, and error-triggered unload can SIGABRT the whole server

2 participants