is_server_running marks a healthy job FAILED on libfabric `<warn>` lines (matches "unable to")

[dadamsncsa]

Version: vec-inf 0.9.0; the same logic is on main (vec_inf/client/_utils.py, is_server_running).

What happens: before Application startup complete. appears, any stderr line containing one of traceback, exception, fatal error, critical error, failed to, could not, unable to, error: returns (ModelStatus.FAILED, line). On an HPE Slingshot system (NCSA Delta) with libfabric logging enabled (FI_LOG_LEVEL=warn, common in Slingshot-enabled containers), the CXI provider writes warnings such as

libfabric:132:1787839803::cxi:ep_ctrl:cxip_ep_close():869<warn> gpua001.delta.ncsa.illinois.edu: Unable to free EP object -16 : Device or resource busy

to stderr during NCCL initialisation. The server continues and is ready about a minute later, but the status is already FAILED with that line as failed_reason, and a caller that acts on it (LLMHub's backend) cancels the healthy job. Observed on SLURM jobs 21500607 and 21500755 (vllm serve, tensor-parallel 4, single node); weights had loaded, KV cache was being allocated.

Suggestions (any one is enough):

• skip lines that carry an explicit warning marker (<warn>, WARNING, warn:) before applying the fatal patterns;
• or only report FAILED when the SLURM step has actually ended (or the ready signature is absent after a grace period), treating stderr matches as pending_reason until then;
• or make the fatal patterns configurable in environment.yaml.

Workaround in use: FI_LOG_PROV=none in the job environment, which silences the provider logs.