feat: add priority-aware request scheduling

[xiaoba17]

Summary

• Add request priority 0–10 to the OpenAI-compatible Chat Completions API.
• Admit higher effective-priority requests first in both Static and Paged schedulers while keeping Decode non-preemptive.
• Add configurable aging, admission observability, scheduling statistics, focused regression tests, and user-facing documentation.
• Preserve FIFO behavior by default when requests have equal effective priority.

Motivation

InfiniLM currently admits waiting requests in FIFO order. Mixed online workloads need a bounded way to reduce queueing latency for latency-sensitive requests without disrupting in-flight Decode batches.

Priority affects Prefill admission only. Aging raises the effective priority of waiting requests to prevent starvation, and requests deferred by KV-cache pressure remain queued.

Closes #570.

Type of Change

• feat — new feature / new model
• fix — bug fix
• perf — performance improvement (no behavioral change)
• refactor — code restructuring without behavior change
• test — adding or fixing tests only
• docs — documentation only
• build / ci — build system or CI configuration
• chore — tooling, formatting, or other non-code changes
• Breaking change

Test Results of Involved Models on Supported Platforms (Please attach screenshots)

Platform	Model / path	Result
Local CPU	Focused priority-scheduling regression	14/14 passed
NVIDIA RTX 4090 D	DeepSeek-R1-Distill-Qwen-1.5B, Static service	Passed with 0 failed requests and no error logs
NVIDIA RTX 4090 D	DeepSeek-R1-Distill-Qwen-1.5B, Paged service	Passed with 0 failed requests and no error logs
MetaX C500 25% sGPU	DeepSeek-R1-Distill-Qwen-1.5B, Static service and paired benchmark	16/16 scenarios significantly reduced high-priority TTFT p95; 0 failed requests
MetaX C500 25% sGPU	DeepSeek-R1-Distill-Qwen-1.5B, Paged service and paired benchmark	16/16 scenarios significantly reduced high-priority TTFT p95; 0 failed requests
Other accelerator backends	Shared Python scheduling path	Not run because hardware was unavailable; maintainer validation is requested

The MetaX validation used PR commit 6683db7 and completed 14 reports, 320 paired FIFO/Priority runs, and 20,480 requests with no failures. The benchmark was executed headlessly; the measurements and server logs were archived outside this PR, so no UI screenshot is attached.

Benchmark / Performance Impact

This feature redistributes admission waiting time; it is not intended to increase aggregate throughput.

NVIDIA smoke benchmark

Configuration: RTX 4090 D, FP16, DeepSeek-R1-Distill-Qwen-1.5B, concurrency 16, max_batch_size=1, input/output lengths 128/32, 25% high-priority traffic, and two repetitions with 64 requests per cache mode.

Scheduler	High-priority TTFT p50	High-priority TTFT p95	Throughput
Static	2.5320 s → 0.5938 s (-76.55%)	3.8388 s → 1.1567 s (-69.87%)	-5.98%
Paged	0.1095 s → 0.0501 s (-54.27%)	0.1515 s → 0.0694 s (-54.22%)	+6.71%

MetaX C500 paired benchmark

Configuration: C500 25% sGPU with 16 GiB memory, MACA 3.1.0.14, DeepSeek-R1-Distill-Qwen-1.5B, max_batch_size=1, Prefix Cache disabled, and EOS ignored. FIFO uses priorities 0/0; Priority uses 10/0. Each scenario contains 64 requests and five paired repetitions with alternating FIFO/Priority execution order. Confidence intervals use per-run paired percentage differences and a Student-t 95% CI (df=4).

The shared baseline uses concurrency 32, 25% high-priority traffic, input/output lengths 128/32, and a 5-second aging interval. Values below are means from the baseline repeated across the concurrency, priority-ratio, input-length, output-length, and aging scans.

Scheduler	FIFO high-priority TTFT p95	Priority high-priority TTFT p95	Reduction	Normal-priority TTFT p95	Throughput
Static	14.144 s	3.809 s	73.1%	+18.7%	+0.3%
Paged	0.359 s	0.169 s	53.0%	+10.3%	+0.0%

Across the complete MetaX matrix:

• All 32 Static/Paged scenarios significantly reduced high-priority TTFT p95.
• All 16 Static scenarios significantly reduced high-priority E2E p95; Paged did so in 8/16 because priority changes Prefill admission while Decode remains non-preemptive.
• Neither the 16 Static nor the 16 Paged scenarios showed a statistically significant throughput change.
• Static normal-priority TTFT p95 increased significantly in 14/16 scenarios; Paged did so in 2/16.
• A 1-second aging interval reduced the Static TTFT benefit to 51.1%, compared with 73.2% at 5 seconds and 71.8% at 10 seconds. Aging should be configured relative to the platform's typical queueing time.

The broader paired NVIDIA and MetaX suites contain 40,960 successful requests. On the shared baseline, Static high-priority TTFT p95 falls by 73.1% on MetaX versus 74.6% on RTX 4090 D; Paged falls by 53.0% versus 52.6%.

Absolute device performance is not compared because the MetaX run used a 25% C500 slice and a different software stack. The Paged 512-token input case lies on the KV-capacity boundary and is excluded as a standalone conclusion. MetaX used InfiniCore build-compatibility patches in the remote test copy; those patches are not part of this InfiniLM PR. The MetaX client reproduced the NVIDIA workload and statistical method but was not the same source file. Streaming responses lacked usage metadata, so absolute output-token throughput uses non-empty content chunks; relative paired comparisons remain consistent.

Notes for Reviewers

Please focus on:

• Stable FIFO ordering when effective priorities are equal.
• Aging behavior and the distinction between original and effective priority.
• Re-queueing requests deferred by KV-cache pressure.
• Admission statistics and remote-KV admission accounting.
• The intentional non-preemptive Decode behavior.

Queue selection is O(n log n) over the current waiting snapshot. Priority redistributes waiting time rather than removing it, so normal-priority latency can increase under mixed-priority traffic. The implementation is independent of the protocol refactor in #554.

CI / ChatOps

Local Ruff formatting/checks and the focused 14-test scheduling regression passed. NVIDIA and MetaX service-level validation is summarized above.

The upstream CI run and Ruff run currently report action_required with no jobs executed. Maintainer approval or /retest is still required.

---

Checklist

Title, Branch, and Commits

• PR title follows Conventional Commits (e.g. feat(nvidia): …, fix(cuda/gemm): …).
• Branch name follows <type>/xxx-yyyy-zzzz where <type> matches the PR title's Conventional Commits type and words are joined with hyphens (see CONTRIBUTING.md §Branches).
• Each commit message follows Conventional Commits.
• Small PR is a single squashable commit; or, for a large PR, every commit is meaningful, well-formed, and independently reviewable (see CONTRIBUTING.md §Pull Requests).
• N/A — This PR intentionally targets InfiniLM-v0.2.9c; it is cleanly based on that release branch rather than current main.
• No fixup! / squash! / wip commits remain.
• N/A — The PR, branch, and commit already use the current Conventional Commits format.

Scope and Design

• Changes are minimal — nothing unrelated to the stated motivation was added (CONTRIBUTING.md §Code/General).
• No dead code, commented-out blocks, debug prints, printf/std::cout/print(...) left behind, or TODO without an owner and issue link.
• No unrelated formatting churn that would obscure the diff.
• Public API changes are intentional, documented, and reflected in affected callers/tests.

General Code Hygiene (applies to all languages)

• The code is self-explanatory; comments were added only where the why is non-obvious (CONTRIBUTING.md §Code/General).
• Every modified or added file ends with a single trailing newline (CONTRIBUTING.md §Code/General).
• No trailing whitespace, tab/space mixing, or stray BOMs.
• Identifiers in comments and error messages are wrapped in backticks (e.g. the \seqlens_k` tensor) (CONTRIBUTING.md` §Code/General).
• All comments and error messages are in English (CONTRIBUTING.md §Code/General).
• Comments and error messages are complete sentences — capitalized first letter, terminal punctuation — unless the language/framework convention says otherwise (CONTRIBUTING.md §Code/General; §Python).

C++ Specific (if C++ files changed)

• N/A — No C++ files are changed; Google C++ style does not apply.
• N/A — No C++ error or warning messages are changed.
• N/A — No C++ constructors are changed.
• N/A — No C++ allocation or ownership code is changed.
• N/A — No C++ files require formatting.
• N/A — No changes/reference to csrc/models/llama_legacy/.

Python Specific (if Python files changed)

• Code is PEP 8 compliant.
• Comments are complete English sentences, starting with a capital letter and ending with punctuation; Markdown backticks are used for code references (CONTRIBUTING.md §Python).
• Docstrings follow PEP 257 where applicable (CONTRIBUTING.md §Python).
• Changed files are formatted by scripts/format.py.
• No changes/reference to python/infinilm/auto_config.py.

Testing

• Other accelerator backends could not be tested because hardware was unavailable; maintainer validation is still requested.
• N/A — examples/test_infer.py was not run; equivalent model execution was exercised through the service path on NVIDIA and MetaX.
• N/A — examples/bench.py was not run; performance was measured with the paired HTTP service benchmark described above.
• N/A — test/bench/test_benchmark.py was not run because this change does not alter model numerics; focused scheduler tests and service output controls were used.
• Passed the service path with python/infinilm/server/inference_server.py and the paired performance harness on NVIDIA and MetaX.

Build, CI, and Tooling

• N/A — This is a Python-only change and introduces no compiled target; clean source environments were used for the focused tests and hardware validation.
• Upstream CI and Ruff are action_required; maintainer approval or /retest is required.

Documentation

• README.md was updated for the new API field and configuration.
• N/A — No user-visible breaking change is introduced.

Security and Safety

• No secrets, access tokens, internal URLs, customer data, or personal hardware identifiers have been committed.
• No third-party code is added.
• No unsafe pointer arithmetic, uninitialized reads, or missing bounds checks are introduced.