perf: reuse running reservations within each scheduler step

[T4t4KAU]

Summary

Compute running requests' remaining KV reservations once per prefill admission loop, then reuse that total for subsequent waiting requests in the same schedule() call. The previous implementation dequeued and requeued the complete running queue for every admission attempt.

Motivation

With R running requests and W waiting requests, repeated reservation accounting costs O(R × W) queue operations per prefill batch. Running requests do not advance during the prefill loop, so their reservation total can be reused. Capacity, remote-transfer reservations, and newly admitted prefill reservations continue to be checked for every candidate.

Type of Change

• perf — performance improvement with unchanged scheduling decisions

Test Results of Involved Models on Supported Platforms

Local CPU validation covered queue order, per-step refresh, standalone admission checks, live prefill/remote reservations, canceled requests, token budgets, decode-only steps, and Mamba exhaustion.

A temporary differential harness compared 500 seeded workloads across six scheduling steps (3,000 exact state comparisons) against the original scheduler from 270feb3. Compared batches, queues, page allocation/order, references, hashes, request state, and remote reservations, including local prefix hits, asynchronous remote transfers, failed transfers, and cancellation. Combined application with #576, #577, and #578 was also validated locally.

These checks were performed during development; no unit-test or test-support files are included in the final PR diff. Repository formatting and git diff --check passed.

Full-model single-request, offline, sanity, and service tests were not run: local validation is limited to Python scheduler/cache metadata without model weights. Accelerator/model throughput is unmeasured.

Benchmark / Performance Impact

Intel Core i5-14600KF, Python 3.12.13, integer metadata (dtype N/A), CPU not pinned. Timed the entire schedule() against 270feb3 with perf_counter_ns. Fresh fixture per call, setup/teardown excluded; 10 warmups, median of 7 rounds of 30 calls. Pool: 4,096 pages, 256 tokens/page; prefix caching off; waiting prompts 16 tokens; running prompts one token plus one generated token; all requests have a 512-token output limit. Batch size equals waiting-request count, token budget 16,384. Temporary local harness is not committed.

Running requests	Waiting requests	Before (µs/step)	After (µs/step)
0	32	96.100	91.380
1	1	6.682	6.786
32	16	353.776	72.539
128	32	2492.319	228.562
512	64	19583.171	987.492

Benefits grow with both queue sizes. A single candidate has no repeated work to eliminate and measured about 0.1 µs slower. These are local scheduler microbenchmarks, not model throughput.

Notes for Reviewers

• Directly based on 270feb3 / InfiniLM-v0.2.9c; no dependency on the other optimization PRs.
• Snapshot lifetime is one schedule() call, computed only when admission needs it. Decode-only steps and Mamba-full rejection avoid the scan.
• can_accept_request gains an optional keyword-only running_required_blocks argument. Existing calls remain compatible and calculate fresh reservations by default; a supplied total must describe the current prefill loop.
• No long-lived reservation counter or invalidation machinery is introduced.

CI / ChatOps

Manual CI has not been triggered; model/platform validation remains outstanding.

Checklist

• Conventional Commits title/commit, matching branch, squashable commit history, no merge/fixup/WIP commits.
• Direct base is the requested InfiniLM-v0.2.9c release branch rather than main.
• N/A — legacy issue-format exception.
• Minimal scope; no debug/dead code or unrelated formatting.
• Optional API argument documented; internal caller and compatibility checks updated.
• English comments/docstrings, final newlines, and whitespace checked.
• Python files checked with scripts/format.py; no auto_config.py changes.
• N/A — C++ checks/native build: Python-only changes.
• Local correctness, differential, integration, and performance checks completed; omitted model tests explained above.
• Additional platform validation and reviewer assignment.
• Manual CI run.
• N/A — user documentation/breaking-change notice: external behavior and workflows unchanged.
• No secrets, new third-party code, or unsafe memory operations.