docs: steady-state metrics reporting design proposal by nv-alicheng · Pull Request #447 · mlcommons/endpoints

nv-alicheng · 2026-08-13T23:55:24Z

Post-processing step (and ad-hoc CLI over events.jsonl) that reports steady-state metrics — super-pass windowing, guarded drain-tail cut, CoV ensemble + trend gate (drift up/down), hairball, edge-case handling.

What does this PR do?

Adds a design doc for a new steady-state window detection and metrics reporting feature

Type of change

Bug fix
New feature
Documentation update
Refactor/cleanup

Related issues

cc @arekay-nv to tag any related issues

Testing

Tests added/updated
All tests pass locally
Manual testing completed

Checklist

Code follows project style
Pre-commit hooks pass
Documentation updated (if needed)

github-actions · 2026-08-13T23:55:32Z

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

nvzhihanj · 2026-08-14T01:02:16Z

+  a run killed by `SIGKILL`/OOM before the log is flushed yields a truncated log;
+  the step must degrade to a best-effort result with a status flag rather than
+  fail (§5.6).


Maybe I am not reading it in the right way - but I thought it should fail if the original run fails?

What does "failing" mean here — a request timeout or a process crash? My intent is that a timeout should still yield a best-effort (partial) result over whatever was logged, while a hard crash of the post-process can just fail. Could you clarify which case you had in mind so I word the edge-case handling to match?

Two run-failure modes, handled differently. A timeout / interrupt (SIGTERM, drain-timeout) still flushes a partial event log, so the step computes a best-effort steady result over what was logged and marks status = partial — the same way the report already flags interrupted runs. A hard crash before the log is flushed (SIGKILL/OOM) yields a truncated log; the step still degrades to best-effort + partial rather than erroring. The post-process itself never hard-fails a run. Separately, a run that completes but never reaches steady state is reported with the offending metrics flagged unstable (§5.6), not failed — whether that should count as an invalid submission is the open TF question in §6. Does that cover the case you had in mind?

This reply was drafted by Claude (Claude Code) and posted on my behalf.

arekay-nv · 2026-08-17T20:03:01Z

+- **Relative active concurrency.** In-flight count as a percentage of the
+  max-concurrency budget (100% at saturation).
+- **Issue-time window.** A contiguous range of super-passes `[start, end)`. Its
+  measured set is every request _issued_ in that range. Throughput uses the


This might requires some clarification - as far as i understand from below a sample issued but whose response is not received has no logged TTFT and is naturally excluded. So either it is issue only or issue with at least one event to be included in a window.

Membership is issued within the window; a sample then contributes to a given metric only if that metric was logged inside the window (e.g. a first-token event for TTFT). So an issued-but-no-response sample is a member of the issue set (it counts toward the offered / issue-time denominator) but contributes no TTFT/TPOT — hence "naturally excluded" from the latency families, not from issuance. I'll reword §5.1 to separate membership from per-metric contribution so it's unambiguous.

This reply was drafted by Claude (Claude Code) and posted on my behalf.

arekay-nv · 2026-08-17T20:46:35Z

+  (everything issued at `t=0`), so there is no client-side issue-time window and no
+  client-side drain boundary. The step detects the mode and finds the steady region


for TPOT calculations, we could potentially use the completion-first_token time, but that would require the server to not evict requests in decode phase.

Agreed — that's exactly the §5.6 caveat: the timestamp-derived TPOT (complete − first_token) assumes the request stays resident through decode; if the server evicts/pages it mid-decode, the span includes non-decode queue time and inflates TPOT, so a server-reported per-token count is preferred where eviction is possible. This is also why the design now adds staircase / level-shift detection (Pettitt + multi-plateau): KV-cache-eviction-driven degradation toward the end of long runs shows up as a step in TPOT, which we flag as an anomaly rather than fold into the steady number.

This reply was drafted by Claude (Claude Code) and posted on my behalf.

arekay-nv · 2026-08-17T20:49:31Z

+- **ISL / OSL** are reported for analytics, not as validation numbers: once the
+  window is allowed to drop samples (not enforcing full dataset-pass boundaries),
+  the input/output-length distribution is skewed relative to the constructed
+  dataset and is no longer a meaningful validation quantity, though it remains
+  useful for analysis.


If the window is allowed to drop samples, should the ISL/OSL statistics be computed for the samples included in the measurement?

Yes — ISL/OSL are computed over the included (windowed) samples today, which skews them relative to the constructed dataset once the window is allowed to drop samples. §5.8 flags them as analytics-only, not validation numbers. Whether the official artifact should report them over the full issued set instead (or carry both) is deferred to the benchmark TF (§6, "ISL/OSL reporting basis").

This reply was drafted by Claude (Claude Code) and posted on my behalf.

arekay-nv · 2026-08-17T20:52:00Z

+
+# 6 Open questions {#6-open-questions}
+
+- **Default parameters.** The warmup band, trailing-window length, per-percentile


Can we ignore the first pass/super-pass over the dataset as a warmup pass and the other parameters can be benchmark specific decided by TF?

Agreed, and captured as the §6 open question "First pass as warmup (task force)." The current design infers the warmup band adaptively from the data, but treating the first full pass as warmup-by-construction is a reasonable TF-set default; the accompanying band / window / bound defaults would be the TF's to fix.

This reply was drafted by Claude (Claude Code) and posted on my behalf.

arekay-nv · 2026-08-17T20:53:42Z

+- **Guard distance measure.** The exact two-sample statistic and acceptance bound
+  for the per-token-invariance guard (§5.4) need to be fixed.
+- **No-steady-state runs.** When no steady state is found (a tracked metric is
+  Drifting Up), should the run be reported _invalid_ — analogous to legacy


We should declare the gates for each benchmark and also define whether a submission should be invalid if it fails a certain gate. It should be a benchmark TF decision on what gates to include.

Agreed — captured in §6 ("Per-benchmark gates and invalidation"): the CoV bounds and trend thresholds are tunable per benchmark, and whether failing a gate invalidates a submission versus reports-with-flags is explicitly a benchmark-TF decision, not fixed by this proposal.

This reply was drafted by Claude (Claude Code) and posted on my behalf.

hvagadia · 2026-08-19T18:40:01Z

+  client. The window measures the sustained behavior of a healthy server; a
+  genuinely unhealthy server produces bad-but-real numbers, which the drift
+  detector (§5.5) distinguishes from transient pollution.
+- **Super-pass.** The atomic unit of the analysis: a contiguous block of requests


Is there an assumption here that we need more than one super-pass? How many would be needed to get a good estimate?

Yes. The coverage-status table (§5.6) requires ≥ warmup + 1 full super-passes to window normally; below that it's best-effort / low-confidence. For a trustworthy estimate the trend test needs ≥ 4 super-passes (its minimum), so in practice you want ~5–6+ post-warmup super-passes. Fewer still yields a number, but flagged insufficient_passes.

This reply was drafted by Claude (Claude Code) and posted on my behalf.

hvagadia · 2026-08-19T18:43:02Z

+  [ ingest -> per-super-pass series ]     issue-order bucketing
+     |
+     v
+  [ adaptive warmup crop ]                remove the ramp


How is ramp detected?

Band-adaptive warmup crop (§5.4): estimate the steady level from the median of the series' back half for a driver metric (e.g. TTFT p99), then drop leading super-passes whose value exceeds steady × (1 + band). Band-based (not slope-based) so a single settling outlier can't collapse the crop, and only high leading values are dropped so a later upward drift is still measurable. (The reference script currently exposes a fixed --warmup; the adaptive crop is specced but not yet ported to it.)

This reply was drafted by Claude (Claude Code) and posted on my behalf.

hvagadia · 2026-08-19T18:49:35Z

+**Ensemble and window selection.** CoV alone is insufficient — a window is only
+meaningful if the metric is in Plateau. In practice, over a large set of runs, no
+single `(window, bound)` fits all metrics and workloads, so the rule is run as an
+**ensemble** of preset `(window, bound)` settings. Among the settings that report


Can you explain what does "ensemble of (window, bound)" mean exactly? How are the values of window and bound decided?

It means the CoV rule is evaluated at several preset (trailing-window, bound) settings rather than one; a window passes CoV if at least one setting certifies every gated metric. No single setting fits all metrics/workloads, so the ensemble adds robustness (concordance across settings is a corroborating guardrail). The values come from sweeps over recorded runs (defaults: bounds 0.03 / 0.05 / 0.08 loosening toward the tail, trailing windows 4–5); §6 flags that these defaults should be reviewed on a wider set before being locked.

This reply was drafted by Claude (Claude Code) and posted on my behalf.

Post-processing step (and ad-hoc CLI over events.jsonl) that reports steady-state metrics — super-pass windowing, guarded drain-tail cut, CoV ensemble + trend gate (drift up/down), hairball, edge-case handling. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

…aly gates) + tests Implement the steady-window selection in scripts/steady_state_diagnostics.py: grow-from-left plateau segmentation (admissible = trend-steady + within a CoV bound on TTFT/TPOT p50+p95), first-plateau selection per the MSER maximize-precision principle, pooled TTFT/TPOT histograms + per-user/system TPS with batch-means CIs, Pettitt + multi-plateau staircase (level-shift) detection, and a whole-tail Drifting-Up gate that flags a locally-flat window sitting inside a globally-drifting metric. Adds tests (47) and a README alongside the script. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Lower the tokenizer-flush buffer for long reasoning outputs (e.g. DeepSeek-R1) to avoid OOM on large event logs; threaded through run()/build_super_pass_series. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The steady-window system TPS divided total output tokens by the completion span (first issue -> last event), so the decode drain inflated the denominator and deflated TPS -- ~12x too low on high-tail workloads (DeepSeek-R1 c28k: 63.7k -> 746.9k tok/s). Use the offered-load issue span (first issue -> last issue) per design section 5.1; track last_issue_ns, add window_issue_span_ns, and export a raw per_super_pass series for plotting. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Replace the fixed --warmup with 'auto': estimate the steady level from the driver metric's back-half median (default tpot_p50 -- smooth per super-pass, unlike the volatile TTFT) and crop leading super-passes >band off it (symmetric, since TPOT ramps up to steady). Configurable via --warmup/--warmup-band/--warmup-driver. On the DeepSeek-R1 c28k run this crops 24 ramp super-passes and moves the window from the ramp shoulder (SP13, TPOT p50 40.4ms, spurious tpot_p95 drift warning) onto the true plateau (SP24-27, TPOT p50 46.1ms, no warning). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

hvagadia · 2026-08-24T20:24:26Z

+## Run
+
+```bash
+uv run scripts/steady_state_diagnostics.py <run>/events.jsonl \


Is the expectation here that we will "fix" this command for a particular benchmark (across all systems)?

The ramp crop is data-driven off TPOT p50 (smooth per super-pass; TTFT too volatile under closed-loop admission), symmetric band around the back-half median, capped at half the run. Matches the shipped scripts/steady_state_diagnostics.py. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

nv-alicheng requested a review from a team August 13, 2026 23:55

github-actions Bot requested review from arekay-nv and nvzhihanj August 13, 2026 23:55

nvzhihanj reviewed Aug 14, 2026

View reviewed changes

Comment thread docs/steady-state-detection.md Outdated

nvzhihanj reviewed Aug 14, 2026

View reviewed changes

arekay-nv reviewed Aug 17, 2026

View reviewed changes

arekay-nv requested review from TheKanter, attafosu, mrmhodak and tianmu-li August 17, 2026 20:54

nv-alicheng force-pushed the doc/alicheng-steady-state-design branch 2 times, most recently from 5e936f6 to bad5e83 Compare August 18, 2026 17:54

arekay-nv requested a review from hvagadia August 18, 2026 18:05

hvagadia reviewed Aug 19, 2026

View reviewed changes

nv-alicheng and others added 2 commits August 20, 2026 14:21

Add script and update docs

158d294

nv-alicheng force-pushed the doc/alicheng-steady-state-design branch from bad5e83 to 158d294 Compare August 21, 2026 00:23

github-actions Bot added the size/very-large PR Review Policy: >1500 lines or >50 files label Aug 21, 2026

nv-alicheng and others added 4 commits August 21, 2026 16:20

feat(metrics): add --tokenize-batch-size to bound peak parse memory

7d2fbe4

Lower the tokenizer-flush buffer for long reasoning outputs (e.g. DeepSeek-R1) to avoid OOM on large event logs; threaded through run()/build_super_pass_series. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

hvagadia reviewed Aug 24, 2026

View reviewed changes

		(everything issued at `t=0`), so there is no client-side issue-time window and no
		client-side drain boundary. The step detects the mode and finds the steady region


		# 6 Open questions {#6-open-questions}

		- Default parameters. The warmup band, trailing-window length, per-percentile

Uh oh!

Conversation

nv-alicheng commented Aug 13, 2026

What does this PR do?

Type of change

Related issues

Testing

Checklist

Uh oh!

github-actions Bot commented Aug 13, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants