feat(statistics): reconstruct a sequential e-process after a restart

CHANGELOG.md

@@ -4,7 +4,11 @@ All notable changes to `@tangle-network/agent-eval` and its sibling `agent-eval-
 
 ---
 
-## [Unreleased]
+## [0.154.0] — 2026-08-21
+
+### Added
+
+- Sequential state reconstruction after a process restart (#411, criterion 2). `EProcessState` now carries the running sums `sumX` and `varSum` next to wealth, n, and the parameters, so the snapshot `state()` returns is sufficient to continue the betting test-martingale. `eProcess({ resume })` rebuilds a process from such a snapshot, and `sequentialPairedGate({ resume })` rebuilds the gate's observe-stream from the new exported `SequentialStreamState` (the e-process state plus the gate decision). A process or gate interrupted at any n and resumed from a JSON round-trip of its state produces the identical wealth sequence, observation sequence, decision, and final state as an uninterrupted run. The snapshot never supplies the parameters: a snapshot recorded under a different alpha, maxBet, null boundary, or threshold is refused with a `ValidationError`, as is one whose fields cannot all be true at once (running sums out of range, a latch without its n, a gate decision this configuration could not have reached). `sequentialDecide` needs no snapshot: it is replayed from the generation history it is handed.
 
 ---
 

clients/python/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "agent-eval-rpc"
-version = "0.153.0"
+version = "0.154.0"
 description = "Python RPC client, official optimizer bridge, and DSPy metric adapter for @tangle-network/agent-eval."
 readme = "README.md"
 requires-python = ">=3.10"

clients/python/src/agent_eval_rpc/__init__.py

@@ -53,7 +53,7 @@
 try:
     __version__ = version("agent-eval-rpc")
 except PackageNotFoundError:
-    __version__ = "0.153.0"
+    __version__ = "0.154.0"
 
 __all__ = [
     "Client",

docs/design/statistics-decisions.md

@@ -90,7 +90,7 @@ These 17 matched their references with zero mismatches and stay as they are, pin
 | `wilson` | keep-with-CI-oracle | Matched the closed form to `< 1e-8` on all 13 `(successes, n)` pairs including `0/1`, `10/10`, `1/1000`, `0/0`. Correctly asymmetric at the boundaries: `wilson(0, 10)` returns `[0, 0.27753280]`. |
 | `passAtK` | keep-with-CI-oracle | Chen et al. 2021 unbiased estimator, exhaustively verified for every `(n, c, k)` with `n = 1..6` against exact integer `math.comb`: zero mismatches. Stable at scale (`n=1000, c=3, k=100` agreed to `1.11e-16`). `passAtK(10, 3, 5) = 0.9166666667`. |
 | `corpusInterRaterAgreement` | keep-with-CI-oracle | The ICC(2,1) it surfaces matched a hand-derived two-way random-effects ANOVA reference to `< 1e-9` on 5 matrices, including the inverted case at `−1.959459`. This is a genuinely different and correct computation from `interRaterReliability`: it pivots to a proper items × judges matrix and delegates to `continuousAgreement`. Its fail-loud contract behaves — empty input, fewer than two judges, fewer than two common items, duplicate records, and absent dimensions all throw `ValidationError`. |
-| `eProcess` | keep-with-CI-oracle | Betting test-martingale. Empirical type-I over 4,000 sequences of 200 observations at `α = 0.05`: 2.40 % at the null boundary, 0.00 % in the null interior, 1.33 % on continuous uniform — all inside Ville's bound. Power at `E[x] = 0.7` is 99.775 %. The predictability invariant holds: the first update leaves wealth at exactly 1. |
+| `eProcess` | keep-with-CI-oracle | Betting test-martingale. Empirical type-I over 4,000 sequences of 200 observations at `α = 0.05`: 2.40 % at the null boundary, 0.00 % in the null interior, 1.33 % on continuous uniform — all inside Ville's bound. Power at `E[x] = 0.7` is 99.775 %. The predictability invariant holds: the first update leaves wealth at exactly 1. Restart reconstruction: `state()` carries the running sums (`sumX`, `varSum`) next to wealth and n, and `eProcess({ resume })` rebuilds the process from that snapshot; a process interrupted at any n and resumed from a JSON round-trip of its state reproduces the uninterrupted wealth sequence and decision exactly (`sequential-eprocess.test.ts`, interruptions at n = 0, 1, 7, 30, 119 and at the crossing), and `sequentialPairedGate({ resume })` does the same for the gate's observe-stream. A snapshot recorded under other parameters, or one whose fields cannot all be true at once, is refused. |
 | `holm` | keep-with-CI-oracle | Matched `statsmodels.multipletests` to `1e-9` on a 6-value reference vector, and uses `≤` at the boundary, which is the correct rule. It also validates both `alpha` and the p range, which `bonferroni` does not. |
 | `zQuantile` | keep-with-CI-oracle | Acklam inverse-normal, structurally independent of `normalCdf`. This independence is why the sample-size functions were untouched by the CDF defect, and why `mcnemarPower`'s failure to invert `mcnemarRequiredN` was a valid detector of it. |
 | `mcnemarRequiredN` | keep-with-CI-oracle | Matched the Lachin closed form exactly on all 4 parameter sets: `234`, `77`, `155`, `Infinity`. Unchanged by the CDF fix. Round-trip against the repaired `mcnemarPower` now holds across 16 configurations: power at `requiredN` meets the target with overshoot `≤ 0.0049`, and power at `requiredN − 1` is below target in all 16. |

package.json

@@ -1,6 +1,6 @@
 {
   "name": "@tangle-network/agent-eval",
-  "version": "0.153.0",
+  "version": "0.154.0",
   "description": "Evaluate and improve AI agents from runs, traces, judges, and feedback. Compare candidates, cluster failures, measure lift, and gate releases.",
   "homepage": "https://github.com/tangle-network/agent-eval#readme",
   "repository": {

src/analyst/benchmark-implementation.ts

@@ -10,7 +10,7 @@ export const ANALYST_BENCHMARK_DEPENDENCY_LOCK_FILES = Object.freeze([
 ])
 
 export const ANALYST_BENCHMARK_DEPENDENCY_LOCK_SHA256 =
-  'eba5216907300465abf4fdb084c04156d4c247a122645a2240690c5349500aff'
+  'c306327eaf924a0f51ede71f5cd5e4efc17fc1dc4920d80e03a71a6c98a6e87d'
 
 /** The published benchmark evidence was produced at this package version, by
  * the retired one-shot direct runner, before trace analysts moved to the
@@ -137,7 +137,7 @@ export const ANALYST_BENCHMARK_IMPLEMENTATION_FILES = Object.freeze([
 ])
 
 export const ANALYST_BENCHMARK_IMPLEMENTATION_SHA256 =
-  'a42b96f7d7329cc5b1f4f7ee81d387dd60664f6719a03686f1a74b2c315e08e9'
+  '3bae068f5efdeb37ea3af721b0771848a28148e1a2b21f2d22afd3292ffa1041'
 
 export function analystBenchmarkImplementationDigest() {
   return ANALYST_BENCHMARK_IMPLEMENTATION_SHA256

src/campaign/gates/sequential.test.ts

@@ -2,7 +2,13 @@ import { describe, expect, it } from 'vitest'
 import { type HypothesisManifest, signManifest } from '../../pre-registration'
 import { eProcess, mulberry32 } from '../../statistics'
 import type { GateContext, GenerationRecord, JudgeScore, Scenario } from '../types'
-import { sequentialDecide, sequentialPairedGate } from './sequential'
+import {
+  type SequentialObservation,
+  type SequentialPairedGateOptions,
+  type SequentialStreamState,
+  sequentialDecide,
+  sequentialPairedGate,
+} from './sequential'
 
 // ── Fixtures ──────────────────────────────────────────────────────────
 
@@ -329,6 +335,200 @@ describe('sequentialPairedGate.decide — gate contract', () => {
   })
 })
 
+// ── sequentialPairedGate — restart reconstruction ─────────────────────
+
+describe('sequentialPairedGate.resume — reconstruction after a process restart', () => {
+  const opts = { alpha: 0.05, minN: 5, maxN: 80, maxBet: 0.5, scale: 1 }
+
+  /** Paired deltas with a real edge (≈ +0.3 on scale 1) so the reference
+   *  promotes inside the budget. */
+  function deltas(seed: number, length: number): number[] {
+    const rng = mulberry32(seed)
+    return Array.from({ length }, () => Math.min(1, Math.max(-1, 0.3 + (rng() - 0.5) * 1.2)))
+  }
+
+  function observeAll(
+    ds: number[],
+    o: SequentialPairedGateOptions = opts,
+  ): SequentialObservation[] {
+    const gate = sequentialPairedGate(o)
+    const out: SequentialObservation[] = []
+    for (const d of ds) out.push(gate.observe(d))
+    return out
+  }
+
+  /** Observe the first k deltas, persist `state()` through JSON, rebuild
+   *  the gate with `resume`, and observe the rest. */
+  function observeWithRestart(
+    ds: number[],
+    k: number,
+    o: SequentialPairedGateOptions = opts,
+  ): { observations: SequentialObservation[]; final: SequentialStreamState } {
+    const first = sequentialPairedGate(o)
+    const observations: SequentialObservation[] = []
+    for (const d of ds.slice(0, k)) observations.push(first.observe(d))
+    const persisted = JSON.parse(JSON.stringify(first.state())) as SequentialStreamState
+    const second = sequentialPairedGate({ ...o, resume: persisted })
+    for (const d of ds.slice(k)) observations.push(second.observe(d))
+    return { observations, final: second.state() }
+  }
+
+  const ds = deltas(99, 60)
+  const reference = observeAll(ds)
+  const promoteAt = reference.findIndex((o) => o.decision === 'promote') + 1
+
+  it('the reference stream promotes inside the budget (fixture has an edge)', () => {
+    expect(promoteAt).toBeGreaterThan(opts.minN)
+    expect(promoteAt).toBeLessThan(ds.length)
+  })
+
+  it.each([0, 1, 4, 5, 20, 59])(
+    'interrupting at n=%i and resuming from state() yields the identical observation sequence and final state',
+    (k) => {
+      const resumed = observeWithRestart(ds, k)
+      expect(resumed.observations).toEqual(reference)
+      const uninterrupted = sequentialPairedGate(opts)
+      for (const d of ds) uninterrupted.observe(d)
+      expect(resumed.final).toEqual(uninterrupted.state())
+    },
+  )
+
+  it('interrupting around the promote crossing keeps the sticky decision and its n', () => {
+    for (const k of [promoteAt - 1, promoteAt, promoteAt + 1]) {
+      const resumed = observeWithRestart(ds, k)
+      expect(resumed.observations).toEqual(reference)
+      expect(resumed.final.decision).toBe('promote')
+    }
+  })
+
+  it('a stream resumed after promote keeps the sticky decision against contrary evidence', () => {
+    const first = sequentialPairedGate(opts)
+    for (const d of ds.slice(0, promoteAt + 3)) first.observe(d)
+    const second = sequentialPairedGate({ ...opts, resume: first.state() })
+    const next = second.observe(-0.9)
+    expect(next.decision).toBe('promote')
+    expect(next.n).toBe(promoteAt + 4)
+  })
+
+  it('a stream resumed at undecided-at-maxN refuses further observations exactly like the original', () => {
+    const flat = Array.from({ length: 6 }, () => 0)
+    const small = { alpha: 0.05, minN: 5, maxN: 6 }
+    const first = sequentialPairedGate(small)
+    for (const d of flat) first.observe(d)
+    expect(first.state().decision).toBe('undecided-at-maxN')
+    const second = sequentialPairedGate({ ...small, resume: first.state() })
+    expect(second.state()).toEqual(first.state())
+    expect(() => second.observe(0)).toThrow(/optional stopping/)
+  })
+
+  it('a resumed gate reaches undecided-at-maxN at the same n as the uninterrupted gate', () => {
+    const noise = Array.from({ length: 30 }, (_, i) => (i % 2 === 0 ? 0.05 : -0.05))
+    const small = { alpha: 0.05, minN: 5, maxN: 30 }
+    const ref = observeAll(noise, small)
+    expect(ref[ref.length - 1]!.decision).toBe('undecided-at-maxN')
+    const resumed = observeWithRestart(noise, 17, small)
+    expect(resumed.observations).toEqual(ref)
+  })
+
+  it('resume under a bound manifest continues the registered statistic', async () => {
+    const signed = await signManifest(manifestBase())
+    const bound = { preRegistration: signed, minN: 5 }
+    const ref = observeAll(ds.slice(0, 30), bound)
+    const first = sequentialPairedGate(bound)
+    for (const d of ds.slice(0, 9)) first.observe(d)
+    const second = sequentialPairedGate({ ...bound, resume: first.state() })
+    const tail = ds.slice(9, 30).map((d) => second.observe(d))
+    expect([...ds.slice(0, 9).map((_, i) => ref[i]!), ...tail]).toEqual(ref)
+  })
+
+  it('decide(ctx) is unaffected by resume — it always runs its own fresh stream', async () => {
+    const first = sequentialPairedGate(opts)
+    for (const d of ds.slice(0, 10)) first.observe(d)
+    const resumed = sequentialPairedGate({ ...opts, resume: first.state() })
+    const fresh = sequentialPairedGate(opts)
+    const ctx = ctxFrom(
+      Array.from({ length: 12 }, (_, i) => ({
+        scenarioId: `s${i}`,
+        reps: 2,
+        candidate: 0.8,
+        baseline: 0.5,
+      })),
+    )
+    const [a, b] = await Promise.all([resumed.decide(ctx), fresh.decide(ctx)])
+    expect(a).toEqual(b)
+    expect(resumed.state().n).toBe(10)
+  })
+
+  it('refuses a snapshot recorded under different e-process parameters (alpha / minEffect / maxBet)', async () => {
+    const first = sequentialPairedGate(opts)
+    for (const d of ds.slice(0, 10)) first.observe(d)
+    const snap = first.state()
+    expect(() => sequentialPairedGate({ ...opts, alpha: 0.01, resume: snap })).toThrow(
+      /cannot resume — snapshot alpha=0\.05 differs from the process alpha=0\.01/,
+    )
+    expect(() => sequentialPairedGate({ ...opts, maxBet: 0.3, resume: snap })).toThrow(
+      /snapshot maxBet=0\.5 differs/,
+    )
+    // A manifest with minEffect shifts the null boundary: a snapshot taken at
+    // minEffect 0 does not continue a minEffect 0.2 statistic.
+    const shifted = await signManifest({ ...manifestBase(), minEffect: 0.2 })
+    expect(() => sequentialPairedGate({ preRegistration: shifted, minN: 5, resume: snap })).toThrow(
+      /snapshot nullMean=0\.5 differs from the process nullMean=0\.6/,
+    )
+    expect(() => sequentialPairedGate({ ...opts, resume: { ...snap, threshold: 7 } })).toThrow(
+      /snapshot threshold=7 differs/,
+    )
+  })
+
+  it('refuses a snapshot whose gate decision this configuration could not have reached', () => {
+    const first = sequentialPairedGate(opts)
+    for (const d of ds.slice(0, 10)) first.observe(d)
+    const snap = first.state()
+    expect(snap.decision).toBe('continue')
+    expect(() => sequentialPairedGate({ ...opts, maxN: 9, resume: snap })).toThrow(
+      /snapshot n=10 exceeds the pre-registered maxN=9/,
+    )
+    expect(() => sequentialPairedGate({ ...opts, maxN: 10, resume: snap })).toThrow(
+      /decision 'continue' at n=10 with maxN=10; the stream is finished/,
+    )
+    expect(() =>
+      sequentialPairedGate({
+        ...opts,
+        resume: { ...snap, decision: 'promote', n: 3, sumX: 1, varSum: 0.1 },
+      }),
+    ).toThrow(/decision 'promote' at n=3 is below minN=5/)
+    expect(() =>
+      sequentialPairedGate({ ...opts, resume: { ...snap, decision: 'undecided-at-maxN' } }),
+    ).toThrow(/decision 'undecided-at-maxN' at n=10 does not match maxN=80/)
+    expect(() =>
+      sequentialPairedGate({
+        ...opts,
+        resume: { ...snap, decision: 'continue', wealth: 25, decided: true, decidedAtN: 9 },
+      }),
+    ).toThrow(
+      /decision 'continue' at n=10 ≥ minN=5 with e-value 25 ≥ 1\/α=20; this stream would have promoted/,
+    )
+    expect(() =>
+      sequentialPairedGate({
+        ...opts,
+        resume: { ...snap, decision: 'later' as unknown as SequentialStreamState['decision'] },
+      }),
+    ).toThrow(/unknown decision 'later'/)
+  })
+
+  it('refuses a tampered e-process snapshot through the core validator', () => {
+    const first = sequentialPairedGate(opts)
+    for (const d of ds.slice(0, 10)) first.observe(d)
+    const snap = first.state()
+    expect(() => sequentialPairedGate({ ...opts, resume: { ...snap, sumX: 11 } })).toThrow(
+      /eProcess: cannot resume — sumX must lie in \[0, n=10\]/,
+    )
+    expect(() => sequentialPairedGate({ ...opts, resume: { ...snap, wealth: -2 } })).toThrow(
+      /wealth must be a finite positive number/,
+    )
+  })
+})
+
 // ── Pre-registration binding ──────────────────────────────────────────
 
 describe('sequentialPairedGate — pre-registration binding', () => {