feat(campaign): retry a transient cell failure in the same slot

[drewstone]

Why

Closes tangle-network/agent-runtime#723 — the issue is filed on Runtime, but runImprovementLoop is Eval code (Runtime only re-exports it).

Observed in the agent-graphs gen2 run: one candidate cell died on a router HTTP 503. That single transport failure made candidate coverage incomplete, so runImprovementLoop refused the holdout comparison (assertCompleteHoldout: "holdout is incomplete … Refusing to compare unequal holdout results") and the loop held the baseline without ever scoring the candidate. The only remedies were a full re-run of every arm or a caller-side measurement outside the loop.

Failing closed on incomplete evidence is correct. But a 503 is distinguishable from a judge error, and the per-cell failure receipt already carries the stage and the error text, so the campaign already holds everything needed to tell them apart.

What

RunCampaignOptions.cellRetry?: { attempts, retryable } — opt-in bounded in-run retry, absent by default.

• A failed attempt the retryable predicate accepts is dispatched again in the same slot: same cellId, same seed, same cost tags, same manifest. The schedule and the baseline/candidate pairing are unchanged.
• Every attempt charges the shared cost ledger, so the final cell's costUsd, tokenUsage, and costCallIds cover all attempts.
• A retried attempt keeps its evidence at <cell>/failure-receipt.attempt-<n>.json; a final failed attempt keeps the usual <cell>/failure-receipt.json. Same CampaignCellFailureReceipt shape, only the file name differs.
• The final cell records retryAttempts.
• abortOnCellError fires only when a cell's final attempt fails.
• A cancelled campaign is never retried, and a CostAccountingIncompleteError is never retried.
• transientDispatchFailure(opts?) is the shipped predicate: failure.stage === 'dispatch' && isTransientTransportFailure(failure.error.message, opts). A judge-stage failure is never transport, so it is never retried — the dispatch already produced an artifact, and re-dispatching would score a different sample.
• selfImprove({ cellRetry }) forwards the policy to the baseline, candidate, and held-out campaigns; runImprovementLoop inherits it through the existing options spread (no new plumbing in that file).

Fail-closed default is unchanged: with no cellRetry, a failed cell is final on its first attempt and coverage stays incomplete.

Absence proof (Rule 1)

• git grep -w cellRetry origin/main -- src tests → 0 hits. No retry option existed.
• git grep -w isTransientTransportFailure origin/main → 11 hits, all of them the definition (src/campaign/transient-failure.ts:41), the barrel re-export (src/campaign/index.ts:430), and its own unit test. Zero production consumers.
• src/campaign/run-campaign.ts:282-300 on main called executeCell exactly once per slot; reuseFailedCells only re-dispatches on a resumable re-run, not in-run.

Simplification (Rule 2)

Simplification: the exported-but-uncalled isTransientTransportFailure classifier gets its first
  and only production consumer (transientDispatchFailure); retry reuses the existing executeCell
  path, the existing shared CostLedger, and the existing CampaignCellFailureReceipt shape — no
  second classifier, no parallel dispatch, no second receipt schema. cellRetry threads exactly
  the way selectParent (#669) threads: one option on the options type, inherited by
  run-improvement-loop through the existing spread, passed explicitly in selfImprove.
Net: +576 / -37 lines over 19 files (src non-test +163/-32; tests +387/-1), 0 paths removed,
  0 copies collapsed.
Tests: +12, -4 deleted before pushing. Kept: 503-retried-to-success yields a scorable cell with
  the failed attempt's receipt retained and BOTH attempts charged to the ledger (a silently
  unbilled retry is the dangerous bug); attempts exhausted still leaves coverage incomplete and
  the loop still throws (fail-closed preserved, at both the campaign and runImprovementLoop
  layer); a judge-stage failure is never retried (at the predicate and through the campaign);
  abort deferral and the abort that still fires once attempts run out; no retry after
  cancellation (a retry storm after abort burns tokens); a retried cell still caches, so a
  resume does not re-bill it; selfImprove forwards the policy. Deleted: the "predicate rejects
  a non-transport failure" campaign case (the predicate's own unit test covers it), the
  malformed-policy refusal (type validation in disguise), the "deadline is not transient by
  default" case (already in isTransientTransportFailure's own test), and the option-forwarding
  case for the classifier options (change detector). Also dropped from the kept tests: the
  same-cellId/seed pin (the lane passes one `slot` object to every attempt — obvious by
  reading) and the receipt-filename mirror assertions.
Not done here: honest answer — this is a net addition and it is irreducible. The retry loop is
  ~14 new lines in the lane and ~10 in the receipt write; the rest is the option contract, the
  predicate, and the tests. Nothing on main duplicated it, so there was nothing to collapse.
  Left as a follow-up boundary: routing the runProfileMatrix segment-retry path
  (presets/segmented-profile-matrix.ts, which retries via reuseFailedCells across segments)
  through the same policy — that is its own architecture PR, not this fix.

Proof

Local, node v24.11.1, pnpm 10.34.5, on feat/campaign-cell-retry rebased onto origin/main@7541efe8 (after #669):

Check	Result
pnpm typecheck	pass
pnpm lint (biome, 719 files)	pass, no fixes applied
pnpm build (tsdown + openapi)	pass
pnpm verify:package (analyst digests, skill, model-ids, publint, attw, packed exports, evidence index)	pass — exit 0, "evidence index is fresh: 9 records"
focused: run-campaign.test.ts + transient-failure.test.ts + presets.test.ts + contract-self-improve.test.ts	133/133 pass

New tests (12), each named for the failure it prevents:

tests/campaign/run-campaign.test.ts — new runCampaign — cellRetry block, 6 cases:

1. 503 retried to success → scorable cell, retryAttempts: 1, cost summed (0.02 = 2 × 0.01) with 2 costCallIds, campaignCoverage(...).complete === true, cellsFailed: 0, the failed attempt's receipt retained at failure-receipt.attempt-1.json carrying that attempt's own error, and a rerun reuses the retried cell from cache without re-dispatching (a retried cell that failed to cache would re-bill on resume).
2. attempts exhausted (3/3 fail) → 3 dispatches, retryAttempts: 2, every attempt's receipt on disk, final receipt cost 0.03, coverage incomplete, cellsFailed: 1. Fail-closed is preserved.
3. judge-stage 503 is not retried — 1 dispatch, no attempt receipt.
4. abortOnCellError does not fire while a retryable failure has attempts left (the cell recovers) — a transient blip must not tear down the campaign.
5. abortOnCellError still rejects with the final attempt's error once retry is exhausted — retry must not swallow a real abort.
6. a cancelled campaign is never retried — no retry storm burning tokens after abort.

src/campaign/transient-failure.test.ts — new transientDispatchFailure block, 3 cases: a dispatch-stage transport failure is retried; a judge-stage failure never is, even with a 503-shaped message; a non-transport dispatch failure is scored, not retried (retrying real failures silently drops the hard cells and inflates every arm).

tests/campaign/presets.test.ts — 2 cases through the real runImprovementLoop, reproducing the incident: a transient holdout 503 is recovered and the gate ships (exactly 1 cell carries retryAttempts: 1), and with attempts exhausted the loop still throws baseline holdout is incomplete (2/3 designed cells scorable).

tests/contract-self-improve.test.ts — 1 case: the same flaky agent rejects with /holdout is incomplete/ without cellRetry and completes with it, proving selfImprove forwards the policy.

Full-suite disclosure

pnpm test on this branch: 5298 passed / 67 failed / 3 skipped across 378 files; 13 files carry the failures. None of them is caused by this change, measured rather than asserted:

• I ran the same 13 files in the reference clean origin/main worktree (no part of this change present): 11 of 13 failed there too, 53 failed / 220.
• The 2 that happened to pass in that batch I re-ran individually on an idle machine: tests/campaign/official-optimizer-abort.test.ts fails 2/2 on clean origin/main with Error: optimizer did not start (the test's detached optimizer subprocess never comes up in this sandbox), and src/sandbox-harness.test.ts passes on this branch when it is not competing for CPU — it was a load flake from running 378 files at once.

The failures are environment-dependent: git worktree creation (worktree.test.ts, 29/29 on both), sandbox harness, detached optimizer subprocesses, and network-backed benchmark fixtures. None of the 13 files imports cellRetry, transientDispatchFailure, or executeCell. The 4 files this change touches are 133/133 green. CI is the arbiter.

Docs

docs/eval-surface-map.md gains a "Failed cells: receipts and bounded retry" section under the run* primitive table — the doc that already owns campaign-cell semantics. CHANGELOG.md records 0.152.0.

Version

Minor bump 0.151.0 → 0.152.0 (#669 took 0.151.0; re-bumped from the new main after rebasing) across the five lockstep files (package.json, clients/python/pyproject.toml, clients/python/src/agent_eval_rpc/__init__.py, clients/python/uv.lock) plus the recomputed dependency-lock digest in src/analyst/benchmark-implementation.ts. The wire contract is untouched, so the Python RPC client needs no change beyond the version lockstep.

Closes tangle-network/agent-runtime#723

[tangletools]

✅ Auto-approved drewstone PR — 22fd9fe4

This PR was opened by the trusted drewstone account.

This approval is provisional and was applied by the local stand-in because the pr-reviewer webhook host is unreachable (2026-08-21). CI on this head is fully green. The full PR reviewer audit re-runs via the resweep when the service returns and will publish findings if it detects issues.