Skip to content

feat(history): record the candidate search into the canonical ledger - #673

Merged
drewstone merged 1 commit into
mainfrom
feat/run-optimization-search-ledger
Aug 21, 2026
Merged

feat(history): record the candidate search into the canonical ledger#673
drewstone merged 1 commit into
mainfrom
feat/run-optimization-search-ledger

Conversation

@drewstone

Copy link
Copy Markdown
Contributor

Reconcile

git grep -n "openSearchLedger(\|searchLedger" origin/main -- src outside search-ledger*.ts returns nothing: the ledger has no caller in its own package, exactly as #633 states. git grep "search-plan-extended" origin/main returns nothing. Nothing in the last 60 merged PRs emits ledger events. This PR wires the existing FileSearchLedger; it does not add a second lineage record.

Why

FileSearchLedger implements a hash-chained, fsynced, replayable search audit with multi-parent lineage and a hard generation invariant — and had no caller in its own package. runOptimization emitted nothing, so the search loop and the lineage recorder never met, and docs/search-history-receipts.md:67 ("first-party optimizer adapters should do this automatically") was unfulfilled: no first-party method returned searchHistory, so compareOptimizationMethods({ searchHistoryPolicy: 'require-complete' }) refused every one of them. A downstream consumer that tried to adopt the ledger for a rolling search hit four refusals, one of which — no plan extension — has no workaround inside one ledger.

What

  • runOptimization({ searchLedger }) and selfImprove({ searchLedger }) record the search and return searchHistory: SearchHistoryReceipt. One recorder (src/campaign/search-ledger-recording.ts) emits the plan (slots = generations x populationSize, one candidate-generation operation per generation, one selection operation, one task per designed scenario-replicate cell), one registration per candidate carrying the exact parent surface it mutated (PR feat(optimization): let a policy choose the parent surface each generation #669's selectParent decides which), one attempt per scored cell with the cell's own outcome and accounting, one decision per candidate, and the terminal event.
  • Honest completion. The terminal event is appended only when canonical replay already accounts for the whole planned denominator. An interrupted search, or a candidate that left a designed cell unscored, stays in-progress, and the receipt reports the exact gap rather than claiming a closed search.
  • Identity the campaign cannot infer is declared once (agent, proposer, search, model). A measured value wins wherever execution reported one; a cell that ran a moving model alias is refused rather than recorded as an immutable identity.
  • Rolling search: new search-plan-extended event appends candidate slots and operations. Replay merges the first plan with every extension, the generation invariant continues across rounds, and the planless refusal is untouched. The planned task denominator does not extend — that would reopen candidates that already closed theirs. One ledger per open-ended search stays the rule; an external parent enters as a generation-0 registration whose surface artifact references the prior ledger. The loop itself uses the extension when a proposer returns more candidates than the planned population.
  • GEPA feeds the same ledger. recordCandidatePopulationSearch() turns GEPA's candidate-population artifact (its parent graph and per-scenario selection scores) into the same event stream, wired behind gepaOptimizationMethod({ searchLedger: { identity } }). This is the artifact becoming a source for the canonical ledger, not a parallel lineage.
  • Docs: docs/search-history-receipts.md gains the loop recipe, the identity contract, and the rolling-search rules. Version 0.152.0 -> 0.153.0 with the Python client and both source digests in lockstep.

Proof

  • pnpm typecheck, pnpm typecheck:examples, pnpm typecheck:scripts, pnpm lint (Biome, 723 files): clean.
  • pnpm test (full): every failing file re-run serially passes except the environment-bound set that also fails on clean origin/main (GEPA/SkillOpt Python-bridge suites without the uv venv CI installs, git-worktree suites, macOS /private/var tmpdir symlink). The full parallel run on this machine reports ~13 timeout failures at the 5s per-test limit that pass individually — including src/campaign/search-ledger.test.ts (37 tests, 5.5s alone). One genuine regression the serial re-run caught is fixed in this branch: a version bump had overwritten ANALYST_BENCHMARK_EVIDENCE_DEPENDENCY_LOCK_SHA256, a historical constant, which broke src/analyst/benchmark-reference-result.test.ts; only the live dependency-lock digest moves now, and that suite is 5/5 green.
  • pnpm build: clean (tsdown + OpenAPI emit), exit 0.
  • pnpm verify:package: clean, exit 0 (publint, attw, packed-export checks, source digests, evidence index fresh).
  • Focused: vitest run src/campaign/search-ledger.test.ts src/campaign/presets/run-optimization-search-ledger.test.ts src/campaign/presets/compare-optimization-methods-history.test.ts — 120 tests, 118 passed and the 2 failures were the digest regression above, now green.

Simplification: FileSearchLedger gains its first caller instead of a second lineage recorder — GEPA's candidate-population artifact now feeds the SAME ledger through recordCandidatePopulationSearch(), so the package has one lineage record with two sources, not two records. The projector's four scattered planEvent.plan.candidateSlots.find(...) / .operations.find(...) scans collapse into two effective-plan maps that plan and extension both write, so a rolling search cannot diverge from the plan-only path.
Net: +1766 / -217 lines, 20 files; 4 ad-hoc plan scans -> 2 maps; 0 new lineage formats.
Tests: +6 (rolling-round replay from durable bytes; the planless refusal the downstream consumer hit; an extended slot left open still blocks completion; the loop's five event kinds with a complete, digest-verified receipt; a failed cell reporting the exact gap instead of a closed search; require-complete accepting a first-party method), -3 deleted (two selectParent unit assertions that restated the type, and a selfImprove forwarding change-detector whose 2-generation run also pushed its file over the 10s limit).
Not done here: the ledger records one attempt per cell with maxAttempts: 1; #670's in-run cell retry (merged after this branch started) means a retried cell now has attempts the ledger sees only as its final result. Recording each retry attempt is a follow-up that belongs with the retry owner.

Closes #633

@tangletools tangletools left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Auto-approved drewstone PR — ace153e1

This PR was opened by the trusted drewstone account.

This approval is provisional and was applied by the local stand-in because the pr-reviewer webhook host is unreachable (2026-08-21). CI on this head is fully green. The full PR reviewer audit re-runs via the resweep when the service returns and will publish findings if it detects issues.

@drewstone
drewstone merged commit 6c2bab1 into main Aug 21, 2026
2 checks passed
@drewstone
drewstone deleted the feat/run-optimization-search-ledger branch August 21, 2026 04:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

FileSearchLedger has no caller in its own package and refuses rolling searches

2 participants