Skip to content

feat(receipts): reference the visibility snapshot from a retrieval receipt - #163

Merged
drewstone merged 1 commit into
mainfrom
feat/retrieval-receipt-visibility-ref
Aug 21, 2026
Merged

feat(receipts): reference the visibility snapshot from a retrieval receipt#163
drewstone merged 1 commit into
mainfrom
feat/retrieval-receipt-visibility-ref

Conversation

@drewstone

Copy link
Copy Markdown
Contributor

Why

The 1.0.0 retrieval receipt embeds the complete ordered visibility snapshot, so every receipt is O(visible pages) in bytes and hashing. Measured on this branch: at 10,000 visible pages the embedded snapshot is 2,186,833 bytes per receipt; the referencing receipt is 2,192 bytes, and stays flat (2,190 bytes at 100 pages). A campaign that queries one knowledge view N times re-serialized and re-hashed the same inventory N times. That is evidence duplication, not useful redundancy (#154).

The three facts stay separate and the join does not weaken: which exact ordered page versions were visible, which exact ranked results came back, which returned rank a consumer selected.

Reconcile (Rule 1 — proved absent before writing)

  • git grep KnowledgeVisibilityRef|assertKnowledgeRetrievalMatchesVisibilityArtifact|encodeKnowledgeVisibilitySnapshot origin/main -- src tests → no matches. Nothing to extend; this is the first implementation.
  • Consumers of the receipt API on origin/main: src/knowledge-use-receipts.ts, its test, README.md, docs/knowledge-use-receipts.md. Nothing else in this repo, and zero matches in agent-runtime/src. The 1.0.0 shape (merged 4 days ago in feat(evidence): prove retrieval-to-outcome knowledge use #153) has no reader, so this is a replacement, not a migration: v1 is deleted outright, with no compatibility reader.

What

  • KnowledgeRetrievalReceipt.visibility becomes KnowledgeVisibilityRef { snapshotDigest, pageCount, artifact?: { uri, digest, byteLength } }. pageCount stays in the digest material, so a truncated snapshot cannot satisfy the reference.
  • createKnowledgeRetrievalReceipt takes a precomputed visibility: KnowledgeVisibilitySnapshot (plus optional visibilityArtifact) instead of visiblePages. Results still join against snapshot.entries; only the reference is stored. The snapshot object is the cache seam: verified and indexed once per object, so N retrievals over one view cost O(results) each after the first.
  • One verification path per proof, each named for what it proves:
    • verifyKnowledgeRetrievalReceipt(receipt) — receipt shape, canonical digest, rank continuity, finite scores, well-formed reference. Makes no join claim.
    • assertKnowledgeRetrievalMatchesVisibility(receipt, snapshot | visiblePages) — recomputes digest and count, joins every result.
    • assertKnowledgeRetrievalMatchesVisibilityArtifact(receipt, loadArtifact) — loads by uri, checks stored-byte digest and byteLength, decodes, verifies snapshotDigest, then calls the same join. Unobtainable snapshot raises typed KnowledgeVisibilityUnavailableError { reason, snapshotDigest, uri?, cause? } — never an empty snapshot.
  • verifyKnowledgeVisibilitySnapshot, encodeKnowledgeVisibilitySnapshot, decodeKnowledgeVisibilitySnapshot, knowledgeVisibilityArtifactRef({ uri, bytes }) so an adapter persists and reloads a snapshot without inventing a serialization.
  • Schema version 2.0.0; every verifier refuses 1.0.0 loudly.
  • Per the issue's "Do not add": no Merkle tree, no second page registry, no SDK copy, no generic Interface artifact type, no implicit trust that matching ids mean matching bytes.

Simplification

Simplification: the join rule now has one owner (assertReceiptJoinsVisibility) instead of one copy in the verifier and another in the assert path; snapshot verification+indexing has one owner (visibilityIndexOf) used by the create, verify, and decode paths; the rank-continuity rule and the finite/normalized-score rule each collapsed from 2 copies to 1 (assertContiguousRanks, assertResultScores), routing both the create and verify paths through the same code; the embedded-snapshot shape and its assertions are deleted, not deprecated.
Net: +881 / -99 lines, 6 files; 4 duplicated rule copies removed, 1 receipt shape removed, 0 compatibility paths added. Honest accounting: this is a net addition — the artifact locator, its loader contract, and the snapshot codec are new surface the issue requires, and they are irreducible if a compact receipt is to remain verifiable.
Tests: +7, -0 of main's 15 (all carried and adapted). New: snapshot bytes round-trip and a tampered snapshot is refused (on-disk format + fail-closed); the inventory is hashed once per view across N retrievals (asserted as an observable page-read count, not wall time); a receipt cannot verify against a different snapshot with the same page ids but different bytes; the artifact verifier proves the join through stored bytes; a missing artifact is an explicit unverifiable-evidence error rather than an empty snapshot; a loader failure and a receipt with no artifact reference are both explicit; stored bytes with a wrong digest, a wrong byteLength, or a different snapshot are refused. Dropped before pushing: the 10²/10⁴ vitest bench (asserted no bound — measured numbers recorded here instead), a v1-schema-refusal test (v1 is deleted), a not.toThrow-only test, and shape/frozen assertions the type system and check:api-surface already cover.
Not done here: the Runtime/SDK adapter that persists one snapshot per knowledge view and emits trace attributes stays out of this repo — no such consumer exists yet.

Proof (local, macOS, Node 24.11.1)

  • pnpm run typecheck: 0 errors (src + contracts). pnpm run lint: 230 files, no fixes.

  • pnpm test: 688 passed, 16 skipped, 64 failed. Every failure is environment-blocked and reproduces on unmodified origin/main: 48 in tests/kb-improvement/* throwing exact knowledge candidate workflows require Linux directory descriptors on macOS, and 16 in tests/version-bump-check.test.ts, whose fixtures git commit inside throwaway repos and hit this machine's global core.hooksPath pre-commit hook (verified: the same file times out on the clean _wt/agent-knowledge-main worktree, exit 124). Zero failures in any file this PR touches.

  • Focused: src/knowledge-use-receipts.test.ts 22/22.

  • Measured with a scratch script against dist/ (not committed):

    visible pages receipt bytes snapshot (once/view) receipt over reused snapshot verify
    10² 2,190 2.84 ms 0.95 ms 0.30 ms
    10⁴ 2,192 238.24 ms 0.11 ms 0.04 ms

    Receipt bytes and per-retrieval cost are independent of the visible-page count; only the once-per-view snapshot scales with N. Embedded (v1) equivalents for contrast: 21,433 bytes at 10² and 2,186,833 bytes at 10⁴, per receipt.

  • pnpm run build: 35 files. pnpm api:surface: 938 exports across 6 entry points; diff vs origin/main is exactly the 10 new symbols. pnpm run check:version-bump: 10 export changes needing a minor, paid for by 9.0.0 -> 10.0.0 (major — the visibility field's type change is breaking and the symbol-level record cannot see it). pnpm run check:skills: valid.

  • Rebased onto origin/main after fix(write-protocol): thread pagesDirectory through the write, index, search, and chain paths #160 merged; git merge-tree --write-tree origin/main HEAD: clean.

Closes #154

…ceipt

Bind a retrieval receipt to a snapshot digest, page count, and optional
artifact locator instead of the complete ordered page inventory. The receipt
input takes a precomputed snapshot, so repeated retrievals over one view no
longer re-serialize or re-hash the inventory.

Split the verification claim: verifyKnowledgeRetrievalReceipt proves the
receipt, assertKnowledgeRetrievalMatchesVisibility proves the result-to-
snapshot join, and assertKnowledgeRetrievalMatchesVisibilityArtifact loads the
stored snapshot and proves the same join. An unobtainable snapshot raises
KnowledgeVisibilityUnavailableError.

Refuse schema version 1.0.0, which embedded the snapshot and has no reader.

Closes #154

@tangletools tangletools left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Auto-approved drewstone PR — 8bd6d55a

This PR was opened by the trusted drewstone account.

This approval is provisional and was applied by the local stand-in because the pr-reviewer webhook host is unreachable (2026-08-21). CI on this head is fully green. The full PR reviewer audit re-runs via the resweep when the service returns and will publish findings if it detects issues.

@drewstone
drewstone merged commit 4b4c837 into main Aug 21, 2026
2 checks passed
@drewstone
drewstone deleted the feat/retrieval-receipt-visibility-ref branch August 21, 2026 01:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make retrieval receipts reference, not embed, the complete visibility snapshot

2 participants