Skip to content

Evidence-tier publishing: contract sibling, latest-evidence.json, builder --evidence-release (#506) - #674

Draft
MaxGhenis wants to merge 5 commits into
mainfrom
evidence-tier-506-port
Draft

Evidence-tier publishing: contract sibling, latest-evidence.json, builder --evidence-release (#506)#674
MaxGhenis wants to merge 5 commits into
mainfrom
evidence-tier-506-port

Conversation

@MaxGhenis

Copy link
Copy Markdown
Contributor

Implements the microcosm#506 design (owner directive: the best available artifact should be publishable at all times WITHOUT weakening certified gating), generalizing the #490/#497 register pattern — adjudicate, record, carry the caveat in the artifact. Leaves #506 open for the owner to close after the first evidence publish (the current best dense artifact).

The tier, structurally

  • validate_evidence_release_dir (microcosm-data): a sibling of validate_release_dir, not a relaxation — same required files, same shape/provenance/hash/cross-manifest checks, plus a mandatory non-empty known_failures block: every recorded gate failure verbatim, each with an owner issue ref. Gate-verdict requirements (US critical-target fit, source-coverage gate.passed) are exactly what the tier records instead of enforcing. Dirty-git, hash mismatches, and build-id mismatches still refuse: the tier relaxes verdicts, never auditability.
  • Mutual exclusion is structural, both directions. Evidence manifests declare schema_version: "1-evidence" + tier: "evidence"; the certified contract rejects that schema marker no matter which gates failed, and a certified-shape manifest (no tier, no known_failures, schema 1) fails the evidence contract. validate_release_dir itself is untouched.
  • Scope: national-default releases only. Local-area roles (Release contract needs dataset-role classes: national-default checks reject valid non-default local-area artifacts #398) and UK exact-k releases (One country-agnostic gate battery: phased evaluation, unconditional reporting, per-country gate selection (#578) #611 gate battery) are refused by name — no evidence semantics have been adjudicated for them.

Publish path

  • publish_release(evidence=True) validates via the evidence contract, creates the immutable tag exactly as today, and moves only a new latest-evidence.json pointer (payload mirrors latest_pointer_payload + tier). The certified latest.json path never appears in an evidence commit — pointer isolation is pinned in both directions by tests.
  • latest_evidence_release() is the consumer; latest_release() now refuses a pointer payload naming a foreign tier (defense if an evidence payload ever landed in latest.json).
  • microcosm-publish-release --evidence; Slack alerts label the tier so they can never read as a certified release announcement.

Builder --evidence-release

Tests

49 new tests across the three suites (19 contract, 13 release/publish, 2 CLI guard, 11 builder + the #568 AST guard now also covering the evidence branch shape). Full workspace suite green (uv run pytest: exit 0 on this branch), ruff clean.

Notes for the merge gate

🤖 Generated with Claude Code

MaxGhenis and others added 5 commits August 11, 2026 18:22
validate_evidence_release_dir as a sibling of the certified contract: same
required files and shape/provenance/hash checks, plus a mandatory non-empty
known_failures block (verbatim gate-failure strings, each with an owner issue
ref). Evidence manifests declare a distinct schema marker (1-evidence) and
tier field, so the certified contract structurally rejects them in one
direction and the evidence contract rejects certified-shape manifests in the
other; validate_release_dir itself is untouched. Scoped to national-default
releases: local-area roles (#398) and UK exact-k releases (#611) are refused
pending their own adjudicated evidence semantics.

publish_release grows evidence=True: validates via the evidence contract,
publishes the immutable tag as usual, and moves only latest-evidence.json
(payload mirrors latest_pointer_payload + tier) — the certified latest.json
path never appears in an evidence commit. latest_evidence_release reads the
new pointer; latest_release now refuses a pointer payload naming a foreign
tier. The publish CLI grows --evidence. Slack alerts label the evidence tier
so they can never read as a certified release announcement.

Ported from the pre-rename populace tree (branch evidence-tier-506, commit
9012391d) onto the microcosm namespace.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… owned known_failures (#506)

On terminal-gate failure the certified path is unchanged: batched pre-export
raise, #568 weight-evidence sidecar, no H5, no manifests. Under
--evidence-release the same recorded failures instead ride into the release
manifest's known_failures block and the run continues through the H5 write,
reform smoke, take-up contract, and manifests. Every conversion point
resolves owners immediately — a failure matching neither the standing
US_EVIDENCE_FAILURE_OWNERS register (the two #506-adjudicated dense
families: SOI Table 1.4 dollar fit -> #487, QRF tail concentration ->
#481/#487) nor the --evidence-failure-owners per-run file refuses the
export. An all-green run under the flag is refused: the flag is structurally
incapable of minting a certified-shape manifest (the release manifest gets
the 1-evidence schema marker + tier + known_failures via
_evidence_release_manifest_fields, which raises on an empty record).

Release ids: --evidence-release auto-ids carry the -evidence- segment;
certified builds refuse ids that squat it. Incompatible with --exact-k (the
ladder lane keeps its own tag-only contract). Preflight and mid-build
source-stage gates, --audit-export-targets, and the dirty-worktree refusal
abort in both modes: the tier relaxes terminal gate verdicts, never
artifact auditability.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…test-evidence.json (#506)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…r paths, harden scope (#506)

Findings fixed from the cross-family adversarial review of the committed
diff:

- (HIGH) Pointer isolation was bypassable through manifest-declared root
  artifacts: an artifact entry whose path IS latest.json (or
  latest-evidence.json) uploaded at that root path. publish_release now
  refuses reserved pointer paths as root artifacts on BOTH tiers, pinned in
  both directions.
- (HIGH) known_failures was unbound from the artifact's own record. The
  evidence contract now requires every build_manifest gates.calibration
  failure verbatim in known_failures, and recomputes the certified
  critical-target verdicts requiring each breach acknowledged by name — a
  softened or dropped entry refuses. (Over-disclosure stays legal: the tier
  can under-promise, never under-disclose.)
- (MEDIUM) The evidence contract now refuses non-populace-us release ids:
  a generic id with the segment must not buy a weaker contract by
  deactivating the US-specific requirements.
- (MEDIUM) latest_release now refuses ANY tier field in latest.json —
  explicit 'certified' and null included; no certified producer writes one.
- (PLAUSIBLE, confirmed) An evidence attempt refused for an unowned failure
  now falls through to the same failed-run path as a certified gate failure,
  #568 weight-evidence sidecar included, before re-raising the refusal.
- (COVERAGE) New AST guard pins the main() conversion shape: every terminal
  'Release gates failed' raise after the #548 accumulator is conditioned on
  --evidence-release, every one before it is unconditional, the all-green
  refusal precedes the manifest write, and _build_manifests receives the
  owned record.

Adjudicated without change: the certified-path refusal of '-evidence-' ids
is the intentional reserved-namespace guard (disclosed in the PR body);
LatestPointer.tier is a deliberate additive API field, now in the changelog
fragment.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant