Signed organizational world models: intake, refinery, and a ratification bench

.gitignore

@@ -78,3 +78,11 @@ film/src/node_modules/
 verify/tla/tla2tools.jar
 verify/tla/states/
 verify/tla/*.old
+
+# Refinery run artifacts
+refinery-staged.json*
+
+# Reviewer signing keys and ratification sessions
+.ratify/
+demo-session.*
+*.reviewer.key

.impeccable.md

@@ -0,0 +1,38 @@
+# Impeccable Design Context — smesh-rust
+
+## Design Context
+
+### Users
+Internal domain experts with tribal knowledge — compliance-minded operators
+(Zuub world: dental RCM / payer operations) acting as the merge authority
+over machine-extracted world models. Desk work, careful evidence reading,
+daytime. They are legally accountable for what they sign.
+
+### Brand Personality
+Evidentiary, institutional, calm. Zuub brand family: deep navy `#05375E`
+dominant, teal `#6CDCD8` reserved for action, mid-blue `#1870C9`, pale
+navy-tinted surfaces (`#E9F1FB` family), indigo `#2B3C6A`. Three words:
+**careful, evidentiary, institutional.**
+
+### Aesthetic Direction
+"Institutional ledger": a court docket crossed with a well-set legal brief.
+Light theme (evidence reading, daytime desk work). Paper-like surfaces
+tinted toward navy; teal appears only on actions and progress. Rich inline
+SVG iconography — a consistent hand-drawn 20px/1.5px-stroke set (seal,
+scales, quote, per-format document glyphs, verdict marks) — never emoji,
+never icon fonts. Anti-references: SaaS metric dashboards, cards-in-cards,
+dark-mode-with-glow, gradient text.
+
+### Design Principles
+1. **The mesh proposes; the human disposes.** Attention ranks by lane:
+   Red (mandatory judgment) before Amber before Green (batch, spot-check).
+2. **Evidence is the interface.** Every claim renders with its verbatim
+   quote highlighted in context plus its native anchor (page / heading
+   path / message-id). Nothing asks for trust without showing its source.
+3. **Signing is ceremony.** The ratify action is visually terminal and
+   weighty — disabled until coverage is total, followed by a receipt state
+   showing the new revision hash.
+4. **Transparency is credibility.** Firewall rejections and verdict
+   rationales are visible, not hidden.
+5. **Typography:** display Familjen Grotesk, body Atkinson Hyperlegible
+   (misread-proof evidence), Red Hat Mono strictly for hashes/anchors.

Cargo.toml

@@ -6,6 +6,9 @@ members = [
     "smesh-agent",
     "smesh-bounty",
     "smesh-cli",
+    "smesh-world",
+    "smesh-refinery",
+    "smesh-ratify",
 ]
 
 [workspace.package]
@@ -32,7 +35,17 @@ async-trait = "0.1"
 
 # Crypto
 sha2 = "0.10"
-ed25519-dalek = { version = "2.1", features = ["rand_core", "serde", "pkcs8"] }
+ed25519-dalek = { version = "2.2", features = ["rand_core", "serde", "pkcs8"] }
+blake3 = "1.5"
+
+# PDF (Tier 0 intake + Meridian renderer)
+# default-features off: lopdf's `rayon` feature parses objects on the
+# global thread pool, which is nondeterministic under test parallelism.
+lopdf = { version = "0.36", default-features = false }
+pdf-extract = "0.10"
+
+# Ratification dashboard
+axum = "0.8"
 rand = "0.8"
 uuid = { version = "1.6", features = ["v4", "serde"] }
 

WORLD-MODEL.md

@@ -0,0 +1,215 @@
+# WORLD-MODEL — Signed Organizational World Models via SMESH × AION
+
+**Status:** Design accepted, scaffold in `smesh-world/`
+**Depends on:** `smesh-core` (signal field, reputation, trust), `smesh-agent` (LLM backends, tool contracts), [aion-context](https://github.com/aion-context/aion-context) (signed graph substrate)
+
+---
+
+## 1. Problem
+
+An agent without a structured model of the organization it serves does not say
+"I don't know your PTO policy." It **confabulates context**: it imports the
+nearest-neighbor organization from pretraining and speaks fluent Generic Corp —
+wrong verbiage, wrong policy, wrong org chart, delivered with full confidence.
+
+The fix is not more prompt. It is converting "fill in the blank" into
+"look up the node" — and making the *absence* of a node itself signed and
+queryable (**verifiable negative space**). An agent grounded in a signed world
+model can refuse to invent what the world does not contain.
+
+## 2. Thesis
+
+> **SMESH is how the world model earns its edges. AION is how the world model
+> proves its identity. The revision loop is how it stays alive. The human is
+> the merge authority.**
+
+- **AION = the truth plane.** Content-addressed nodes (idempotent ingestion),
+  Ed25519-signed revision chain (an agent verifies it loaded *this org's*
+  world, untampered), append-only history (replay any decision against the
+  world *as it was*), deny/allow (retraction without deletion).
+- **SMESH = the refinery.** Expert agents emit candidate nodes/edges as
+  signals. Reinforcement = independent corroboration. Decay = unsupported
+  claims fade. Contested edges escalate to a human instead of being resolved
+  by the mesh.
+- **Governing principle — separation of powers:** extractors propose,
+  verifiers judge, one non-LLM curator signs. No expert holds two of those
+  powers.
+
+## 3. Revision loop
+
+```
+raw corpus ──► Tier 0 intake ──► CDM docs ──► Tier 1 experts ──► candidate signals
+                                                                       │ reinforce / decay
+                                                                       ▼
+                                                    Tier 2 verification (adversarial)
+                                                                       │
+                                                                       ▼
+                                       staged changeset  (rev N → proposed rev N+1)
+                                                                       │
+                                                                       ▼
+                                    HUMAN RATIFICATION (dashboard, tiered lanes)
+                                                                       │
+                                                                       ▼
+                              Curator signs ratified changeset ──► world rev N+1
+                                                                       ▲
+                          new docs / drift ────────────────────────────┘
+```
+
+Nothing reaches the signed graph without passing `Ratified`. The Curator's
+signing key is only ever applied to a changeset carrying a human ratification
+record.
+
+## 4. Expert roster
+
+### Tier 0 — Intake ("Registrar of Record")
+
+Any artifact in → **Canonical Document Model (CDM)** out. Tier 1 never sees a
+raw format; adding a format touches Tier 0 only.
+
+| Concern | Design |
+|---|---|
+| Identity | `doc_id = BLAKE3(bytes)` — content-addressed, idempotent re-ingestion |
+| Detection | Deterministic sniffing first; LLM triage only for ambiguous blobs |
+| Provenance | **Dual anchoring**: extractor citations anchor to CDM spans; every CDM span maps back to native coordinates (PDF page/rect, email message-id + line, markdown heading path). The dashboard highlights the *original* artifact |
+| Metadata | Email headers are native provenance — `From:` + `Date:` yield attribution and temporal edges for free |
+
+Tooling (mostly assembled from the existing portfolio): `scry` for format
+triage, AkamaiForms PDF kernel as the PDF adapter, `whatthediff` for
+cross-format structural unification and near-duplicate detection, Prévue's
+pdf-extract path as text fallback. New work: `.eml`/mbox adapter.
+
+### Tier 1 — Extraction (read CDM → emit candidate signals)
+
+| Expert | Lens | Tools |
+|---|---|---|
+| **Lexicon** | Org verbiage: defined terms, acronyms, canonical names — department-scoped | `corpus_read`, `concordance`, `emit_candidate(Term)` |
+| **Policy** | Normative statements (must/shall/may), effective dates, supersession | `corpus_read`, `emit_candidate(PolicyRule)` |
+| **Structure** | Org chart, roles, systems, ownership | `corpus_read`, `emit_candidate(Entity\|Relation)` |
+| **Process** | Workflows, lifecycles, temporal ordering | `corpus_read`, `emit_candidate(ProcessStep)` |
+
+### Tier 2 — Verification (adversarial; may corroborate/refute, **cannot emit facts**)
+
+| Expert | Lens | Tools |
+|---|---|---|
+| **Grounding Auditor** | Does the cited span *entail* the claim? Runs a **different model family** than the extractor — always | `span_fetch`, `verdict(corroborate\|refute)` |
+| **Contradiction Sentinel** | Candidate deltas vs current rev: `aion_contradictions` + typed-edge constraints (one `reports_to` per person per timeslice; `supersedes` acyclic) | `graph_read`, `aion_contradictions`, `escalate` |
+
+### Tier 3 — Stewardship
+
+| Expert | Lens | Tools |
+|---|---|---|
+| **Ontologist** | Owns the edge vocabulary (rev-0 ontology node). Only role that may propose new edge kinds — via human-approved escalation only | `graph_read`, `ontology_map`, `propose_edge_kind` |
+| **Curator** | Sole AION writer. Batches consensus survivors, runs final gates, signs. **Deliberately not an LLM** — signing authority is code | `aion_annotate`, `aion_link`, `aion_snapshot` |
+
+## 5. Hard rules
+
+1. **Provenance is enforced at the tool layer.** `emit_candidate`
+   schema-requires `{doc_id, span, quote}`. A citation-free candidate is
+   rejected by the tool contract before it becomes a signal.
+2. **Capability-scoped tools per role.** Extractors: corpus read only.
+   Verifiers: graph read, no corpus write. Curator alone writes AION.
+3. **Reputation-weighted amplitude.** Auditor verdicts feed
+   `smesh-core::reputation`; a repeatedly-refuted extractor emits weaker
+   signals.
+4. **Model diversity is a requirement.** Extraction: `moonshotai/kimi-k3`
+   (1M ctx — whole-corpus passes, no chunking). Verification: a different
+   family (Claude, or minimally `kimi-k2-thinking`). Same-model verification
+   lets correlated hallucinations survive.
+5. **Consensus thresholds are per-edge-kind.** `defines_term` with one
+   corroboration may pass; `governed_by` needs ≥2 independent corroborations
+   and zero contradictions.
+
+## 6. Human ratification protocol
+
+The mesh never publishes — it stages. A staged changeset is a PR against the
+world; the dashboard is the review screen.
+
+### Delta lifecycle
+
+```
+Draft ──► Staged ──┬── approve ──► Ratified ──► Signed (rev N+1)
+                   ├── edit ─────► HumanAmended ──► Ratified (human version wins)
+                   ├── reject ───► Killed (+reason → expert reputation hit)
+                   └── defer ────► Parked (decays back into the field)
+```
+
+### Reviewer identity
+
+Each reviewer holds an Ed25519 identity; ratification is a signed act recorded
+in the graph. Every edge answers four questions forever: what it says, where
+it came from (span), which experts corroborated it, **which human approved it
+and when**.
+
+### Provenance classes
+
+- `corpus_derived` — extracted, span-cited, mesh-corroborated.
+- `human_attested` — no document exists; the provenance *is* the human's
+  signature. Highest-trust class. The dashboard is the tribal-knowledge
+  capture instrument: reviewers author facts the corpus never contained.
+
+Human edits to staged candidates are labeled error signals — fed back into
+expert reputation, retained as future tuning data.
+
+### Attention lanes (ratification is total; attention is tiered)
+
+| Lane | Contents | Interaction |
+|---|---|---|
+| **Green** | High-consensus, exact-quote, low-risk (terms, org structure) | Summarized batch, spot-checkable |
+| **Amber** | Normative/policy edges; single corroborator | Individually surfaced with evidence |
+| **Red** | Contested edges, contradictions, ontology changes | Mandatory individual decision |
+
+Dashboard write path is a thin Axum layer over the prevue-api registry
+pattern (browse, drill into cited evidence) plus Solv's cited-cards queue UX.
+
+## 7. Showcase corpus: Meridian Mutual
+
+**Author the gold graph first; render documents from it.** The answer key is
+exact by construction; validation is scored, not vibes.
+
+Meridian Mutual: fictional regional insurance company. Departments: Claims,
+Clinical Policy, Finance, HR, IT/Security, Legal/Compliance. Formats:
+markdown handbooks, PDF policy manuals, `.eml` threads, YAML
+config-as-policy, CSV org roster, wiki export, scanned-memo PDFs.
+
+### Planted-defect manifest
+
+| Planted challenge | Exercises |
+|---|---|
+| "Claim" means different things in Legal / Claims / Finance | Lexicon + department-scoped terms |
+| Two departments with contradictory rules | Contradiction Sentinel → red lane → human |
+| Policy PDF superseded by a later email memo | Cross-format supersession chains |
+| Same policy as PDF and markdown, slightly divergent | Consensus + divergence detection |
+| Rule existing only in an email thread | Tribal-knowledge analog; email provenance |
+| Question the corpus deliberately never answers | **Confabulation trap** — must yield negative space |
+| Attached vendor policy document | Attribution scoping — external ≠ Meridian |
+| Generic-corp boilerplate contradicting Meridian's quirky rule | The founding failure mode |
+
+### Scorecard (also the permanent regression gate)
+
+- **Edge precision / recall** vs gold graph
+- **Contradiction detection rate** (caught / planted)
+- **Confabulation rate** — edges in no document with no human signature. Target: **zero**. Headline number for the thesis
+- **Provenance integrity** — cross-model audit that every `corpus_derived` span entails its claim
+- **Escalation quality** — red-lane items reached the human; green-lane items deserved green
+
+## 8. Scaffold map (`smesh-world/`)
+
+| Module | Contents |
+|---|---|
+| `role.rs` | `WorldRole` taxonomy (Tiers 0–3), capability sets, per-role tool inventories, model assignment |
+| `cdm.rs` | Canonical Document Model: content-addressed docs, blocks, dual-anchor spans |
+| `ontology.rs` | Typed edge vocabulary + per-edge-kind consensus thresholds + structural constraints |
+| `candidate.rs` | Candidate node/edge emissions with schema-mandatory citations |
+| `delta.rs` | Changeset lifecycle state machine; type-state ratification (unsigned changeset cannot reach `Signed`) |
+| `corpus.rs` | Gold graph, planted-defect manifest, scorecard types for Meridian |
+| `intake/` | Tier 0 Registrar: deterministic format sniffing; markdown adapter (heading-path anchors), `.eml` adapter (header harvest, body-line anchors), PDF adapter (pdf-extract per-page → page anchors; textless PDFs loudly malformed), deterministic lopdf renderer |
+| `meridian.rs` | The instrumented corpus: authored gold graph (11 edges), 8-defect manifest, 9 rendered artifacts (5 md / 3 eml / 1 two-page PDF), evidence table binding gold edges to artifact quotes |
+
+## 9. Phasing
+
+- **Phase 0 — DONE:** types + state machine + gates green.
+- **Phase 1 — DONE:** Tier 0 intake adapters → CDM (markdown, `.eml`, PDF). MIME multipart + OCR deferred to the vendor-attachment / scanned-doc work.
+- **Phase 2 — DONE:** Meridian gold graph + deterministic renderer + planted defects; every artifact ingests through the Registrar, every evidence quote grounds a citation, negative space verified truly absent.
+- **Phase 3 — SHIPPED (`smesh-refinery`):** Tier 1/2 experts over `smesh-agent`: 4 extractor lenses on K3, grounding firewall (verbatim quote → `Citation::grounded` or attributed rejection), Grounding Auditor (cross-family entailment), Contradiction Sentinel (deterministic supersession/uniqueness resolution + cross-family semantic pairs), staged changeset + lanes + scorecard. `cargo run -p smesh-refinery` = live Meridian run. Field/reputation feedback into `smesh-core` is Phase 3b, deferred.
+- **Phase 4 — SHIPPED (`smesh-ratify`):** the ratification bench. Ed25519 reviewer identity, crash-safe decision ledger, total-coverage gate, lanes with evidence drill-in, firewall transparency, kernel type-state signing with revision receipt. `cargo run -p smesh-ratify -- --demo` or point it at `refinery-staged.json`. Curator → AION write is Phase 4b.
+- **Phase 5:** Scorecard harness; Meridian becomes the regression gate.

smesh-ratify/.gitignore

@@ -0,0 +1,2 @@
+demo-session.*
+.ratify/

smesh-ratify/Cargo.toml

@@ -0,0 +1,20 @@
+[package]
+name = "smesh-ratify"
+version.workspace = true
+edition.workspace = true
+authors.workspace = true
+license.workspace = true
+repository.workspace = true
+description = "Ratification dashboard: the human merge authority over staged world-model changesets, with Ed25519 sign-off"
+
+[dependencies]
+smesh-world = { path = "../smesh-world" }
+axum = { workspace = true }
+tokio = { workspace = true }
+serde = { workspace = true }
+serde_json = { workspace = true }
+thiserror = { workspace = true }
+tracing = { workspace = true }
+tracing-subscriber = { workspace = true }
+ed25519-dalek = { workspace = true }
+rand = { workspace = true }