Add benchmark contamination graph guard

benchmark-contamination-graph-guard/README.md

@@ -0,0 +1,47 @@
+# Benchmark Contamination Graph Guard
+
+This contribution adds a focused guardrail for SCIBASE's Scientific Knowledge Graph Integration. It evaluates model-to-benchmark graph edges before entity pages, benchmark badges, or AI recommendations treat an evaluation result as clean independent evidence.
+
+The slice is intentionally narrow. It does not rebuild graph ingestion, entity extraction, ontology aliasing, geospatial provenance, organism or strain boundaries, chemical identity, software dependency provenance, calibration evidence, negative evidence, or recommendation path explanation. It covers the model evaluation leakage question: did the model already see the benchmark, labels, split, or derived data?
+
+## What It Checks
+
+- Model graph edges such as `trained_on`, `pretrained_on`, `fine_tuned_on`, `derived_from`, and `label_source`.
+- Benchmark claim edges such as `evaluated_on`, `claims_sota_on`, `recommended_for`, and `compared_against`.
+- Direct training/pretraining overlap with the claimed benchmark.
+- Training/evaluation split hash collisions.
+- Shared training and benchmark label sources.
+- Benchmark release date versus model training cutoff.
+- Independent holdout evidence, split checksums, and locked evaluation-set evidence.
+- Unsafe entity-page and recommendation publication when contamination risk is unresolved.
+
+## Running Locally
+
+```bash
+npm test
+npm run demo
+npm run check
+npm run video
+```
+
+The demo uses only synthetic graph packets in `data/sample_benchmark_graph_packets.json`.
+
+Generated files:
+
+- `reports/summary.json`
+- `reports/reviewer-packet.md`
+- `reports/summary.svg`
+- `reports/demo.mp4`
+
+The MP4 is generated from an FFmpeg color source and drawtext slate. It is not a screen recording and contains no desktop capture or private information.
+
+## Why This Matters
+
+Knowledge graphs can amplify benchmark claims through entity pages, related-work recommendations, and model/dataset discovery surfaces. If a model was trained on a benchmark, reused its labels, or leaked evaluation examples through derived datasets, the graph should not present that benchmark edge as clean evidence.
+
+This guard gives curators a deterministic review packet so SCIBASE can:
+
+1. Suppress contaminated benchmark badges.
+2. Downgrade recommendation confidence.
+3. Preserve split and label-source evidence.
+4. Require independent holdout proof before publishing model evaluation graph edges.

benchmark-contamination-graph-guard/data/sample_benchmark_graph_packets.json

@@ -0,0 +1,165 @@
+[
+  {
+    "id": "benchmark-publish-001",
+    "title": "Clean independent benchmark edge for microscopy segmentation model",
+    "reviewDate": "2026-08-15",
+    "model": {
+      "id": "model-microseg-v4",
+      "trainingCutoff": "2025-11-01"
+    },
+    "benchmark": {
+      "id": "benchmark-cellmask-2026",
+      "releaseDate": "2026-02-15"
+    },
+    "graphEdges": [
+      {
+        "type": "trained_on",
+        "source": "model-microseg-v4",
+        "target": "dataset-labcells-2025"
+      },
+      {
+        "type": "evaluated_on",
+        "source": "model-microseg-v4",
+        "target": "benchmark-cellmask-2026",
+        "doi": "10.5555/cellmask.2026"
+      }
+    ],
+    "splitEvidence": {
+      "trainingHashes": ["tr-a1", "tr-b2", "tr-c3"],
+      "evaluationHashes": ["ev-x9", "ev-y8", "ev-z7"],
+      "trainingLabelSources": ["labcells-curation-v1"],
+      "benchmarkLabelSources": ["cellmask-independent-v2"],
+      "independentHoldout": true,
+      "splitChecksumPresent": true,
+      "evaluationLockedBeforeTraining": true
+    },
+    "recommendation": {
+      "publishToEntityPage": true
+    },
+    "safeguards": {
+      "freezeRecommendations": false
+    }
+  },
+  {
+    "id": "benchmark-hold-002",
+    "title": "Benchmark reused in model pretraining and public recommendation",
+    "reviewDate": "2026-08-15",
+    "model": {
+      "id": "model-proteinrank-7",
+      "trainingCutoff": "2026-04-01"
+    },
+    "benchmark": {
+      "id": "benchmark-proteinbind-2025",
+      "releaseDate": "2025-10-15"
+    },
+    "graphEdges": [
+      {
+        "type": "pretrained_on",
+        "source": "model-proteinrank-7",
+        "target": "benchmark-proteinbind-2025",
+        "datasetId": "benchmark-proteinbind-2025"
+      },
+      {
+        "type": "claims_sota_on",
+        "source": "model-proteinrank-7",
+        "target": "benchmark-proteinbind-2025",
+        "doi": "10.5555/proteinbind.2025"
+      }
+    ],
+    "splitEvidence": {
+      "trainingHashes": ["pb-001", "pb-002", "pb-003"],
+      "evaluationHashes": ["pb-002", "pb-044"],
+      "trainingLabelSources": ["proteinbind-labels-v1"],
+      "benchmarkLabelSources": ["proteinbind-labels-v1"],
+      "independentHoldout": false,
+      "splitChecksumPresent": true,
+      "evaluationLockedBeforeTraining": false
+    },
+    "recommendation": {
+      "publishToEntityPage": true
+    },
+    "safeguards": {
+      "freezeRecommendations": false
+    }
+  },
+  {
+    "id": "benchmark-review-003",
+    "title": "Older benchmark with missing holdout lock evidence",
+    "reviewDate": "2026-08-15",
+    "model": {
+      "id": "model-solar-forecast-2",
+      "trainingCutoff": "2026-01-01"
+    },
+    "benchmark": {
+      "id": "benchmark-solar-2024",
+      "releaseDate": "2024-07-01"
+    },
+    "graphEdges": [
+      {
+        "type": "trained_on",
+        "source": "model-solar-forecast-2",
+        "target": "dataset-grid-2025"
+      },
+      {
+        "type": "evaluated_on",
+        "source": "model-solar-forecast-2",
+        "target": "benchmark-solar-2024"
+      }
+    ],
+    "splitEvidence": {
+      "trainingHashes": ["grid-1", "grid-2"],
+      "evaluationHashes": ["solar-a", "solar-b"],
+      "trainingLabelSources": ["grid-labels"],
+      "benchmarkLabelSources": ["solar-benchmark-labels"],
+      "independentHoldout": true,
+      "splitChecksumPresent": false,
+      "evaluationLockedBeforeTraining": false
+    },
+    "recommendation": {
+      "publishToEntityPage": false
+    },
+    "safeguards": {
+      "freezeRecommendations": true
+    }
+  },
+  {
+    "id": "benchmark-hold-004",
+    "title": "Label source leakage without direct dataset edge",
+    "reviewDate": "2026-08-15",
+    "model": {
+      "id": "model-neuroreader-5",
+      "trainingCutoff": "2025-08-01"
+    },
+    "benchmark": {
+      "id": "benchmark-neurocases-2025",
+      "releaseDate": "2025-12-01"
+    },
+    "graphEdges": [
+      {
+        "type": "fine_tuned_on",
+        "source": "model-neuroreader-5",
+        "target": "dataset-hospital-notes-2025"
+      },
+      {
+        "type": "recommended_for",
+        "source": "model-neuroreader-5",
+        "target": "benchmark-neurocases-2025"
+      }
+    ],
+    "splitEvidence": {
+      "trainingHashes": ["note-1", "note-2"],
+      "evaluationHashes": ["case-a", "case-b"],
+      "trainingLabelSources": ["neuro-labeler-shared"],
+      "benchmarkLabelSources": ["neuro-labeler-shared", "expert-panel-b"],
+      "independentHoldout": false,
+      "splitChecksumPresent": true,
+      "evaluationLockedBeforeTraining": true
+    },
+    "recommendation": {
+      "publishToEntityPage": true
+    },
+    "safeguards": {
+      "freezeRecommendations": false
+    }
+  }
+]

benchmark-contamination-graph-guard/package.json

@@ -0,0 +1,13 @@
+{
+  "name": "benchmark-contamination-graph-guard",
+  "version": "1.0.0",
+  "description": "Dependency-free knowledge graph guard for model benchmark contamination and leakage edges.",
+  "main": "src/index.js",
+  "scripts": {
+    "test": "node --test",
+    "demo": "node scripts/demo.js",
+    "video": "node scripts/render-demo-video.js",
+    "check": "node --check src/index.js && node --check scripts/demo.js && node --check scripts/render-demo-video.js"
+  },
+  "license": "MIT"
+}

benchmark-contamination-graph-guard/reports/reviewer-packet.md

@@ -0,0 +1,71 @@
+# Benchmark Contamination Graph Guard Report
+
+Generated: 2026-08-15T21:57:22.692Z
+Packets analyzed: 4
+Decision counts: PUBLISH 1, REVIEW 1, HOLD 2
+
+## Findings
+
+### benchmark-publish-001: Clean independent benchmark edge for microscopy segmentation model
+
+Decision: PUBLISH
+Risk score: 0
+Benchmark claim edges: 1
+Contamination edges: 1
+
+- No benchmark contamination risk detected.
+
+Recommended actions:
+- Publish benchmark graph edge and recommendation with normal provenance metadata.
+
+### benchmark-hold-002: Benchmark reused in model pretraining and public recommendation
+
+Decision: HOLD
+Risk score: 100
+Benchmark claim edges: 1
+Contamination edges: 1
+
+- CRITICAL DIRECT_TRAINING_BENCHMARK_OVERLAP: The model graph links training or pretraining data directly to the claimed benchmark.
+- CRITICAL TRAIN_EVAL_SPLIT_HASH_COLLISION: Training and evaluation split hashes overlap.
+- CRITICAL LABEL_SOURCE_LEAKAGE: Training labels and benchmark labels share a source that can leak answers into evaluation.
+- MAJOR BENCHMARK_PREDATES_TRAINING_CUTOFF: The benchmark was released before the model training cutoff, increasing contamination risk.
+- MAJOR INDEPENDENT_HOLDOUT_EVIDENCE_MISSING: The packet lacks complete independent holdout evidence for the benchmark claim.
+- CRITICAL UNSAFE_RECOMMENDATION_PATH_NOT_FROZEN: Entity-page or recommendation publication is enabled while contamination risk is unresolved.
+
+Recommended actions:
+- Freeze model recommendation paths and entity-page benchmark badges.
+- Attach contamination findings to the graph curation queue.
+- Downgrade or suppress benchmark-confidence badges until evidence is resolved.
+- Generate a reviewer packet with split, label-source, and edge-provenance evidence.
+
+### benchmark-review-003: Older benchmark with missing holdout lock evidence
+
+Decision: REVIEW
+Risk score: 44
+Benchmark claim edges: 1
+Contamination edges: 1
+
+- MAJOR BENCHMARK_PREDATES_TRAINING_CUTOFF: The benchmark was released before the model training cutoff, increasing contamination risk.
+- MAJOR INDEPENDENT_HOLDOUT_EVIDENCE_MISSING: The packet lacks complete independent holdout evidence for the benchmark claim.
+
+Recommended actions:
+- Attach contamination findings to the graph curation queue.
+- Downgrade or suppress benchmark-confidence badges until evidence is resolved.
+- Generate a reviewer packet with split, label-source, and edge-provenance evidence.
+
+### benchmark-hold-004: Label source leakage without direct dataset edge
+
+Decision: HOLD
+Risk score: 92
+Benchmark claim edges: 1
+Contamination edges: 1
+
+- CRITICAL LABEL_SOURCE_LEAKAGE: Training labels and benchmark labels share a source that can leak answers into evaluation.
+- MAJOR INDEPENDENT_HOLDOUT_EVIDENCE_MISSING: The packet lacks complete independent holdout evidence for the benchmark claim.
+- CRITICAL UNSAFE_RECOMMENDATION_PATH_NOT_FROZEN: Entity-page or recommendation publication is enabled while contamination risk is unresolved.
+
+Recommended actions:
+- Freeze model recommendation paths and entity-page benchmark badges.
+- Attach contamination findings to the graph curation queue.
+- Downgrade or suppress benchmark-confidence badges until evidence is resolved.
+- Generate a reviewer packet with split, label-source, and edge-provenance evidence.