Skip to content

Add benchmark contamination graph guard - #609

Open
pine-harbor wants to merge 1 commit into
SCIBASE-AI:mainfrom
pine-harbor:benchmark-contamination-graph-17
Open

Add benchmark contamination graph guard#609
pine-harbor wants to merge 1 commit into
SCIBASE-AI:mainfrom
pine-harbor:benchmark-contamination-graph-17

Conversation

@pine-harbor

Copy link
Copy Markdown

/claim #17

Summary

Adds benchmark-contamination-graph-guard, a focused Scientific Knowledge Graph Integration guard for model-to-benchmark graph edges. It detects when a model evaluation claim may be contaminated because the benchmark, held-out split, label source, or derived data appeared in training/pretraining/fine-tuning data before entity pages or AI recommendations treat the result as clean evidence.

This is intentionally narrow: it does not rebuild graph ingestion, entity extraction, ontology aliasing, geospatial provenance, organism/strain boundaries, chemical identity, software dependency provenance, calibration evidence, negative evidence, funding provenance, or generic recommendation-path explanation.

What It Covers

  • Model graph edges such as trained_on, pretrained_on, fine_tuned_on, derived_from, and label_source.
  • Benchmark claim edges such as evaluated_on, claims_sota_on, recommended_for, and compared_against.
  • Direct training/pretraining overlap with the claimed benchmark.
  • Training/evaluation split hash collisions.
  • Shared training and benchmark label sources.
  • Benchmark release date versus model training cutoff.
  • Independent holdout evidence, split checksums, and locked evaluation-set evidence.
  • Unsafe entity-page and recommendation publication while contamination risk is unresolved.

Included Artifacts

  • Dependency-free Node module: benchmark-contamination-graph-guard/src/index.js
  • Synthetic sample packets: benchmark-contamination-graph-guard/data/sample_benchmark_graph_packets.json
  • Tests: benchmark-contamination-graph-guard/test/benchmark-contamination-graph-guard.test.js
  • Demo/report generator: benchmark-contamination-graph-guard/scripts/demo.js
  • Generated reviewer packet, JSON summary, SVG summary, and MP4 slate in benchmark-contamination-graph-guard/reports/

Validation

  • npm.cmd test: 8 tests passed.
  • npm.cmd run check: syntax checks passed.
  • npm.cmd run demo: generated 4-packet report with PUBLISH: 1, REVIEW: 1, HOLD: 2.
  • npm.cmd run video: generated reports/demo.mp4.
  • git diff --check: passed.
  • ffprobe: H.264, 1280x720, 8.000 seconds, 30 fps, 240 frames.

The MP4 is a generated FFmpeg color-source/drawtext slate, not a desktop or browser recording. It contains no private data.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants