Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions PR_BODY_COMPUTE_RERUN_FEASIBILITY_16.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
/claim #16

## Summary

Adds `research-compute-rerun-feasibility-guard`, a standalone local package for the AI-Powered Research Assistant Suite.

The guard prevents AI assistant reproducibility language from being released until the project has enough compute-resource evidence for a practical reviewer rerun. It checks hardware specs, RAM/VRAM feasibility, runtime budget, checkpoint/resume evidence, container/lockfile/runtime digests, accelerator determinism, large data-transfer fixture availability, paid cloud/HPC disclosure, and long-run resource notes.

## Demo Video

`research-compute-rerun-feasibility-guard/reports/demo.mp4`

The MP4 is a synthetic FFmpeg-rendered slate. It does not capture a desktop, private data, credentials, real manuscripts, or paid services.

## Validation

From `research-compute-rerun-feasibility-guard`:

- `npm run check`
- `npm test`
- `npm run demo`
- `npm run video`

Repository-level:

- `git diff --check`

## Notes

- No external APIs.
- No paid cloud resources.
- No credentials or private data.
- Synthetic fixtures only.
- Scope is distinct from existing #16 slices around general dependency locks, evidence grounding, external validity, image integrity, statistical/model assumptions, data fabrication, and generic reproducibility attempts. This package specifically gates AI assistant output on practical compute rerun feasibility and resource disclosure.
47 changes: 47 additions & 0 deletions research-compute-rerun-feasibility-guard/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# Research Compute Rerun Feasibility Guard

This package adds a local release gate for the SCIBASE AI-Powered Research Assistant Suite. It checks whether an AI assistant can safely describe a manuscript or project as practically reproducible before the assistant output reaches authors, reviewers, journals, or funders.

The guard focuses on compute feasibility rather than general dependency locking or claim grounding. It asks whether a reviewer can actually rerun the work with the stated resources, time budget, determinism controls, and cost disclosure.

## What It Checks

- Hardware profile completeness: CPU, RAM, accelerator, and VRAM.
- Dataset working set vs RAM and GPU memory.
- Runtime estimate vs reviewer budget.
- Long-run checkpoint/resume evidence.
- Container, lockfile, runtime, or notebook environment digests.
- Accelerator determinism controls and seed policy.
- Large data transfer and fixture availability.
- Paid cloud or queued HPC requirements and disclosure.
- Resource-impact note for long reruns.

## Release Decisions

- `RELEASE_ASSISTANT_OUTPUT`: the assistant can present the rerun as reviewer-feasible.
- `REVISE_ASSISTANT_OUTPUT`: wording must be softened until missing resource evidence is repaired.
- `HOLD_ASSISTANT_OUTPUT`: the assistant must not endorse reproducibility because the compute plan is materially unsafe.

## Local Demo

```bash
npm run check
npm test
npm run demo
npm run video
```

Generated artifacts:

- `reports/summary.json`
- `reports/reviewer-packet.md`
- `reports/summary.svg`
- `reports/demo.mp4`

All fixtures are synthetic. The package uses no external APIs, paid cloud, credentials, private data, real manuscripts, or desktop capture.

## Why This Fits Issue #16

Issue #16 includes a reproducibility checker as a core AI research assistant capability. A research assistant can mislead reviewers if it says a project is reproducible while the rerun secretly requires unavailable GPUs, huge memory, long queues, paid cloud resources, or nondeterministic accelerator settings.

This guard adds a practical compute/resource release gate so the assistant can distinguish a reviewer-feasible rerun from an overclaimed reproducibility endorsement.
Original file line number Diff line number Diff line change
@@ -0,0 +1,143 @@
[
{
"id": "paper-cpu-fixture-pass",
"title": "Open fixture rerun for a tabular treatment-effect study",
"assistantOutput": "pre-submission reproducibility checker",
"dataset": {
"sizeGb": 1.2,
"downloadGb": 1.2,
"workingSetMultiplier": 1.8,
"syntheticFixtureAvailable": true,
"requiresGpuResidency": false
},
"compute": {
"cpuCores": 8,
"ramGb": 32,
"accelerator": "cpu",
"gpuCount": 0,
"gpuVramGb": 0,
"estimatedRuntimeHours": 1.5,
"reviewerBudgetHours": 4,
"paidCloudRequired": false,
"hpcQueueRequired": false
},
"environment": {
"containerDigest": "sha256:fixture-pass-container",
"lockfileDigest": "sha256:fixture-pass-lock"
},
"reproducibility": {
"seedPolicy": "fixed seed family with three rerun seeds",
"determinismControls": ["seed manifest", "variance tolerance report"],
"checkpointResumeEvidence": "not required for short CPU rerun"
},
"disclosure": {
"estimatedCostUsd": 0,
"carbonOrResourceNote": "CPU fixture rerun completes locally under two hours."
}
},
{
"id": "paper-gpu-overclaim-hold",
"title": "Large multimodal training rerun marked reproducible without resources",
"assistantOutput": "AI reviewer reproducibility endorsement",
"dataset": {
"sizeGb": 480,
"downloadGb": 620,
"workingSetMultiplier": 2.2,
"syntheticFixtureAvailable": false,
"requiresGpuResidency": true
},
"compute": {
"cpuCores": 16,
"ramGb": 64,
"accelerator": "cuda",
"gpuCount": 1,
"gpuVramGb": 24,
"estimatedRuntimeHours": 38,
"reviewerBudgetHours": 8,
"paidCloudRequired": true,
"hpcQueueRequired": false
},
"environment": {},
"reproducibility": {
"seedPolicy": "",
"determinismControls": [],
"checkpointResumeEvidence": ""
},
"disclosure": {
"estimatedCostUsd": 0,
"carbonOrResourceNote": ""
}
},
{
"id": "paper-hpc-revise",
"title": "Queued HPC simulation with enough hardware but weak disclosure",
"assistantOutput": "journal internal reviewer copilot",
"dataset": {
"sizeGb": 32,
"downloadGb": 40,
"workingSetMultiplier": 1.5,
"syntheticFixtureAvailable": true,
"requiresGpuResidency": false
},
"compute": {
"cpuCores": 48,
"ramGb": 256,
"accelerator": "cpu",
"gpuCount": 0,
"gpuVramGb": 0,
"estimatedRuntimeHours": 12,
"reviewerBudgetHours": 8,
"paidCloudRequired": false,
"hpcQueueRequired": true
},
"environment": {
"runtimeDigest": "sha256:hpc-module-stack"
},
"reproducibility": {
"seedPolicy": "fixed Monte Carlo seed manifest",
"determinismControls": ["seed manifest", "variance tolerance"],
"checkpointResumeEvidence": "checkpoint manifest for every 90 simulated minutes"
},
"disclosure": {
"estimatedCostUsd": 0,
"queuePolicy": "institutional queue, expected wait not stated",
"carbonOrResourceNote": ""
}
},
{
"id": "paper-vram-fixture-needed",
"title": "Single-cell embedding rerun with GPU memory shortfall",
"assistantOutput": "research-gap assistant reproducibility sidebar",
"dataset": {
"sizeGb": 78,
"downloadGb": 92,
"workingSetMultiplier": 1.4,
"syntheticFixtureAvailable": false,
"requiresGpuResidency": true
},
"compute": {
"cpuCores": 24,
"ramGb": 192,
"accelerator": "cuda",
"gpuCount": 1,
"gpuVramGb": 40,
"estimatedRuntimeHours": 7,
"reviewerBudgetHours": 6,
"paidCloudRequired": true,
"hpcQueueRequired": true
},
"environment": {
"containerDigest": "sha256:single-cell-container"
},
"reproducibility": {
"seedPolicy": "seeded embedding initialization",
"determinismControls": ["seed manifest", "cuda deterministic kernels"],
"checkpointResumeEvidence": ""
},
"disclosure": {
"estimatedCostUsd": 48,
"queuePolicy": "shared GPU queue",
"carbonOrResourceNote": ""
}
}
]
21 changes: 21 additions & 0 deletions research-compute-rerun-feasibility-guard/package.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
{
"name": "research-compute-rerun-feasibility-guard",
"version": "1.0.0",
"description": "Local guard that checks whether AI research assistant reproducibility claims are compute-feasible before reviewer release.",
"type": "module",
"main": "src/index.js",
"scripts": {
"check": "node --check src/index.js && node --check scripts/demo.js && node --check scripts/render-demo-video.js && node --check test/compute-rerun-feasibility.test.js",
"test": "node --test",
"demo": "node scripts/demo.js",
"video": "node scripts/render-demo-video.js"
},
"keywords": [
"research-assistant",
"reproducibility",
"compute-feasibility",
"peer-review",
"scibase"
],
"license": "MIT"
}
Binary file not shown.
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# Research Compute Rerun Feasibility Guard

Generated: 2026-08-16T19:40:17.361Z

This packet is a local synthetic demonstration for SCIBASE issue #16. It checks whether an AI-powered research assistant should release reproducibility language before the compute plan is practically rerunnable by a reviewer.

| Packet | Release gate | Score | Working set | Top findings |
| --- | --- | ---: | ---: | --- |
| paper-cpu-fixture-pass | RELEASE_ASSISTANT_OUTPUT | 100 | 2.16 GB | None |
| paper-gpu-overclaim-hold | HOLD_ASSISTANT_OUTPUT | 0 | 1056 GB | CONTAINER_OR_LOCKFILE_MISSING (high); DATASET_MEMORY_EXCEEDS_NODE (critical); GPU_VRAM_FEASIBILITY_GAP (high) |
| paper-hpc-revise | REVISE_ASSISTANT_OUTPUT | 40 | 48 GB | RUNTIME_BUDGET_UNREALISTIC (medium); CLOUD_OR_HPC_COST_UNDISCLOSED (high); RESOURCE_IMPACT_NOTE_MISSING (low) |
| paper-vram-fixture-needed | HOLD_ASSISTANT_OUTPUT | 0 | 109.2 GB | GPU_VRAM_FEASIBILITY_GAP (high); RUNTIME_BUDGET_UNREALISTIC (medium); CHECKPOINT_RESUME_MISSING (high) |

## Release Policy

- `RELEASE_ASSISTANT_OUTPUT`: the assistant can present the rerun as reviewer-feasible.
- `REVISE_ASSISTANT_OUTPUT`: assistant wording must be softened until missing resource evidence is repaired.
- `HOLD_ASSISTANT_OUTPUT`: the assistant must not endorse reproducibility because compute feasibility is materially unsafe.

No external API, private data, paid cloud, real manuscript, or user desktop capture is used.
Loading