feat(data-repo): runlog-index GitHub Action keeps runlogs/<folder>/index.json current - #187
Merged
Conversation
…<folder>/index.json Option 1 from the review: the data repo itself keeps the catalog index current. scripts/data-repo-workflows/runlog-index.yml (template installed into each data repo) runs on every push under runlogs/ (ignoring its own index.json commits), serializes concurrent runs, sparse-checks-out only .github, computes the touched folders from the push payload (added/modified/removed), and rebuilds just those via build-runlog-index.py --github with GITHUB_TOKEN. workflow_dispatch rebuilds all folders. Arena Studio does NOT write the index (option 2 dropped: one writer, no commit-path change on running rigs, covers hand pushes/migrations/deletions). build-runlog-index.py: --folder is repeatable; index PUT retries on a stale-sha 409/422. Comments/README updated to name the Action as the writer. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ange reads; skip unchanged index.json
First live run rebuilt ALL folders (push-payload parse found no paths) and hit
a transient connection reset on one of ~340 range reads. Detection now uses
GET /compare/{before}...{sha} (paginated); raw range reads retry 4x; an index
that is byte-identical is not re-committed (no no-op bot commits).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Contributor
Author
Live verification on
|
| event | run | detection | rebuilt | bot commits |
|---|---|---|---|---|
| add throwaway | success | changed files: 1 → folders: bench00 |
bench00 only (4 runs) | 1 |
| delete throwaway | success | changed files: 1 → folders: bench00 |
bench00 only (3 runs) | 1 |
- Index after add contained
idxtest3withduration_s123.456 /complete: true; after delete it is back to the 3 real bench00 runs. No re-trigger from the bot's own index commits (loop guard works). - Earlier iteration (before the compare-API change) fell back to rebuilding all 12 folders and hit one transient
ConnectionResetErroron ~340 range reads — hence the read retries and the unchanged-index skip in this PR. - Follow-ups on the feat(studio): data-repo registry — lab repo picker, "Rig id", no default repo (v0.72) #178 branch (files live only there):
scripts/seed-data-repo.shinstalls both files into new data repos;docs/development/data-repo-setup.mdgets the index section. Arena Studio does not write the index.
mbreiser
added a commit
that referenced
this pull request
Sep 7, 2026
…ex.json isRunlogName accepted bare .json, and #188 applies it to directory listings (data-repo browse, URL index) and the corpus parity script. Since #187 every runlogs/<folder>/ carries an index.json catalog, so the dashboard would list each folder's index as a run and the corpus gate failed on all 12 of them (174/174 logs themselves pass). Narrow the shared filter to .jsonl|.ndjson (+ .gz); the drop handler keeps accepting a user-dropped .json to match the file input's accept list. Vendored copy synced; test updated. Corpus gates on this branch (course clone @ 37d003e, 174 logs): scripts/runlog-v2-corpus.py 174/174 ok dashboard/data-browser/tests/corpus-v2-parity.js 174/174 ok (was 174 ok + 12 index.json FAIL) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
mbreiser
added a commit
that referenced
this pull request
Sep 9, 2026
…ex.json isRunlogName accepted bare .json, and #188 applies it to directory listings (data-repo browse, URL index) and the corpus parity script. Since #187 every runlogs/<folder>/ carries an index.json catalog, so the dashboard would list each folder's index as a run and the corpus gate failed on all 12 of them (174/174 logs themselves pass). Narrow the shared filter to .jsonl|.ndjson (+ .gz); the drop handler keeps accepting a user-dropped .json to match the file input's accept list. Vendored copy synced; test updated. Corpus gates on this branch (course clone @ 37d003e, 174 logs): scripts/runlog-v2-corpus.py 174/174 ok dashboard/data-browser/tests/corpus-v2-parity.js 174/174 ok (was 174 ok + 12 index.json FAIL) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
mbreiser
added a commit
that referenced
this pull request
Sep 10, 2026
…v0.73 readers (#186, #188) (#188) * feat(studio): gzip run-log commits (.jsonl.gz), behavior_v2 default, bridge log-level ack (PR 2 of runlog-behavior-v2-plan) — Studio v0.72 Studio side of docs/development/runlog-behavior-v2-plan.md (Part 1 §5 + Part 2). Stacks on PR #183 (bridge). Readers (dashboard / replay viewer) are PR 3 — until it lands, .jsonl.gz files need gunzip or `bridge.py --convert`. - js/fictrac-bridge-client.js: LOG_LEVELS [behavior_v2, behavior_v1, full] (v2 default); handles hello_ack (bridgeInfo, bridgeSupportsLevel) and log_control_ack (ackedLogLevel, cleared on every setLogging and on close); new 'loglevel' event + an err log line when the bridge cannot write the requested level; waitForLogLevelAck(ms) → acked level or null (old bridge). - js/studio-github.js: gzipBytes (CompressionStream) + isGzip; Git Database builders (reqCreateBlob/reqGetCommit/reqCreateTree/reqCreateCommit/ reqUpdateRef, allowlisted, fast-forward only); directCommitLarge (7-call blob→tree→commit→ref sequence, per-step errors); commitFile routes by size (LARGE_FILE_BYTES = 30 MiB) and reports via + bytes. - arena_studio.html v0.72: File ▾ → Run logging offers behavior_v2 (default) / behavior_v1 (legacy) / full; Console mirror follows the bridge's ack (⚠ on mismatch); run start awaits the ack ≤ 800 ms, names the level in the banner + transcript and warns "bridge too old for behavior_v2 — logging behavior_v1" (or "did not confirm" for a pre-3.0 bridge); run_metadata gains log_format; commitRunLog gzips the export → runlogs/<bench>/<name>.jsonl.gz via GH.commitFile (raw .jsonl fallback without CompressionStream) and shows raw → gz size + the path used. - Tests: test-fictrac-bridge-client.js 42 → 71 checks (levels, hello_ack, log_control_ack, mismatch, waiters, close reset); test-studio-github.js 118 → 160 (gzip round trip via zlib, git-db builders + token containment, directCommitLarge sequence + failure step, commitFile routing, Studio wiring assertions on the HTML). - Docs: release notes v0.72, cshl-pipeline-test-plan size-limits, CLAUDE.md Studio rule, plan doc status + PR 2 notes (merge with/after PR 3). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * feat(readers): open .jsonl.gz + behavior_v2 run logs everywhere — shared js/runlog-format.js (PR 3 of runlog-behavior-v2-plan) — Studio v0.73 Reader side of docs/development/runlog-behavior-v2-plan.md (Part 3). Stacks on the Studio PR (v0.72). With this, everything that opens a run-log FILE inflates gzip on the magic bytes and expands behavior_v2's compact arena echoes back to the v1 `arena_command` object at parse time, so every existing consumer (p3LedEpochs, stall forensics, replay frame decoding) is unchanged. - js/runlog-format.js (new; classic dual-export, vendored byte-identical at dashboard/data-browser/vendor/): isGzip, isRunlogName, stripGz, inflateIfGzip, readRunlogText (string|bytes|ArrayBuffer|Blob → text), readRunlogPrefixText (truncation-tolerant gunzip for the catalog's 64 KB metadata reads), isArenaArray, expandV2Line (exact v1 key order; timeout ⇒ status/echo/ok all null), compactV1Line, detectFormat, createNormalizer() (per-file state machine), convertV1ToV2Text / convertV2ToV1Text (JS mirror of the bridge converter, for tests + parity). - Dashboard: analysis-core.js parseJsonl normalizes every line, run.logFormat + run.rawBytes, parseFilename strips .gz; github-repo.js fetchRaw reads bytes and inflates (prefix mode inflates a truncated gz); app.js loaders (repo / URL / local server / dropped file) go through readRunlogText, catalog filters accept .jsonl.gz, new size column (compressed size; inflated size + format in the hover), ?v= cache busters bumped, file input accepts .gz. - Replay: js/runlog-replay.js parseRunLog normalizes records (loads js/runlog-format.js first in arena_studio.html; degrades to "v2 echoes skipped" + console warning under a stale cache); js/arena-studio-alt.js replay picker lists .jsonl.gz and inflates the picked file. arena_replay_viewer.html (protocol-fed) and js/studio-runlog-adapter.js (no file reads) unchanged. - Tests: tests/test-runlog-format.js (new, 86 checks, in pixi run test — also enforces the vendored copy); tests/test-runlog-replay.js v2 case identical to the equivalent v1 (76 checks); dashboard test-analysis.js re-reads the P3 fixture as v2 + gz asserting identical frames, arena_command objects, preference indices, LED epochs and page CSV rows, plus metadata from a gz prefix; dashboard/data-browser/tests/corpus-v2-parity.js runs the same over a whole clone (164/164 on cshl-2026-course origin/main). - Browser-verified: the dashboard opens a converted .jsonl.gz via ?data= (52,369 frames / 29 steps, size column "1.6 MB gz", no console errors). - Docs: dashboard README, release notes v0.73, plan doc status + PR 3 notes, CLAUDE.md reader rule. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(readers): run-log name filter must not match runlogs/<folder>/index.json isRunlogName accepted bare .json, and #188 applies it to directory listings (data-repo browse, URL index) and the corpus parity script. Since #187 every runlogs/<folder>/ carries an index.json catalog, so the dashboard would list each folder's index as a run and the corpus gate failed on all 12 of them (174/174 logs themselves pass). Narrow the shared filter to .jsonl|.ndjson (+ .gz); the drop handler keeps accepting a user-dropped .json to match the file input's accept list. Vendored copy synced; test updated. Corpus gates on this branch (course clone @ 37d003e, 174 logs): scripts/runlog-v2-corpus.py 174/174 ok dashboard/data-browser/tests/corpus-v2-parity.js 174/174 ok (was 174 ok + 12 index.json FAIL) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(readers): restore strict mode in runlog-replay.js; mark rawBytes hover approximate Codex cross-review 2026-09-09 (all three reviewers): the new `const Fmt` / `let warnedNoFmt` declarations displaced 'use strict' so it became a parenthesised expression after executable code — no longer a directive, the replay IIFE silently ran in sloppy mode. Directive first again. rawBytes is text.length (UTF-16 code units), not bytes; the catalog hover now says ≈ and the field is annotated, rather than re-encoding a 50 MB string. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Option 1 from today's review: the data repo keeps its own catalog index. A workflow template,
scripts/data-repo-workflows/runlog-index.yml, runs on every push underrunlogs/, rebuilds only the touched folders'index.jsonwithscripts/build-runlog-index.py --github(64 KB head + 4 KB tail per file, never a full log), and commits them withGITHUB_TOKEN.pathsfilter excludesrunlogs/**/index.json, so the bot's own commits don't retrigger.workflow_dispatchrebuilds every folder by hand..githubonly — the workflow never downloads the multi-GB logs.--folderis now repeatable.Arena Studio does not write the index (option 2 dropped): one writer, no change to the run-log commit path while experiments are running, and hand-pushed large logs, migrations and deletions are all covered because each folder is rebuilt from what is present.
Installed into
reiserlab/cshl-2026-coursetoday (.github/workflows/runlog-index.yml+.github/scripts/build-runlog-index.py) and verified with a throwaway push (see PR comment). The seed-script install step and the setup-doc section go into #178 (those files live only on that branch).🤖 Generated with Claude Code