fix(data-repo): runlog index builder reads .jsonl.gz run logs - #193
Merged
Conversation
…or_v2 commit path)
scripts/build-runlog-index.py skipped *.jsonl.gz locally and filtered
.endswith('.jsonl') in --github mode, so once Arena Studio v0.72 commits
gzipped logs the per-folder index.json would stop gaining entries and the
dashboard catalog's Start/Duration would read '—' for every new run.
gzip is not seekable, so a .gz is read whole (a 1 h run is ~4 MB compressed)
and inflated; head/tail parsing is unchanged. 'file' keeps the .gz name, 'size'
is the committed (compressed) size — what the catalog shows. A truncated .gz
still yields metadata + start and never raises.
Verified: index.json byte-identical for all 149 v1 logs in the local course
clone (old vs new script); 9 gzipped twins of real logs index identically to
their originals apart from file/size. New tests/test-build-runlog-index.py
(17 checks) added to the pixi test task.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
mbreiser
added a commit
that referenced
this pull request
Sep 7, 2026
…rebuild green Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
mbreiser
added a commit
that referenced
this pull request
Sep 9, 2026
…-line collision note Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
mbreiser
added a commit
that referenced
this pull request
Sep 10, 2026
…rebuild green Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
mbreiser
added a commit
that referenced
this pull request
Sep 10, 2026
…-line collision note Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
mbreiser
added a commit
that referenced
this pull request
Sep 10, 2026
… lab-PC handover + 2026-09-10 bench results (#192) * docs: controller telemetry ring buffer — feasibility, design, T1–T6 + L1–L4 bench plan (no code) Proposal for a controller-side RAM ring buffer (frame/command/state events, analog ticks yoked to frame index) drained live over USB with an ack-cursor, replacing the SD-card event-log shape of LAB-149. Answers the two-file vs integrated question (sidecar first), timestamps/clock-fit, and lays out the glass-to-glass latency measurement plan for FicTrac closed loop. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(telemetry proposal): use the 0xA8/0xA9 opcodes reserved by firmware PR #47 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs: merge + bench plan for 2026-09-08/09 (run-log stack, analog F1/F2 + S1/S2, cleanup) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(merge plan): #186/#188 rebased; flag gzip-blind runlog index builder Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(merge plan): #189 merged; gzip index-builder fix built + tested (PR pending) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(merge plan): #193 merged + installed in the course repo; Action rebuild green Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(merge plan): corpus gates 174/174 on rebased #188; index.json filter fix Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs: Codex cross-review of the pending merge-plan work (2026-09-09) — fixed bugs, discussion items, open questions Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(merge plan): stacks rebased onto main after #193; pixi.toml test-line collision note Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(codex review + merge plan): second pass — discussion items 4/6/7/8/9/10 done Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs: lab-PC handover — install list, branch state, gate B1 step-by-step, minimal analog, firmware, gotchas Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(lab-pc handover): optional bench launcher — FicTrac / configGui / bridge / Studio buttons with status Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs: 2026-09-10 bench day — gates B1/B2 passed, 12-18 arena flashed (fw #48), first reworked-board analog results Merge plan: B1 rows ticked with the rig05-mr run (#188 merged f744e12, #186 closed), B2 ticked (#190 merged 880ffd5, #191 rebased --onto and held), 12-18 bring-up + loopback numbers. Handover: status table + Windows gotchas (Git Bash TZ prints UTC, base-retarget runs no CI, rebase --onto after a squashed parent, PYTHONUTF8, teensy upload port/power). Analog plan § 5.1: T1/T6 results — AI1 tolerance-class, AI2 stage-2 divider ≈ 0.26 (inspect R179/R181). CLAUDE.md: Windows-safe ET timestamp recipe. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #187, prerequisite for shipping #186 (Studio v0.72 commits run logs as
.jsonl.gz).Problem
scripts/build-runlog-index.pyskipped*.jsonl.gzin local mode (# gz handled once behavior_v2 lands) and filtered.endswith('.jsonl')in--githubmode. After #186 every new run is gzipped, sorunlogs/<folder>/index.jsonwould stop gaining entries and the dashboard catalog's Start / Duration would read "—" for every new run.Change
.jsonland.jsonl.gzin both modes (is_runlog_name)..gzis read whole (--github: one raw download; a 1 h behavior_v2 run is ~4 MB compressed) and inflated; the existing 64 KB head / 4 KB tail parsing then applies unchanged. Plain.jsonlfiles still use the two Range reads — nothing changes for them.filekeeps the on-disk name (x.jsonl.gz) so the dashboard's per-file lookup matches;sizeis the committed (compressed) size, which is what the catalog shows..gz(half-written upload) still yieldsrun_metadata+ start via a streaming decompressor and never raises; end state staysnull..gzwhole, small).Verification
.jsonlin 10 folders) → everyindex.jsonbyte-identical.file/size(gzip ratios 2.8–7.3×).tests/test-build-runlog-index.py(17 checks: plain vs gz twin, truncated gz, aborted run fallback, local CLI on a mixed folder, name filter) added to thepixi run testtask.Deploy
The data repo runs a copy: after merge, re-install into
reiserlab/cshl-2026-course/.github/scripts/build-runlog-index.py(byte-identical) — same as #187's install step. Until then the Action ignores.gzfiles (no harm, just missing rows).🤖 Generated with Claude Code