Skip to content

Docs: the storage position, the URL rule, and the refs that drifted - #59

Merged
mmcky merged 2 commits into
mainfrom
docs/storage-position-and-url-rule
Aug 10, 2026
Merged

Docs: the storage position, the URL rule, and the refs that drifted#59
mmcky merged 2 commits into
mainfrom
docs/storage-position-and-url-rule

Conversation

@mmcky

@mmcky mmcky commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Maintenance pass after #55, #56 and #57. Every correction here is verified against main today; the strict audit exits 0 and the consumed-file check reports 18/18.

The thing worth reviewing first

#57 deleted the "storage choice invisible to consumers" line from PLAN.md, but the same position survived in two more places — and in one of them it was actively instructing authors to do the opposite of what CI now enforces.

README.md told readers the interim form "works for both plain-git and LFS-tracked files", and to avoid raw.githubusercontent.com. That is the only form lecture-wasm can fetch, and the form the strict audit requires there. So the consumer-facing README was sending a repointing author toward the form the audit rejects and away from the one it demands.

AGENTS.md's URL section carried the same "works regardless of storage" claim and never mentioned repoint rule 5 or CORS at all — so an agent working from the conventions doc alone would write the CORS-broken form into a wasm repoint. That is precisely the incident data-lectures#46 records, and #57 rewrote a different part of this file while leaving that section untouched.

Both are replaced with the runtime-keyed rule, stated once and consistently: CPython consumers get github.com/…/raw/, browser consumers get raw.githubusercontent.com, and the media host is never valid for this repo.

New: the ladder above 100 MiB

AGENTS.md gains the decision rule the repo had no answer for — subset → plain git → gzip in place → split → publish from outside git via the Pages artifact, with LFS-under-lectures/ and release assets as hard nevers. Reasoning and measurements are in #58. The gzip rung matters because it is the only lossless escape: "derive a smaller extract" conflicts with repoint rule 4.

PLAN.md corrections

Line Was Now
rules preamble "CI covers only a corner of them — the github.com/*/raw/ form in lecture-wasm and nothing else" rules 5, 6 and resolvability all checked since #55; still blind to lecture-intro.zh-cn and lectures/_static/**
URL table, browser row "…or media.githubusercontent.com/media/… for LFS-tracked files" the last surviving "media host is fine" line in the repo — #55 made it a hard failure
Phase 3 .gitattributes box unticked ticked, #57
Phase 3 lfs: false box unticked, refs :43 / :22 ticked, refs :51 / :26, and the quota restated as measured and not binding ($0.04 net for 2026) while keeping the mechanism (anonymous downloads bill the owner; a $0 budget blocks downloads)
Phase 4 Pages box "lfs: true at checkout (else pointer files publish)" inverted by #57 — a mis-tracked file must publish as its pointer so the mistake is visible
consumed-file check "non-empty consumers list" rekeyed on integrity.sha256 by #56
generating_mini.md "Also uncomment its two to_csv writes" do not, until #14 settles runnable-builder vs provenance — see below
DNS "an old NestJS box on AWS Sydney currently answers this name" NXDOMAIN at quantecon.org's own authoritative nameserver, Pages cname null
Track Y blocker "an external infra answer on 52.64.86.66" nothing external — the record is ours to create
build_audit.py:45-46 / :45-57 off by one after #55 :46-47 / :46-57
wasm mirror-orphans "a dozen" five, matching Track X's own table

Two things that are more than housekeeping

PLAN.md was instructing one branch of an open decision. The Phase 3 checklist said to uncomment generating_mini.md's two to_csv writes — i.e. to ship it as a runnable builder — while #14's own comment argues for provenance-not-runnable-builder and has no reply. Editing an artifact you are recording as provenance destroys what makes it provenance. Now reworded to hold until #14 settles it. The input-URL repoint stands either way, because an archived high_dim_data keeps serving that URL indefinitely.

Track Y has a blocker nobody has written down. classify_url recognises six GitHub-host regexes and neither quantecon.github.io nor data.quantecon.org. So a consumer on the canonical host classifies as external-web, migrated goes false, and migration.yml's final — defined as "every consumer reads data.quantecon.org" — is a status the audit is structurally guaranteed to report as broken. Both of #55's assertions also lapse at that moment, since they are gated on pattern == "data-lectures". Recorded in the Track Y section; it needs to be fixed before the cutover sweep, not after.

Also

scripts/audit_annotations.yml described graph.txt as a "15-node weighted digraph". It is 100 nodes (short_path.md:385 sets num_nodes = 100, and the %%file block emits 100 lines). That string renders into the public audit dashboard.

🤖 Generated with Claude Code

#57 deleted the "storage choice invisible to consumers" line from PLAN, but
the same position survived in two more places, and one of them was telling
authors to do the opposite of what CI enforces.

README.md said the interim URL form "works for both plain-git and
LFS-tracked files" and told readers to AVOID raw.githubusercontent.com —
which is the only form lecture-wasm can fetch, and the form the strict
audit demands there. Replaced with the runtime-keyed rule: CPython gets
the github.com/*/raw/ form, browser consumers get raw.githubusercontent,
and the media host is never valid for this repo.

AGENTS.md's URL section had the same "works regardless of storage" claim
and never mentioned repoint rule 5 or CORS at all — so an agent working
from the conventions doc alone would write the CORS-broken form into a
wasm repoint, which is exactly the incident data-lectures#46 records.

Also adds the ladder for a published file approaching 100 MiB, which the
repo had no answer for: subset, plain git, gzip in place, split, publish
from outside git. Reasoning and measurements in #58.

PLAN corrections, all verified against main today:

  * rule coverage — CI now checks rules 5, 6 and resolvability, not "only
    a corner"
  * the browser row of the URL table still offered the media host for LFS
    files; #55 made that a hard failure
  * the .gitattributes and lfs:false boxes tick, with the line refs moved
    (audit-dashboard.yml:51, consumed-file-check.yml:26) and the LFS
    quota restated as measured-and-not-binding, keeping the mechanism
  * the consumed-file check is rekeyed off consumers (#56)
  * generating_mini.md: do NOT uncomment its to_csv writes until #14
    settles runnable-builder vs provenance — PLAN was instructing one
    branch of an open decision. The input-URL repoint stands either way
  * data.quantecon.org is NXDOMAIN at its own authoritative nameserver
    and the Pages cname is null, so Track Y has no external blocker left
    — but classify_url knows six GitHub hosts and neither canonical one,
    so `final` is a status the audit must report as broken until the
    classifier learns it
  * build_audit.py:45-46 -> :46-47 and :45-57 -> :46-57; wasm
    mirror-orphans "a dozen" -> five, matching Track X

graph.txt is a 100-node digraph, not the 15-node toy audit_annotations
described — that string renders into the public dashboard.

Strict audit exit 0, consumed-file check 18/18.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings August 10, 2026 00:51

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Documentation maintenance pass to keep the repo’s storage/URL guidance and PLAN checkpoints consistent with the current CI-enforced rules (post #55/#56/#57), plus a small public-dashboard annotation correction.

Changes:

  • Update README.md “Referencing data” guidance to runtime-key the interim URL form (CPython vs browser/lecture-wasm) and to hard-ban media.githubusercontent.com.
  • Refresh PLAN.md to reflect current CI coverage (rules 5/6 + resolvability), correct drifted references/figures, and record the Track Y classifier blocker at cutover.
  • Update AGENTS.md URL-form section to include runtime + CORS constraints and add a “>100 MiB ladder” decision rule; fix graph.txt annotation in scripts/audit_annotations.yml.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File Description
scripts/audit_annotations.yml Corrects graph.txt description used by the audit dashboard rendering.
README.md Replaces “single safe interim URL form” with runtime-specific guidance and bans the media host.
PLAN.md Aligns repoint-rule/CI coverage statements and Track Y notes with current enforcement and known gaps.
AGENTS.md Updates URL-form guidance to include CORS/runtime constraints and adds the large-file handling ladder.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread AGENTS.md Outdated
The table used the full header name in one row and abbreviated it to ACAO
in the next, and the abbreviation appears nowhere else in the repo's docs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@mmcky
mmcky merged commit 5ea7927 into main Aug 10, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants