Backfill the CI-invisible consumers across 16 manifests - #91
Merged
Conversation
The `consumers` list exists so a correction knows what to rebuild, and #69 settled that it records ALL consumers rather than the two the audit can see. Eighteen manifests predate that convention and named only repos inside build_audit.py's SCAN_REPOS. Sixteen of them have at least one consumer outside it; this records them. 28 consumer rows added, taking the corpus from 80 to 108. The two manifests left untouched — japan_population_by_age.csv and lingcod_msy_recovery.csv — genuinely have no consumer outside SCAN_REPOS, which the same sweep establishes rather than assumes. Derived by grepping authenticated tarballs of the four repos outside SCAN_REPOS (lecture-intro.zh-cn, lecture-python.zh-cn, test-actions-lecture-intro, lecture-stats) by BASENAME, with a positive control (a file known to have non-CI-visible consumers, which fired in exactly the two repos it should) and a negative control (a name that cannot exist, zero everywhere) in the same pass. A URL-shaped grep finds none of these: every one of the reads builds its URL across two or more source lines, so matching the assembled URL returns a confident zero. That is the same trap recorded for the hansen_singleton pair, and it is why the sweep keys on the filename. Two of the recorded consumers read a LOCAL copy rather than this repo, and are annotated inline as such: test-actions-lecture-intro's french_rev.md still does np.load('datasets/caron.npy') and the same for nom_balances.npy, while the three .xlsx reads in that same file were repointed; and lecture-intro.zh-cn's prob_dist.md reads _static/lecture_specific/prob_dist/us_adult_heights.csv with a comment that still says to switch once the datasets repo exists. They are listed because they consume the data and a correction must reach them — the note keeps the record honest about how. Both are tracked in QuantEcon/workspace-lectures#46. Every consumer entry is a genuine data read, so the open question of whether a prose-only reference counts as a consumer did not have to be settled here: each repo listed has at least one executable read, and the prose links that exist (the french_rev download list, inflation_history's chapter_3 role) sit alongside a code read of the same file in the same repo. CATALOG.md regenerated, as the consumed-file-check gate requires. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Backfills missing “CI-invisible” consumer entries in existing dataset manifests so the consumers blocks reflect all known lecture consumers (including repos outside SCAN_REPOS), and regenerates CATALOG.md from the updated manifests.
Changes:
- Added additional
consumersentries across 16 existing dataset manifests (primarilylecture-intro.zh-cn,lecture-python.zh-cn, andtest-actions-lecture-intro). - Regenerated
CATALOG.mdto reflect the expanded consumer lists.
Reviewed changes
Copilot reviewed 17 out of 17 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| lectures/us_adult_heights.csv.yml | Adds zh-cn consumers (including a local-copy annotation in prob_dist.md). |
| lectures/realwage.csv.yml | Adds lecture-python.zh-cn consumer for pandas_panel.md. |
| lectures/nom_balances.npy.yml | Adds zh-cn + test-actions-lecture-intro consumers (includes local-copy annotation). |
| lectures/mpd2020.xlsx.yml | Adds zh-cn + test-actions-lecture-intro consumers for long_run_growth.md. |
| lectures/longprices.xls.yml | Adds zh-cn + test-actions-lecture-intro consumers for inflation_history.md. |
| lectures/japan_earthquakes.csv.yml | Adds zh-cn consumer for fitting_distributions.md. |
| lectures/japan_deaths_by_age.csv.yml | Adds zh-cn consumers for distribution lectures. |
| lectures/fig_3.xlsx.yml | Adds zh-cn + test-actions-lecture-intro consumers for french_rev.md. |
| lectures/epl_match_goals.csv.yml | Adds zh-cn consumer for fitting_distributions.md. |
| lectures/employ.csv.yml | Adds lecture-python.zh-cn consumer for pandas_panel.md. |
| lectures/dette.xlsx.yml | Adds zh-cn + test-actions-lecture-intro consumers for french_rev.md. |
| lectures/countries.csv.yml | Adds lecture-python.zh-cn consumer for pandas_panel.md. |
| lectures/chapter_3.xlsx.yml | Adds zh-cn + test-actions-lecture-intro consumers for inflation_history.md. |
| lectures/caron.npy.yml | Adds zh-cn + test-actions-lecture-intro consumers (includes local-copy annotation). |
| lectures/assignat.xlsx.yml | Adds zh-cn + test-actions-lecture-intro consumers for french_rev.md. |
| lectures/ames_house_prices.csv.yml | Adds zh-cn consumers for distribution lectures. |
| CATALOG.md | Regenerated catalog to include the expanded consumer lists. |
Suppressed comments (2)
CATALOG.md:42
- Same catalog issue as
caron.npy: this Used by cell includestest-actions-lecture-intro, but the manifest notes it reads a local copy rather than this repo’s published file. Consider updating the catalog generator to display local-copy notes or otherwise distinguish these consumers.
| [**nom_balances.npy**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/nom_balances.npy)<br><sub>French Revolution — monthly nominal assignat balances, 1789-1796</sub> | constructed | unrecorded | | ✅ permitted | ⚠️ unverifiable | ⚠️ unrecovered | 1.4 KB | [lecture-python-intro · french_rev.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/french_rev.md)<br>[lecture-wasm · french_rev.md](https://github.com/QuantEcon/lecture-wasm/blob/main/lectures/french_rev.md)<br>[lecture-intro.zh-cn · french_rev.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/french_rev.md)<br>[test-actions-lecture-intro · french_rev.md](https://github.com/QuantEcon/test-actions-lecture-intro/blob/main/lectures/french_rev.md) |
CATALOG.md:44
- This Used by cell includes
lecture-intro.zh-cn · prob_dist.md, but the manifest annotates that consumer as reading a local_staticcopy rather than this repo’s file. The generated catalog currently can’t reflect that distinction, so readers may assume it is repointed. Consider updatingscripts/build_catalog.pyto surface per-consumer notes (once recorded in manifests) or adjust the catalog wording to match the broader meaning ofconsumers.
| [**us_adult_heights.csv**](https://github.com/QuantEcon/data-lectures/raw/main/lectures/us_adult_heights.csv)<br><sub>United States — adult standing height by sex, NHANES 2015-2018</sub> | constructed | [National Health and Nutrition Examination Survey (NHANES), US Centers for Disease Control and Prevention, National Center for Health Statistics](https://www.cdc.gov/nchs/nhanes/index.htm) | US Government work — public domain | ✅ permitted | ✅ verified | ✅ committed | 123.1 KB | [lecture-python-intro · prob_dist.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/prob_dist.md)<br>[lecture-python-intro · observed_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/observed_distributions.md)<br>[lecture-python-intro · fitting_distributions.md](https://github.com/QuantEcon/lecture-python-intro/blob/main/lectures/fitting_distributions.md)<br>[lecture-intro.zh-cn · fitting_distributions.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/fitting_distributions.md)<br>[lecture-intro.zh-cn · observed_distributions.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/observed_distributions.md)<br>[lecture-intro.zh-cn · prob_dist.md](https://github.com/QuantEcon/lecture-intro.zh-cn/blob/main/lectures/prob_dist.md) |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Copilot review of #91 found a real semantic conflict rather than a comment nit. Eight manifests carried a header reading "Every lecture that reads this file FROM THIS REPO ... it may still be consumed from a lecture repo's own copy", which explicitly EXCLUDES a local-copy reader. The backfill added three such readers, so under that wording those entries were wrong. manifest-schema.yml already documented the opposite, broader meaning — "how a fix-in-place finds the lectures to rebuild" — so the eight manifest headers were the outliers, not the additions. Resolved in favour of the broader meaning, which is the one that makes the field useful: a correction to these bytes must reach every consumer, whether or not it has been repointed yet. Four changes: The eight headers now say the list answers "what must be rebuilt if these bytes change", that it therefore names consumers still on a local copy, and that such an entry carries a `note`. The three local-copy annotations were YAML comments, which nothing can check. They are now a `note:` field on the consumer entry — the same argument as #86's enforcement: a rule or a fact recorded where no tool can read it is not a record. Each cites QuantEcon/workspace-lectures#46 fully qualified, replacing the `ws#46` shorthand that means nothing when read from this repo. manifest-schema.yml documents `consumers[].note`, why a not-yet-repointed consumer is listed at all, and the fully-qualified-reference requirement. build_catalog.py surfaces the note. Without it the "Used by" column reads as "these lectures fetch this file", which is the wrong inference for exactly the rows that need care. Renders the note's first sentence under the consumer link — enough to block the wrong reading, short enough not to blow out a table cell, with the manifest holding the detail. Verified: 33 manifests and the schema parse; 108 consumer rows, 3 carrying a note, no unexpected keys; the "FROM THIS REPO" wording is gone from every manifest; CATALOG.md regenerates deterministically across two runs; strict audit exits 0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
mmcky
added a commit
that referenced
this pull request
Aug 17, 2026
…in fetch() Copilot review of #92, all four comments valid — and three of them catch this PR breaking a convention it inherited from #91. migration.yml records these three as `landed`, so the lecture still reads its own copies. Under the definition settled in #91, `consumers` answers "what must be rebuilt if these bytes change", and a consumer not yet reading this repo carries a `note` saying so. All three entries were listed without one, which made CATALOG.md's "Used by" column imply a repoint that has not happened. Each now carries a note naming where the lecture actually reads from — a local `_static` path for the two bbh files, an own-repo raw URL for the hansen bundle — and the notes come out in the repoint PR. A worse instance of the same drift, which the review pointed at obliquely: hansen_jagannathan_1991_data.json.yml carried the retired "Every lecture that reads this file FROM THIS REPO" header — the exact wording #91 removed from all eight pre-existing manifests, reintroduced here in a new one because it was drafted from a pre-#91 template. Replaced with the current wording. A grep for that phrase now returns zero manifests again, which is the check worth keeping. builders/bbh_macro_quarterly.py: fetch() read from the ZipFile returned by _open_archive() without closing it, leaking the HTTP range reader's socket to the garbage collector. Now a context manager. This matters more than it looks because the builder is designed to be re-run — the byte-identity check reruns it on every verification pass. bbh_michigan_monthly.py already used `with open_deposit()` and needed no change, which is why the review scoped its comment to the one file. Verified after the changes: both builders still reproduce their committed bytes (git diff over lectures/ empty); the fixed builder raises no ResourceWarning under -W error::ResourceWarning; 36 manifests parse with 6 noted consumers; CATALOG.md regenerated and now renders three warning markers; strict audit exits 0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
mmcky
added a commit
that referenced
this pull request
Aug 17, 2026
… (Track C) (#92) * Land wave C1 — the three advanced.myst datasets that needed no rename (Track C) Moves bbh_macro_quarterly.csv, bbh_michigan_monthly.csv and hansen_jagannathan_1991_data.json out of lecture-python-advanced.myst. All three land byte-identical to the blobs the lecture reads today, so the repoint that follows cannot change a figure: sha256 f14f4256…, 567efe5a… and c81333f3…, each checked against `git show HEAD:<path>` in advanced.myst at 00057ba and recorded in the manifests' integrity.sha256. Status is `landed`, not `repointed` — the lecture still reads its own copies. The repoint PR follows this one, in that order: advanced.myst sets execute_notebooks: "cache", so the repointed cells re-execute against whatever is on this repo's main at that moment. TWO BUILDERS RECOVERED, refuting the annotations they replace. Both bbh files were recorded as `constructed-lost` with "extraction not scripted". Both are in fact fully reconstructible from the authors' Zenodo deposit (doi:10.5281/zenodo.10194324), and the builders here reproduce their committed bytes BYTE FOR BYTE — verified by three end-to-end runs each, independently re-run in a clean clone and again in this branch, with `cmp` reporting no difference every time. They therefore land as `builder_status: committed`, not `unrecovered`. The macro builder pulls the 169 KB workbook out of the 198.8 MB archive with HTTP range requests — four requests, ~296 KB — rather than downloading the lot. Neither builder reads live FRED, and the reason is not the obvious one. The bytes are on a 2012 base and FRED has since rebased to 2017, but the lecture takes log first differences and ratios, which are rebasing-invariant — under a pure rebasing 8 of the 9 VAR inputs are bit-identical. The operative risk is upstream REVISIONS, which is why the builders pin the immutable versioned DOI. hansen_jagannathan_1991_data.json stays `unrecovered`, and the distinction is worth keeping. Its construction IS fully recovered — all 2,462 cells re-derive from Shiller's workbooks and seven FRED series to within 4 ULP, and six of eleven columns are bitwise identical end to end — but the BYTES are not reproducible, so no builder can claim to rebuild it. The manifest records the full specification so Phase 9 is a finishing job rather than an investigation, including the unexplained 0.8938191876245914 level factor on annual.consumption, which cancels out of everything the lecture prints. bbh_michigan_monthly.csv lands `redistribution: restricted`. The Zenodo deposit declares CC BY 4.0, but four of its five columns are University of Michigan Surveys of Consumers aggregates and Michigan's usage agreement forbids redistribution without written consent; the depositors are not the rights holder and the deposit does not record consent. Handled as an inherited exposure per AGENTS.md — the file has been served publicly since 2026-06-12 — and registered for licence review. #35's inventory currently records the opposite and needs the correction. Also here because CI enforces them: the three migration.yml records, the three audit_annotations.yml entries retired (a manifest and an annotation together fails the dual_recorded check added in #90), and CATALOG.md regenerated. openpyxl is pinned — both new builders need it, and ames_house_prices.py and japan_population_by_age.py have needed it since they landed without it ever being declared, because CI never runs builders. Verified: strict audit exits 0 with all five problem keys empty; both builders re-run in this branch leave `git diff -- lectures/` empty; the manifests' sha256 fields match the landed bytes; migration.yml parses to 36 datasets, 33 repointed and 3 landed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Record this PR's number in the three migration.yml records Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Note the three consumers as not-yet-repointed, and close the archive in fetch() Copilot review of #92, all four comments valid — and three of them catch this PR breaking a convention it inherited from #91. migration.yml records these three as `landed`, so the lecture still reads its own copies. Under the definition settled in #91, `consumers` answers "what must be rebuilt if these bytes change", and a consumer not yet reading this repo carries a `note` saying so. All three entries were listed without one, which made CATALOG.md's "Used by" column imply a repoint that has not happened. Each now carries a note naming where the lecture actually reads from — a local `_static` path for the two bbh files, an own-repo raw URL for the hansen bundle — and the notes come out in the repoint PR. A worse instance of the same drift, which the review pointed at obliquely: hansen_jagannathan_1991_data.json.yml carried the retired "Every lecture that reads this file FROM THIS REPO" header — the exact wording #91 removed from all eight pre-existing manifests, reintroduced here in a new one because it was drafted from a pre-#91 template. Replaced with the current wording. A grep for that phrase now returns zero manifests again, which is the check worth keeping. builders/bbh_macro_quarterly.py: fetch() read from the ZipFile returned by _open_archive() without closing it, leaking the HTTP range reader's socket to the garbage collector. Now a context manager. This matters more than it looks because the builder is designed to be re-run — the byte-identity check reruns it on every verification pass. bbh_michigan_monthly.py already used `with open_deposit()` and needed no change, which is why the review scoped its comment to the one file. Verified after the changes: both builders still reproduce their committed bytes (git diff over lectures/ empty); the fixed builder raises no ResourceWarning under -W error::ResourceWarning; 36 manifests parse with 6 noted consumers; CATALOG.md regenerated and now renders three warning markers; strict audit exits 0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
mmcky
added a commit
that referenced
this pull request
Aug 17, 2026
#95) lecture-python-advanced.myst#372 merged, so advanced.myst now reads all three files from this repo. This closes the red window that opens the moment a repoint lands: build_audit fails `--strict` on any dataset marked `landed` while a consumer already reads data-lectures, so main has been red since that merge. Measured in both directions rather than asserted, which is what makes this the wave's acceptance test rather than bookkeeping. Against post-merge main: `landed` gives exit 1 with three `migration_inconsistencies`, one per file; `repointed` gives exit 0 with all five problem buckets empty. The dry-run has to run against the consuming repo's post-merge main, because build_audit reads `origin/main` through git show and cannot see an unmerged branch. The three consumer notes come out. They were added in #92 to record that the lecture still read its own copies while the dataset sat at `landed` — that is now false, and a note claiming an unrepointed consumer would be exactly the kind of stale record #91 added them to prevent. CATALOG.md regenerates with those three warning markers gone; the three that remain are the unrelated ws#46 readers (caron.npy, nom_balances.npy, us_adult_heights.csv), which are genuinely still on local copies. Verified the edit touched nothing else: for each of the three manifests, the only top-level key differing from main is `consumers`, and `source.note`, `license.note` and `integrity.upstream.note` are byte-identical to main. That check is not decorative — the first attempt at this edit matched `integrity.upstream.note` instead, because both sit at the same indent, and silently deleted the provenance record. Anchor on the `consumers:` block. Ledger is now 36 datasets, all `repointed`, 0 `landed`. NOT done here, and the order matters: advanced.myst has not published since this repoint, so its published notebooks still carry the old URLs. The three local copies stay until it does. That is load-bearing for hansen_jagannathan_1991_data.json, which is fetched over the network at cell-execution time and therefore has no stale-serving grace period — its deletion breaks every published notebook and Colab reader the instant the blob leaves advanced.myst's main. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Open
41 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The last tidy-up item from the Track C work plan (QuantEcon/workspace-lectures#45): the 18 manifests that predate #69's convention recorded only the consumers
build_audit.pycan see, and #69 settled that the list records all of them — because for the repos outsideSCAN_REPOS, this file is the only machine-readable place they exist.28 consumer rows added across 16 manifests, taking the corpus from 80 to 108.
us_adult_heights.csvfitting_distributions,observed_distributions,prob_distassignat.xlsx,caron.npy,chapter_3.xlsx,dette.xlsx,fig_3.xlsx,longprices.xls,mpd2020.xlsx,nom_balances.npyames_house_prices.csv,japan_deaths_by_age.csvcountries.csv,employ.csv,realwage.csvlecture-python.zh-cnpandas_panel.mdepl_match_goals.csv,japan_earthquakes.csvfitting_distributions.mdTwo manifests were deliberately left alone.
japan_population_by_age.csvandlingcod_msy_recovery.csvhave no consumer outsideSCAN_REPOS— established by the same sweep that found the others, not assumed from silence.Method
Grepped authenticated tarballs of the four repos outside
SCAN_REPOS—lecture-intro.zh-cn,lecture-python.zh-cn,test-actions-lecture-intro,lecture-stats— by basename, over.mdand.ipynb, excluding_build.Two controls ran in the same pass. A positive control on a file already known to have non-CI-visible consumers fired in exactly the two repos it should and stayed silent in the other two. A negative control on a name that cannot exist returned zero everywhere.
Keying on the basename rather than the URL is load-bearing, not stylistic. Every one of these reads builds its URL across two or more source lines — for example zh-cn's
fitting_distributions.mdsplits'https://github.com/QuantEcon/data-lectures/raw/main/'from'lectures/ames_house_prices.csv'— so a grep for the assembled URL returns a clean zero on all of them. This is the same trap already recorded in thehansen_singletonmanifests.An earlier run of this sweep produced a false all-clear worth recording: written as a
forloop over an unquoted shell variable, zsh did not word-split it, so a single 18-filename string was searched and every file reported "no consumer". Re-run under an explicitbasharray with the controls above, it returned the results here. Same family as therg-shim trap in this repo's notes.Two consumers read a local copy, and are annotated as such
The sweep turned up two readers that consume the data but not from this repo. They are listed as consumers — a correction must reach them — with an inline note recording how they actually read it:
test-actions-lecture-intro'sfrench_rev.mdstill doesnp.load('datasets/caron.npy')and the same fornom_balances.npy, while the three.xlsxreads in the same file were repointed tobase_url. A partial repoint.lecture-intro.zh-cn'sprob_dist.mdreads_static/lecture_specific/prob_dist/us_adult_heights.csv, under a comment that still says to switch once the datasets repo exists. Its copy is byte-identical to the canonical one.Neither is broken — both repos hold their own copy — and both are invisible to the strict audit, which scans only
SCAN_REPOS. Filed as QuantEcon/workspace-lectures#46 with the line numbers and the fix for each.The prose-only question did not have to be settled
#69 does not say whether a
{download}role or a markdown link counts as a consumer. It did not arise: every repo listed here has at least one executable read of the file it is listed against, and the prose references that exist — thefrench_revdownload list,inflation_history'schapter_3role — sit alongside a code read of the same file in the same repo. The question stays open for a case where prose is the only reference.Verification
All 33 manifests parse; 108 consumer rows, zero malformed.
CATALOG.mdregenerated as theconsumed-file-checkgate requires. Strict audit exits 0.🤖 Generated with Claude Code