Document public+gated HuggingFace repo posture

[MaxGhenis]

Summary

policyengine/policyengine-uk-data-private on Hugging Face changed from private to public + gated (manual approval) on 31 July 2026. This PR updates the data-safety rules in CLAUDE.md and .github/CONTRIBUTING.md to match: the thing to protect is now the gate, not the private flag.

Why the repo changed

HF access grants are no-ops on private repos — every approved external UKDS-licensed researcher (grants dating to Dec 2024) got 401s despite being on the accepted list. Public + gated makes the grant flow function: file names are world-visible, file contents remain restricted to the org and gate-approved users.

What the new rules say

• Never expose data contents publicly; the gate is the boundary.
• Never weaken/disable the gate or grant access without UKDS FRS proof.
• Never flip the repo back to private (silently voids all external grants) — including as a "fix" for CI auth errors; grant the CI account through the gate instead (see CI HUGGING_FACE_TOKEN failing with 401 on private dataset downloads policyengine-uk#1816).
• Also corrects the public-repo reference to the actual repo id (policyengine-uk-data-public; the previously named policyengine/policyengine-uk-data does not exist).

Docs-only; no pipeline or code changes.

🤖 Generated with Claude Code

[vahid-ahmadi]

Reviewed the substance against the live Hub state. The direction is right and this should land — the gate, not the private flag, is the boundary now. Three things to fix first, one of which changes the security reading of the old text.

Verified accurate. policyengine/policyengine-uk-data-private returns private=False, gated='manual'. Anonymous list_repo_files returns all 60 entries while an anonymous file fetch 401s, so "file names are world-visible, file contents restricted" is exactly right.

1. The -public correction is right, but the stated reason is wrong — and the truth is worse.

The PR body says policyengine/policyengine-uk-data "does not exist". It does. It is an HF redirect to the gated repo:

requested 'policyengine/policyengine-uk-data'         -> canonical id 'policyengine/policyengine-uk-data-private'  sha=b3322dd5  gated=manual
requested 'policyengine/policyengine-uk-data-private' -> canonical id 'policyengine/policyengine-uk-data-private'  sha=b3322dd5  gated=manual

Same repo, same SHA. So the pre-PR CLAUDE.md was describing the gated UKDS repo as "the separate public repo … maintained through a separate process". That is the most dangerous line in the file to have had wrong, and the rename fixes it — but for the opposite reason to the one given. A nonexistent identifier fails loudly. A redirect silently succeeds and lands UKDS-derived data in a repo whose only protection is a manual gate.

Suggest correcting the body, and adding a line to the rule itself: the bare name policyengine/policyengine-uk-data still resolves to the gated repo, so "upload it to policyengine-uk-data" is an exfiltration path rather than a 404. That is the failure mode a future contributor (or agent) is most likely to walk into, precisely because the name reads as the public one.

2. grant the CI account through the gate instead is incomplete, and will send the next person down the wrong path.

Diagnosed today on PolicyEngine/policyengine-uk#1815: policyengine-core's download_huggingface_dataset passes a token only when the repo reports private=True (tools/hugging_face.py:81-90):

is_repo_private = fetched_model_info.private   # False since 31 July
authentication_token = None
if is_repo_private:
    authentication_token = get_or_prompt_hf_token()
hf_hub_download(..., token=authentication_token)   # token=None

Since the flip, private is False, so the download goes out anonymously and the gate returns 401 regardless of who has been approved. Gate-granting the CI account is necessary but not sufficient — no token is being sent at all. This is why replacing the HUGGING_FACE_TOKEN secret (policyengine-uk#1816) changed nothing, and why the out-of-band validation of that token passed: that check supplied the token explicitly, which is the step core skips.

Suggest rewording to: grant the CI account through the gate and ensure a token is actually sent — either fix core's predicate (gated alongside private, or just pass the token whenever one exists) or export HF_TOKEN, which huggingface_hub picks up implicitly when the explicit token is None. policyengine-uk#1815 carries the HF_TOKEN workaround for the UK model's workflows.

3. Both failing checks are a GitHub infrastructure outage, not this PR.

Test and Check changelog fragment both died at the action-resolution step on 6 August:

Failed to resolve action download info. Error: Bad Gateway
Failed to resolve action download info. Error: Service Unavailable
##[error]Failed to resolve action download info.

No code ran. A rerun should clear both — this PR has been sitting five days on a transient failure that reads like a real one.

Follow-up, out of scope here. policyengine-uk still points at the redirecting name in docs/book/index.md, docs/book/usage/{simulations,getting-started,scenarios}.md, and two test files (hf://policyengine/policyengine-uk-data/...). Those all resolve to the gated repo today. Worth an issue on that side once this lands, so the docs stop teaching the identifier this PR is disowning.