docs: say where the tests are and how to reproduce this tree - #3
Draft
open-coder-ai wants to merge 1 commit into
Draft
docs: say where the tests are and how to reproduce this tree#3open-coder-ai wants to merge 1 commit into
open-coder-ai wants to merge 1 commit into
Conversation
A review on the GitHub Copilot marketplace submission flagged that these entries "expose externally implemented security enforcement without repository-local integration tests". The observation is correct and the gap is real, but it is a documentation gap rather than a testing one -- and it will be raised by every marketplace we submit to, so it is worth closing properly rather than answering in a comment thread. The policies ARE tested. Every policy in the catalog ships an eval suite at base/<policy>/evals/suite.yaml, and publish.yml runs `chock check` and `chock check --only evals` as a gate before anything is packaged, so a policy whose evals fail cannot reach this repository at all. None of that was discoverable from here: the word "eval" did not appear in this README or in PLUGINS.md. A reviewer following the listing's repository link landed on a page that asserted trustworthiness without showing any way to check it. So the Trust section now states where the tests live and why they live there -- the catalog holds the policy source; this repository is compiled output -- and adds a runnable reproduction: clone this tree, the catalog and the framework at the release it was published from, rebuild, and diff. Two silent git commands mean the tree is byte-identical to a fresh build. The recipe was executed against this repository before being written down, not derived from reading the workflow, because a verification procedure nobody has run is exactly the kind of unearned claim this section exists to avoid. Also stated plainly: this README is the one file the publisher never writes, so it alone sits outside the generated-only guarantee. A page explaining what may be trusted should be honest about its own standing. Pinning guidance is included because the failure it prevents is concrete. A marketplace that pins a tag alone follows a movable reference; a tag plus the full commit SHA is what holds reviewed bytes still. No generated content is touched, so generated-only is unaffected. Signed-off-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A Copilot review on the GitHub marketplace submission flagged that our entries "expose externally implemented security enforcement without repository-local integration tests."
The observation is correct, and it will be raised by every marketplace we submit to — so this closes it in the repository rather than in one comment thread.
The gap was documentation, not testing
The policies are tested. Every policy in the catalog ships
base/*/evals/suite.yaml, andpublish.ymlrunschock checkandchock check --only evalsas a gate before anything is packaged — a policy whose evals fail cannot reach this repository.None of that was discoverable from here. The word "eval" appeared zero times in this README and in
PLUGINS.md. A reviewer following a listing'srepository:link landed on a page asserting trustworthiness with no way to check it.What this adds to the Trust section
gitcommands mean byte-identical.publish.ymlnever writes, so it alone sits outside the generated-only guarantee. A page about what may be trusted should say where it stands itself.The recipe was run, not derived
I executed it against this repository before writing it down — a verification procedure nobody has run is exactly the unearned claim this section exists to avoid.
All four reproduce byte-identically from catalog
ed2930e+ chockv0.7.0.Checks
generated-onlyis unaffected.Noted while verifying — not fixed here
generated-only.ymlchecks outopen-coder-ai/chockwith noref:, so it verifies against a movingmain, whilepublish.ymlbuilds from a pinnedframework_ref. They agree today only because chock's two commits pastv0.7.0are CI-only and don't touch the emitter. The next chock merge that changes emitter output turns this repo'sverifyred on an untouched tree. Worth its own change; deliberately not bundled with a docs PR.