feat: add doc-accuracy reviewer to catch harmful documentation changes

[Iheanacho-ai]

What

Adds make check-doc-accuracy, a reviewer that checks docs for changes that could harm a reader who follows them (commands that lose data, destroy things, or weaken security, not just typos). Also runs it in CI on every docs PR, forks included, as an advisory check.

Why

The worst doc bugs are commands that run fine but still do damage, like a service started without its data mount or a dropped safeguard. They pass every syntax check, so a flag checker won't catch them. You need something that reads the snippet and reasons about what happens if you run it. Until now that only ran when someone remembered to run it locally.

Change

• tools/doc-accuracy: a Go program that drives the claude CLI as a read-only reviewer (it reports, never edits). It reviews the .mdx files you changed and checks claims about Talos/Omni against the upstream source.
• Looks for harm first: data loss, destructive commands, removed safeguards, security downgrades, then plain wrong flags and values.
• Output: a short terminal summary, a JSON file (verdict + a reason and fix per finding), and in CI, inline annotations plus one summary comment that updates in place on each push.
• CI (.github/workflows/doc-accuracy.yaml): uses pull_request_target so it works on fork PRs. Checks out our base branch only, never the fork's code, and reads the PR as a text diff. Nothing from the PR is run. It never blocks merge.

Usage

make check-doc-accuracy                                # files you changed vs HEAD
make check-doc-accuracy DOC=public/omni/....mdx        # one specific file
make check-doc-accuracy DOC_ACCURACY_BASE=origin/main  # PR-style, vs a branch
make check-doc-accuracy-all                            # the whole public/ tree (slow)

Testing

• go test, go vet, and gofmt all pass.
• Ran it against a sample doc with four planted problems (etcd with no volume mount, rm -rf /, --insecure on a bootstrapped node, a stale image tag). It caught all four at the right severity and checked the current Talos version upstream. Verdict FAIL, exit 1.
• Confirmed only a CRITICAL fails the run; warnings and notices don't. No changed docs pass cleanly, and a missing claude CLI gives a clear message.
• Not yet tested on a live PR, since pull_request_target runs the workflow from main, so the annotations and comment only show up once this merges. The ANTHROPIC_API_KEY secret is set.

[sterlingkoch]

From Claude:
--allowedTools is ignored under --permission-mode bypassPermissions (claude-code#12232). The whitelist here isn't actually a whitelist, so Bash is unrestricted and the read-only claim doesn't hold. --disallowedTools Edit,Write should still block those two. --permission-mode dontAsk with this allowedTools list, or --tools as a real whitelist, would do what this is aiming for.

[sterlingkoch]

--insecure as CRITICAL will fail half the Talos bootstrap corpus (talosctl apply-config --insecure). Worth carving out the documented bootstrap case so this doesn't train us to ignore the tool.

[sterlingkoch]

The default base is HEAD, so after a local commit this won't actually review anything (DOC_ACCURACY_BASE=origin/main is in the README but easy to miss). Defaulting to origin/main (with a fallback) would match how this will actually get run.

[sterlingkoch]

If the review step errors (timeout, claude crash, no verdict), this manufactures a PASS. Combined with the "existing comment" branch below, a failed run will overwrite a previous FAIL with "Nothing flagged."

Gate this on steps.review.outcome. If the review step didn't succeed, post "reviewer didn't finish, see the log" and leave the last real comment alone. A missing findings file is not a clean review.

[Iheanacho-ai]

Good catch, fixed. The tool now emits a tri-state verdict (PASS/FAIL/INCOMPLETE); a crash or no-verdict run is INCOMPLETE and posts a separate 'didn't finish' note instead of overwriting the findings comment

[sterlingkoch]

Review pass. Leading with the good news, because I went in expecting the classic pull_request_target hole and it isn't there.

A fork contributor cannot execute code on the privileged runner. The checkout at .github/workflows/doc-accuracy.yaml:34-42 has no ref:, so the tree is the base branch and the fork's ref is never fetched. Fork content enters only as text via gh pr diff, and I traced every path from that data to exec.Command and found none. In dataOnly mode main.go:206-214 stitches diff sections rather than taking the git diff -- <files> branch, so attacker-derived filenames never reach a subprocess. Diff-header forgery is blocked too, since splitDiffByFile only reads a +++ before the first @@ and hunk body lines always carry a leading space, +, or -. The session runs --tools Read Grep Glob WebFetch with no Bash, Write, or Edit, and WebFetch allowlisted to three GitHub hosts. GITHUB_TOKEN isn't in the review step's env. That's a careful design for a trigger that usually isn't.

The caveat is that prompt injection through the diff can still make the model read the key, which the code honestly admits at main.go:1009-1017, and the only thing stopping egress is the WebFetch domain allowlist enforced by a third-party CLI. One control carrying the whole design is why item 1 matters.

1. Don't leave a second credential next to the first. .github/workflows/doc-accuracy.yaml:34-35. actions/checkout persists credentials by default, so the job's pull-requests: write token lands in .git/config as an extraheader, inside the exact directory we then point a Read/Grep-capable model at while it chews on attacker-controlled diff text. Add with: persist-credentials: false.

2. No cost bound and no job timeout. .github/workflows/doc-accuracy.yaml:31-32 and tools/doc-accuracy/main.go:40. At reviewBatchSize = 20, #735 (157 mdx) and #733 (162 mdx) would each fan out to 8 or 9 separate Claude sessions per push, each with its own 10-minute budget, against the 6-hour default job limit. #735 is pure frontmatter backfill, which is the worst value-per-token case there is. Want a timeout-minutes: 30 and a max-files bail-out that posts "too large to review" instead of fanning out.

3. I don't think the inline annotations render. .github/workflows/doc-accuracy.yaml:197. Under pull_request_target the check run's head SHA is the base commit, not the PR head, so annotations shouldn't attach to the Files changed tab. If that's right, this line points readers at nothing and the details/fix text (the useful half) is only visible in the run log. The whole "terse comment points at the detailed annotation" split depends on it, so worth confirming on a live run.

4. Silent truncation. tools/doc-accuracy/main.go:221. truncateLines drops everything past 4000 diff lines, and in diff-file mode the model is explicitly told not to read the changed files from disk (main.go:831), so it can't recover the rest and has no signal it happened. Append a "diff truncated, N lines omitted" marker so the review doesn't quietly cover less than it claims.

5. Expect false positives until the bar is calibrated. tools/doc-accuracy/reviewer-prompt.md:138-144 puts "a wrong value" and "would break on copy-paste" in CRITICAL, and only CRITICAL produces the red FAIL comment. Advisory so it won't block, but plan on a 🔴 landing on a fair number of routine PRs at first.

Minor. doc-accuracy.yaml:19 includes tools/doc-accuracy/** in paths, so a PR to the tool triggers the job, which under pull_request_target runs the base version against zero mdx files. And reviewer-prompt.md:121-124 claims "only the text of your last message is shown," which isn't true, since runClaude captures every assistant text block at main.go:1080-1086.

One thing to flag: this workflow has never actually run. pull_request_target executes from the default branch and the file isn't on main yet, so the two green checks here are unrelated jobs. Annotation rendering, the comment upsert finding its prior comment, and secret availability on fork runs are all unexercised.

Good work. The annotation escaping is right, with escapeData/escapeProp at main.go:573-587 turning newlines into %0A so a model-authored message can't forge a second ::error command. The INCOMPLETE handling is the best part: overallVerdict at main.go:419-428 returns PASS only on a completed clean run, and the workflow keys off steps.review.outcome rather than conclusion at :103, which is the correct choice under continue-on-error and easy to get wrong. A crash can't overwrite a real FAIL. The prompt has an explicit untrusted-content paragraph, tells the model to check upstream source rather than reason from memory, and carves out the talosctl --insecure maintenance-mode exception, which is real Talos nuance an off-the-shelf prompt would get wrong. 32 tests including path traversal and the +++-inside-a-hunk case. gofmt, go vet, go build, go test all clean, and the CLI pin at @2.1.226 is the right call.