Skip to content

fix(prompt): informational asks are not mutation work (#884) - #922

Draft
justrach wants to merge 1 commit into
mainfrom
fix/884-read-only-summary-scope
Draft

fix(prompt): informational asks are not mutation work (#884)#922
justrach wants to merge 1 commit into
mainfrom
fix/884-read-only-summary-scope

Conversation

@justrach

Copy link
Copy Markdown
Owner

What changed / Why

A default turn treated “go through the codebase and summarize what it does” as implementation work: todos, fan-out, a full test run, and a separate citation pass. That is a policy bug, not tool latency — mutation completion language in the default prompt outranked “match verification to the ask.”

This gates todo, fan-out, tests, and citation-hunting on summarize/explain/inspect verbs and the absence of a requested change. Mutation tasks still require read-before-edit, root-cause fixes, and in-project verification. The harness records the selected intent on the JSONL trace and, after six model rounds on an informational turn, injects one checkpoint asking whether the model can answer now. Lean -p fake_done does not bounce a completed summary.

Evals that would have caught this:

  • Classifier unit tests (informational vs summarize-then-fix)
  • Prompt pins for the default and lean branches
  • Three scripted harness_behavior cases (policy present, checkpoint fires, mutation skips it)
  • readonly-summary graff-eval trap: fail if tests/builds ran or sources were edited

Fixes #884.

Verification

  • zig build test -Dtest-filter=884 — pass
  • zig build test -Dtest-filter=golden — pass
  • zig build test — 2227 pass, 1 skip
  • scripts/eval-tier1.sh via pre-push — green

Default coding policy treated a read-only codebase summary as
implementation work: todos, fan-out, full test runs, and a separate
citation pass. That is the 6.4x slowdown against a matched Codex
trajectory, not tool latency.

Gate todo, fan-out, tests, and citation-hunting on summarize/explain/
inspect verbs with no requested change. Mutation tasks keep
read-before-edit, root-cause, and in-project verification. The harness
records intent on the trace and injects one checkpoint after six model
rounds; lean fake_done does not bounce a completed summary.

Evals pin the classifier, the prompt branch, scripted exploration
checkpoints, and a graff-eval trap that fails on test/build or edits.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Default coding policy over-scopes read-only codebase summaries (6.4x slower than matched Codex trajectory)

1 participant