fix(prompt): informational asks are not mutation work (#884) - #922
Draft
justrach wants to merge 1 commit into
Draft
fix(prompt): informational asks are not mutation work (#884)#922justrach wants to merge 1 commit into
justrach wants to merge 1 commit into
Conversation
Default coding policy treated a read-only codebase summary as implementation work: todos, fan-out, full test runs, and a separate citation pass. That is the 6.4x slowdown against a matched Codex trajectory, not tool latency. Gate todo, fan-out, tests, and citation-hunting on summarize/explain/ inspect verbs with no requested change. Mutation tasks keep read-before-edit, root-cause, and in-project verification. The harness records intent on the trace and injects one checkpoint after six model rounds; lean fake_done does not bounce a completed summary. Evals pin the classifier, the prompt branch, scripted exploration checkpoints, and a graff-eval trap that fails on test/build or edits.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed / Why
A default turn treated “go through the codebase and summarize what it does” as implementation work: todos, fan-out, a full test run, and a separate citation pass. That is a policy bug, not tool latency — mutation completion language in the default prompt outranked “match verification to the ask.”
This gates todo, fan-out, tests, and citation-hunting on summarize/explain/inspect verbs and the absence of a requested change. Mutation tasks still require read-before-edit, root-cause fixes, and in-project verification. The harness records the selected intent on the JSONL trace and, after six model rounds on an informational turn, injects one checkpoint asking whether the model can answer now. Lean
-pfake_done does not bounce a completed summary.Evals that would have caught this:
readonly-summarygraff-eval trap: fail if tests/builds ran or sources were editedFixes #884.
Verification
zig build test -Dtest-filter=884— passzig build test -Dtest-filter=golden— passzig build test— 2227 pass, 1 skipscripts/eval-tier1.shvia pre-push — green