reflect: detect Codex no-verify thrash - #438
Merged
Merged
Conversation
Review claim: Codex rollout audits expose same-problem thrash from tool-call history, and reflect-on-thrash fires on a no-verify fixture while staying silent on clean Codex input. Review lane: behavior Review unit: engine-runtime Safety invariant: Synthetic fixtures only; no live transcript content is committed, no Invoker repair workflow is modified, and no PR is merged.
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_86897a73-945e-4a17-90ff-9363a0901236) |
Contributor
|
Tick the box to add this pull request to the merge queue (same as
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Codex rollout audits now recover patch and command history from tool calls.
That lets reflect-on-thrash flag edit streaks with no verification and repeated failed command outputs.
The fixtures prove a Codex no-verify session fires, while clean Codex input stays silent.
Review Claim
Codex rollout audits expose same-problem thrash from tool-call history, and reflect-on-thrash fires on a no-verify fixture while staying silent on clean Codex input.
Review Lane
behavior
Review Unit
engine-runtime
Safety Invariant
Synthetic fixtures only; no live transcript content is committed, no Invoker repair workflow is modified, and the PR is not merged. Assumption: upstream non-interactive task supplied this invariant, so chat confirmation is unconfirmed.
Slice Rationale
This slice adds the catstack-owned catch for the reflected Codex session. Broader Invoker worker changes, log-bounding, and SQLite fixture linting are separate review units.
Non-goals
Test Plan
Test Plan
python3 -m unittest discover -s engine/skills/reflect/scripts/tests -vpython3 -m unittest discover -s engine/hooks/reflect-on-thrash/tests -vpython3 engine/skills/reflect/scripts/token_audit.py codex <live Codex rollout> --out /tmp/reflect-01a082c3-audit-after.jsonpython3 - <<'PY' import sys sys.path.insert(0, 'engine/hooks/reflect-on-thrash') import detect print(detect.thrash_hits('<live Codex rollout>')) PYpython3 engine/skills/reflect/scripts/top_sessions.py 10git diff --checkpython3 engine/skills/make-pr/scripts/preflight.py --base origin/mainnode engine/skills/draft-pr/scripts/lint-diff-atomicity.mjs --base origin/mainRevert Plan
Revert Plan
git revert 43732b4Note
Low Risk
Offline audit/heuristic changes and synthetic test fixtures only; no auth, data migration, or user-facing runtime paths beyond richer reflect flags on Codex sessions.
Overview
Codex rollout audits now reconstruct Bash and patch/Edit steps from
custom_tool_call/execpayloads (parsedcmdandapply_patchpaths), then apply the same same-problem thrash heuristics Claude mode already used:no-verify-edit-streak(≥3 edits without test/build-style verification or a direct run of the edited file) andrecurring-failure-signatures(repeated non-zero exit tool output with numbers normalized). Those flags land inaudit_codexresults and--outJSON (n_recurring_failures,longest_edit_streak_no_verify).Tests and docs add synthetic Codex JSONL helpers, unit cases for streak vs verified patch vs recurring failures, a reflect-on-thrash hook test that a calm three-patch Codex session hits
no-verify-edit-streakwithoutintervention-must-automate, and an update to cost-audit.md describing Codex--outthrash coverage.Reviewed by Cursor Bugbot for commit 43732b4. Bugbot is set up for automated code reviews on this repo. Configure here.