Skip to content

reflect: detect Codex no-verify thrash - #438

Merged
EdbertChan merged 1 commit into
mainfrom
reflect/codex-thrash-audit-20260911
Sep 12, 2026
Merged

reflect: detect Codex no-verify thrash#438
EdbertChan merged 1 commit into
mainfrom
reflect/codex-thrash-audit-20260911

Conversation

@EdbertChan

@EdbertChan EdbertChan commented Sep 11, 2026

Copy link
Copy Markdown
Owner

Summary

Codex rollout audits now recover patch and command history from tool calls.

That lets reflect-on-thrash flag edit streaks with no verification and repeated failed command outputs.

The fixtures prove a Codex no-verify session fires, while clean Codex input stays silent.

Review Claim

Codex rollout audits expose same-problem thrash from tool-call history, and reflect-on-thrash fires on a no-verify fixture while staying silent on clean Codex input.

Review Lane

behavior

Review Unit

engine-runtime

Safety Invariant

Synthetic fixtures only; no live transcript content is committed, no Invoker repair workflow is modified, and the PR is not merged. Assumption: upstream non-interactive task supplied this invariant, so chat confirmation is unconfirmed.

Slice Rationale

This slice adds the catstack-owned catch for the reflected Codex session. Broader Invoker worker changes, log-bounding, and SQLite fixture linting are separate review units.

Non-goals

  • No Invoker repair workflow changes.
  • No live transcript fixtures or paths.
  • No long-gate monitor, CI-log hook, or SQLite fixture lint.

Test Plan

Test Plan
  • python3 -m unittest discover -s engine/skills/reflect/scripts/tests -v
  • python3 -m unittest discover -s engine/hooks/reflect-on-thrash/tests -v
  • python3 engine/skills/reflect/scripts/token_audit.py codex <live Codex rollout> --out /tmp/reflect-01a082c3-audit-after.json
  • python3 - <<'PY' import sys sys.path.insert(0, 'engine/hooks/reflect-on-thrash') import detect print(detect.thrash_hits('<live Codex rollout>')) PY
  • python3 engine/skills/reflect/scripts/top_sessions.py 10
  • git diff --check
  • python3 engine/skills/make-pr/scripts/preflight.py --base origin/main
  • node engine/skills/draft-pr/scripts/lint-diff-atomicity.mjs --base origin/main

Revert Plan

Revert Plan
  • Safe to revert? Yes
  • Revert command: git revert 43732b4
  • Post-revert steps: None
  • Data migration? No

Note

Low Risk
Offline audit/heuristic changes and synthetic test fixtures only; no auth, data migration, or user-facing runtime paths beyond richer reflect flags on Codex sessions.

Overview
Codex rollout audits now reconstruct Bash and patch/Edit steps from custom_tool_call / exec payloads (parsed cmd and apply_patch paths), then apply the same same-problem thrash heuristics Claude mode already used: no-verify-edit-streak (≥3 edits without test/build-style verification or a direct run of the edited file) and recurring-failure-signatures (repeated non-zero exit tool output with numbers normalized). Those flags land in audit_codex results and --out JSON (n_recurring_failures, longest_edit_streak_no_verify).

Tests and docs add synthetic Codex JSONL helpers, unit cases for streak vs verified patch vs recurring failures, a reflect-on-thrash hook test that a calm three-patch Codex session hits no-verify-edit-streak without intervention-must-automate, and an update to cost-audit.md describing Codex --out thrash coverage.

Reviewed by Cursor Bugbot for commit 43732b4. Bugbot is set up for automated code reviews on this repo. Configure here.

Review claim: Codex rollout audits expose same-problem thrash from tool-call history, and reflect-on-thrash fires on a no-verify fixture while staying silent on clean Codex input.

Review lane: behavior

Review unit: engine-runtime

Safety invariant: Synthetic fixtures only; no live transcript content is committed, no Invoker repair workflow is modified, and no PR is merged.
@cursor

cursor Bot commented Sep 11, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_86897a73-945e-4a17-90ff-9363a0901236)

@mergify

mergify Bot commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

  • Queue this pull request

@EdbertChan
EdbertChan merged commit 07b480e into main Sep 12, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant