Skip to content

feat(vis): add loop trace evaluation metrics - #2897

Open
yu-xin-c wants to merge 2 commits into
MoonshotAI:mainfrom
yu-xin-c:codex/trace-eval-infrastructure
Open

feat(vis): add loop trace evaluation metrics#2897
yu-xin-c wants to merge 2 commits into
MoonshotAI:mainfrom
yu-xin-c:codex/trace-eval-infrastructure

Conversation

@yu-xin-c

Copy link
Copy Markdown

Related Issue

Related to #2622

Problem

The visualizer can reconstruct turns, steps, tools, and context fill, but it cannot quantify the two loop shapes reported in the linked issue: long exact runs and rotating reuse of a small call alphabet. This makes candidate breaker policies difficult to calibrate against healthy traces before changing runtime behavior.

What changed

  • Add a deterministic, client-side loop trace evaluator that segments prompt-to-prompt phases and canonicalizes tool arguments.
  • Report phase-local exact-call repetition, longest immediate runs, peak repetition in a fixed-size rolling window, cancel/steer/compaction markers, and before/after steer histogram overlap.
  • Keep prompt text, tool arguments, tool outputs, and internal fingerprints out of the returned report.
  • Surface the metrics in the Timeline tab and allow copying the aggregate versioned JSON report.
  • Add synthetic regression traces for long exact runs, rotating small alphabets, healthy argument drift, steer response, phase boundaries, markers, and payload exclusion.

This PR is diagnostics-only. It does not classify a session as looping, choose alert thresholds, or change agent/runtime behavior.

Verification

  • pnpm --filter @moonshot-ai/vis-web test
  • pnpm --filter @moonshot-ai/vis-web typecheck
  • pnpm --filter @moonshot-ai/vis-web build
  • pnpm typecheck
  • pnpm exec oxlint --type-aware --quiet

Checklist

  • I have read the CONTRIBUTING document.
  • I have linked a related issue, or explained the problem above.
  • I have added tests that prove my feature works.
  • Ran gen-changesets skill; vis packages are explicitly ignored and need no changeset.
  • Ran gen-docs review; this self-contained debug UI addition needs no docs update.

@changeset-bot

changeset-bot Bot commented Aug 13, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: e0b7db8

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@yu-xin-c
yu-xin-c force-pushed the codex/trace-eval-infrastructure branch from 2a89721 to 73c6063 Compare August 14, 2026 15:42
@yu-xin-c
yu-xin-c marked this pull request as ready for review August 14, 2026 15:43

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 73c60638a4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread apps/vis/web/src/lib/loop-eval.ts
@yu-xin-c
yu-xin-c force-pushed the codex/trace-eval-infrastructure branch from 73c6063 to e0b7db8 Compare August 15, 2026 10:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant