docs: nest Scorers, Self-improvement, and Benchmarks under Measure and improve (GROW-6131) - #721
docs: nest Scorers, Self-improvement, and Benchmarks under Measure and improve (GROW-6131)#721warp-agent-staging[bot] wants to merge 9 commits into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
…d improve (GROW-6131) Split Scorers and Self-improvement out of measure-and-improve.mdx into dedicated pages, nest all three (plus an overview) under one expanded 'Measure and improve' sidebar group, and repoint internal cross-references and the terminology.md Scorer glossary entry at the new pages. Benchmarks keeps its existing URL and content; only its sidebar parent changes, so no vercel.json redirects are needed. The old #configure-scorers and #configure-and-review-self-improvement anchors remain valid as short pointer stubs on the overview page.
|
@warp-agent-staging[bot] I'm starting a first review of this pull request. You can view the conversation on Warp. I completed the review and no human review was requested for this pull request. Comment Powered by Oz |
warp-agent-staging
Bot
changed the title
![warp-for-oss[bot]](https://avatars.githubusercontent.com/in/3450139?s=60&v=4){kind=small}
There was a problem hiding this comment.
Overview
This PR reorganizes the Factories measurement docs by moving Scorers and Self-improvement details into dedicated child pages, updating related links, and nesting the pages under Measure and improve in the sidebar. The split is structurally sound, and no security issues apply to these docs/sidebar changes.
Concerns
- Some changed Scorers links still point to the overview page even though the PR makes
measure-and-improve/scorers.mdxthe canonical Scorers destination. - The updated overview frontmatter description reads like a page label/link list instead of a standalone summary of what the reader can do.
- The PR description says this is spec-only and contains no content changes, but the diff updates public docs content and sidebar navigation; update the PR description/verification before merge.
Verdict
Found: 0 critical, 0 important, 3 suggestions
Approve with nits
Comment /warp-agent-review on this pull request to retrigger a review (up to 3 times on the same pull request).
Powered by Oz
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Review summary
The independent agent completed its review for this commit.
Findings
- PR body (
## Summary/## Verification) — [CRITICAL] The PR body has no## Documentation risksection and no## Unverified claimssection at all, even though the PR carries thewarpy-factorylabel.check_pr_contract.py --bodyconfirms this fails with 'missing required section: '## Documentation risk''. This is a hard v1 contract violation that fails the required 'Docs technical references' CI check and leaves the low-risk-allowlist classification undeclared. Requested resolution: add a## Documentation riskblock (per.agents/skills/doc_quality_policy/finalize_pr_contract.py build) — given the diff only reorganizes and cross-links existing, unchanged prose into new files with no new technical claims,Risk: lowwith a rationale citing the verbatim content move is the accurate classification — plus a## Unverified claimssection (can read 'None'). - PR body
## Summary— [CRITICAL] The description reads 'Spec-only PR for GROW-6131 ... No content changes yet — this PR is a placeholder that implementation will fill in once the spec ... is approved' and## Verificationreads 'N/A yet — spec phase only.' This no longer matches the diff at head 8c689ae: it changes 11 files (86 insertions / 37 deletions), splitsfactories/measure-and-improve.mdxinto three pages, rewrites 6 cross-link call sites, and restructuressrc/sidebar.tsnavigation. A reviewer trusting the description (the PR already hasrachaelrenkrequested as reviewer) would believe there is nothing to check. Requested resolution: update the PR body to describe the actual implementation change and add real verification steps (e.g. broken-link check output, confirmation the moved prose is unchanged) before requesting review.
Verdict
Request changes
|
Re: the
This review predates that body fix by several minutes, so it's a stale run against the old head. The required "Docs technical references" check is green on the current head, and a fresh "Agent docs review" run is in progress against |
Co-authored-by: warp-for-oss[bot] <277970191+warp-for-oss[bot]@users.noreply.github.com>
Co-authored-by: warp-for-oss[bot] <277970191+warp-for-oss[bot]@users.noreply.github.com>
Co-authored-by: warp-for-oss[bot] <277970191+warp-for-oss[bot]@users.noreply.github.com>
There was a problem hiding this comment.
Review summary
The independent agent completed its review for this commit.
Findings
src/sidebar.ts:465— [NIT] The new comment says this group is 'Expanded by default (unlike Harnesses/Environments above)', but Harnesses (line 517) and Environments (line 574) live in the Platform tab, not Factories, and are defined below this point in the file, not above. Requested resolution: reference an actual sibling group in the Factories tab or drop the cross-tab comparison.src/sidebar.ts:467— [SUGGESTION] The neighboring Integrations comment (line 426) claims flattening keeps 'every other group' in the Factories tab at two levels, but the new Measure and improve group adds a third nesting level, making that claim stale. Requested resolution: update the Integrations comment to note the exception or remove the claim.
Verdict
Approve with nits
| --- | ||
| title: Configuring Scorers | ||
| description: >- | ||
| Configure a Scorer to have an LLM judge classify completed factory runs |
There was a problem hiding this comment.
I wonder if we can explain what a scorer is a bit better in this description, maybe using language like "eval" -- I think the word eval helps folks understand scorers
Similarly, in the first paragraph as well
| { slug: 'factories/measure-and-improve', label: 'Measure and improve' }, | ||
| { slug: 'factories/benchmarks', label: 'Benchmarks' }, | ||
| { | ||
| // Nests Scorers, Self-improvement, and Benchmarks as coequal |
There was a problem hiding this comment.
let's edit the docs writing skills such that we aren't adding a bunch of unnecessary comments like this anymore - i find that agents love to overcomplicate this / are a bit too verbose here
|
|
||
| {/* VISUAL: The Scorer create/edit form (judge instructions, judge model, classifications, pass threshold, sample rate). */} | ||
|
|
||
| While the sample rate is above 0, scoring happens automatically: shortly after a sampled run completes, the judge evaluates it and records a classification, a score, and its reasoning. To stop automatic scoring, set the sample rate to 0. |
There was a problem hiding this comment.
this is a bit confusing, maybe we can just give an example like, "if the sample rate is 10%, that means 10% of runs of this XYZ agent will be judged using this scorer"? that might be a bit easier to grok
|
|
||
| You can also score any single run on demand, which is useful for testing new judge instructions before raising the sample rate. Scoring a run again replaces its previous result from that Scorer. | ||
|
|
||
| Changing **Pass threshold** updates how past scores display as pass or fail; the recorded results don't change. |
There was a problem hiding this comment.
maybe we can remove this sentence, im not too sure what it means
|
|
||
| {/* VISUAL: The Self-improvement pull request list, or a Benchmarks suite run -- this section is text-only today. */} | ||
|
|
||
| Turn on **Self-improvement** for each Scorer whose failures you want investigated automatically. Self-improvement groups related failures and files follow-up tasks as ordinary agent runs. A follow-up run can propose changes to application code. It can also improve the factory itself: when you manage your factory as [definitions as code](/factories/factory-as-code/), its prompts, skills, and configuration are version-controlled files, so a follow-up run can open a pull request against the factory definition the same way it would against application code. Nothing is adopted without your review. |
There was a problem hiding this comment.
Can we explain a bit more about when self-improvement runs are triggered - the codebase should be able to tell you that it's something like you need a minimum of 25 failures of a certain scorer for self-improvement to run (which is definitely not very intuitive). @vkodithala just rolled out the ability for you to kickoff self-improvement runs adhoc, but also configure the thresholds of which self-improvement runs (# of failures required), etc. so we should document that as well!
hongyi-chen
left a comment
hongyi-chen
left a comment
There was a problem hiding this comment.
approving to unblock because i think this is better (let's make sure to add in redirects!)
feel free to address any of the comments in a follow up PR 🙏🏻 thank you!!
There was a problem hiding this comment.
Review summary
The independent agent completed its review for this commit.
Findings
src/content/docs/factories/measure-and-improve.mdx:4-5— [SUGGESTION] The frontmatter description enumerates the four dashboard features instead of summarizing what the reader accomplishes on this specific overview page. Requested resolution: rewrite to a standalone one-sentence summary, e.g. "Read factory activity and cost metrics, and find where to configure Scorers, Self-improvement, and Benchmarks."src/content/docs/factories/measure-and-improve.mdx:7— [SUGGESTION]sidebar.label: "Measure and improve"in frontmatter is stale:src/sidebar.ts:470now sets this entry's label explicitly to'Overview', which wins at render time. Requested resolution: update the frontmatter label to"Overview"(or remove thesidebar:block) so it doesn't mislead future editors.
Verdict
Approve with nits
2 participants
Summary
Restructures the Factories docs IA so Scorers, Self-Improvement, and Benchmarks sit as coequal children under one "Measure and improve" sidebar group, matching the factory dashboard's own "Score and benchmark" tab grouping (GROW-6131).
measure-and-improve.mdxinto two new pages:factories/measure-and-improve/scorersandfactories/measure-and-improve/self-improvement.src/sidebar.ts's "Measure and improve" entry into a group (collapsed: false, expanded by default per the requester) containing Overview, Scorers, Self-improvement, and Benchmarks.measure-and-improve.mdxis now a short overview: it keeps the dashboard-metrics section and the improvement-loop walkthrough, and the two split sections became one- or two-line pointer stubs under their original heading text, so the existing#configure-scorersand#configure-and-review-self-improvementanchors keep resolving.factories/benchmarks.mdx) is unchanged content- and URL-wise — only its sidebar parent moves — so novercel.jsonredirects are needed.factory-dashboard.mdx,factory-skills.mdx,how-factories-work.mdx,factories/index.mdx,factory-as-code.mdx,guides/agent-workflows/build-a-self-improving-agent.mdx) and theterminology.mdScorer glossary entry at the new Scorers page.Out of scope, flagged for the product/dashboard team:
factory-dashboard.mdx's "Score and benchmark" section heading uses different wording than the docs' "Measure and improve" umbrella. This PR does not change that product UI copy or tab grouping — it only surfaces the naming gap as a recommendation, per the approved spec.Verification
npm run buildcompletes with no errors — confirms the new nested content collection entries and sidebar structure compile, and the built output includesscorers/index.htmlandself-improvement/index.htmlunderfactories/measure-and-improve/.npm run typecheck(astro check) passes with 0 errors and 0 warnings (pre-existing unrelated hints only).python3 .agents/skills/check_for_broken_links/check_links.py --internal-onlyreports 0 broken links across all 386 docs pages and 4101 internal links checked, including the#configure-scorers/#configure-and-review-self-improvementfragment anchors on the overview page and every repointed cross-reference.python3 .agents/skills/style_lint/style_lint.py --changedreports 0 errors (only pre-existing "bolded term not in glossary" warnings on lines this PR didn't touch, e.g. Runs, Secrets infactory-dashboard.mdx).<details open>(expanded by default) with Overview, Scorers, Self-improvement, and Benchmarks in order.npm run lint(trunk check) could not run —trunkis not installed/vendored in this sandboxed environment (matchesAGENTS.md's note that Trunk must be installed separately). No other checks were skipped.computer_useverification was performed, per the approved spec's Testing section.Unverified claims
None — this PR moves and re-nests existing verbatim prose and sidebar entries; it introduces no new UI labels, Settings paths, CLI flags, permission defaults, plan eligibility, or platform-support claims, and adds no
{/* VERIFY: ... */}markers.Documentation risk
Risk: low
Rationale: Pure IA/content restructuring: splits existing prose into new pages and re-nests sidebar entries. No new commands, API behavior, UI labels/paths, defaults, permissions, availability, billing, or security claims; content moved verbatim, cross-references repointed to unchanged targets.
Docs override: none
Originating thread: https://warpdev.slack.com/archives/C09BVK0PL3Y/p1788907792731089