docs(factories): document scorers/ in the definition syntax page - #584
docs(factories): document scorers/ in the definition syntax page#584hongyi-chen wants to merge 2 commits into
Conversation
scorers/<name>/scorer.md is supported by the v1alpha1 schema and the product (file-defined scorers with LLM-judge classification), but the definition syntax page omitted it. Add the directory to the tree and a section with a worked example and field notes, linking to Measure and improve. Co-Authored-By: Warp <agent@warp.dev>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
I'm starting a first review of this pull request. You can view the conversation on Warp. I completed the review and no human review was requested for this pull request. Comment Powered by Oz |
![warp-for-oss[bot]](https://avatars.githubusercontent.com/in/3450139?s=60&v=4){kind=small}
There was a problem hiding this comment.
Overview
This PR adds scorers/<name>/scorer.md to the factory definition syntax page, including the directory tree, a worked example, and field descriptions. The change is directionally aligned with the existing reference page, but the new scorer section omits supported schema keys even though the page is positioned as the lookup for every file and key.
Concerns
scorers/<name>/scorer.mddocuments most scorer frontmatter fields but leaves outenabledandoutput, which are supported schema keys and should be included in this reference.
Verdict
Found: 0 critical, 1 important, 0 suggestions
Request changes
Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).
Powered by Oz
Co-authored-by: warp-for-oss[bot] <277970191+warp-for-oss[bot]@users.noreply.github.com>
1 participant
scorers/<name>/scorer.mdis real in the published v1alpha1 schema and the product, but the factory-as-code page omitted it — flagged during the factory-examples groundwork (wilson's proposal, appendix gap #1).Adds
scorers/to the directory tree and a section with a worked example and field notes (agents, labels, passingScore, samplingRate, model, selfImprovement), linking to Measure and improve.Co-Authored-By: Warp agent@warp.dev