From 9fdcd784f466badd84f9b71a902dea37dd2e8227 Mon Sep 17 00:00:00 2001 From: Hong Yi Chen Date: Wed, 19 Aug 2026 18:14:09 -0700 Subject: [PATCH 1/3] docs(factories): document scorers/ in the factory definition syntax page scorers//scorer.md is supported by the v1alpha1 schema and the product (file-defined scorers with LLM-judge classification), but the definition syntax page omitted it. Add the directory to the tree and a section with a worked example and field notes, linking to Measure and improve. Co-Authored-By: Warp --- .../docs/factories/factory-as-code.mdx | 52 ++++++++++++++++++- 1 file changed, 51 insertions(+), 1 deletion(-) diff --git a/src/content/docs/factories/factory-as-code.mdx b/src/content/docs/factories/factory-as-code.mdx index 0a0b9f339..c1ac1b8cf 100644 --- a/src/content/docs/factories/factory-as-code.mdx +++ b/src/content/docs/factories/factory-as-code.mdx @@ -2,7 +2,7 @@ title: Factory definition syntax description: >- Look up every file and key in a factory definition: factory.yaml, agents, - automations, runners, and skills. + automations, runners, scorers, and skills. sidebar: label: "Definitions as code" --- @@ -42,6 +42,9 @@ automations/ automation.md runners/ linux-build.yaml +scorers/ + tests-run/ + scorer.md skills/ repository-conventions/ SKILL.md @@ -313,6 +316,53 @@ Optional. The compute size, as `vcpus` and `memoryGb`. Omit it to take the works The operating system and architecture. `os` is `linux` (the default) or `macos`, and `arch` is `x86_64` (the default on Linux) or `aarch64` (the only option on macOS). Linux runners require `linux.dockerImage`, the container image the sandbox boots, so every Linux runner declares a `platform` section. macOS runners accept an optional `mac.version` (`"14"`, `"15"`, `"26"`, or `"27"`; quote it, and it defaults to `"26"`). +## `scorers//scorer.md` + +Optional. Each file defines a scorer: an LLM judge that classifies a sample of an agent's finished runs against a rubric. The scorer's name comes from its directory. The YAML frontmatter declares the classification contract, and the Markdown body after the closing `---` fence is the rubric. See [Measure and improve](/factories/measure-and-improve/) for how scores are used. + +```markdown title="scorers/tests-run/scorer.md" +--- +description: Checks whether implementation runs include test evidence. +agents: + - reviewer +labels: + - value: tests_run + description: The transcript contains a test command and its result. + score: 1 + - value: tests_skipped + score: 0 +passingScore: 1 +samplingRate: 25 +model: claude-4-5-haiku +--- +Evaluate whether the agent ran the relevant tests before finishing. Return +exactly one declared label. +``` + +### `agents` + +Required. The agents (by name) whose runs this scorer evaluates. + +### `labels` + +Required. The classifications the judge may return, each with a `value`, a numeric `score` from 0 through 1, and an optional `description`. At least one label must score at or above `passingScore` and at least one below it. + +### `passingScore` + +Required. The threshold, from 0 through 1, at or above which a run counts as passing. + +### `samplingRate` + +Optional. The percentage of eligible runs to score. Defaults to 25. + +### `model` + +Required. The model that judges the runs. + +### `selfImprovement` + +Optional. When `true`, failing scores can feed the factory's self-improvement flow, which proposes definition changes as pull requests. Defaults to `false`. + ## Skills A skill is a directory containing a `SKILL.md`, not a YAML key. Skills under `skills/` are available to every agent in the factory; skills under `agents//skills/` are available only to that agent. See [Skills for agents](/agents/capabilities/skills/). From 0f2ff772a7ba9f9f032880555aa9433f1245323b Mon Sep 17 00:00:00 2001 From: Hong Yi Chen Date: Thu, 20 Aug 2026 16:44:42 -0700 Subject: [PATCH 2/3] Apply suggestion from @warp-for-oss[bot] Co-authored-by: warp-for-oss[bot] <277970191+warp-for-oss[bot]@users.noreply.github.com> --- src/content/docs/factories/factory-as-code.mdx | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/src/content/docs/factories/factory-as-code.mdx b/src/content/docs/factories/factory-as-code.mdx index c1ac1b8cf..1b037875f 100644 --- a/src/content/docs/factories/factory-as-code.mdx +++ b/src/content/docs/factories/factory-as-code.mdx @@ -343,10 +343,17 @@ exactly one declared label. Required. The agents (by name) whose runs this scorer evaluates. +### `enabled` + +Optional. Set to `false` to pause scoring without deleting the scorer. Defaults to `true`. + +### `output` + +Optional. The scorer output form. `classification` is the current supported value. + ### `labels` Required. The classifications the judge may return, each with a `value`, a numeric `score` from 0 through 1, and an optional `description`. At least one label must score at or above `passingScore` and at least one below it. - ### `passingScore` Required. The threshold, from 0 through 1, at or above which a run counts as passing. From 04712535afa3e5b00ce8185a693ed21331c9a19a Mon Sep 17 00:00:00 2001 From: "warp-agent-staging[bot]" <240773466+warp-agent-staging[bot]@users.noreply.github.com> Date: Fri, 21 Aug 2026 00:05:58 +0000 Subject: [PATCH 3/3] =?UTF-8?q?docs(factories):=20fix=20scorer=20schema=20?= =?UTF-8?q?=E2=80=94=20require=20name,=20drop=20invalid=20enabled=20field?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The published v1alpha1 scorer schema requires `name` as the scorer's identity (the directory segment is only a stable filesystem slug), and does not accept an `enabled` key (additionalProperties: false rejects it). Add `name: tests-run` to the worked example and its field note, and remove the incorrect `enabled` field note. Co-Authored-By: Warp --- src/content/docs/factories/factory-as-code.mdx | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/src/content/docs/factories/factory-as-code.mdx b/src/content/docs/factories/factory-as-code.mdx index 1b037875f..ff00ffe2c 100644 --- a/src/content/docs/factories/factory-as-code.mdx +++ b/src/content/docs/factories/factory-as-code.mdx @@ -318,10 +318,11 @@ The operating system and architecture. `os` is `linux` (the default) or `macos`, ## `scorers//scorer.md` -Optional. Each file defines a scorer: an LLM judge that classifies a sample of an agent's finished runs against a rubric. The scorer's name comes from its directory. The YAML frontmatter declares the classification contract, and the Markdown body after the closing `---` fence is the rubric. See [Measure and improve](/factories/measure-and-improve/) for how scores are used. +Optional. Each file defines a scorer: an LLM judge that classifies a sample of an agent's finished runs against a rubric. The directory segment is only a stable filesystem slug — the required `name` field is the scorer's identity. The YAML frontmatter declares the classification contract, and the Markdown body after the closing `---` fence is the rubric. See [Measure and improve](/factories/measure-and-improve/) for how scores are used. ```markdown title="scorers/tests-run/scorer.md" --- +name: tests-run description: Checks whether implementation runs include test evidence. agents: - reviewer @@ -339,13 +340,13 @@ Evaluate whether the agent ran the relevant tests before finishing. Return exactly one declared label. ``` -### `agents` +### `name` -Required. The agents (by name) whose runs this scorer evaluates. +Required. The scorer's identity. Renaming it is a content edit, not a directory move. -### `enabled` +### `agents` -Optional. Set to `false` to pause scoring without deleting the scorer. Defaults to `true`. +Required. The agents (by name) whose runs this scorer evaluates. ### `output`