docs: gate the new feature docs pipeline on worthiness and trigger it from releases - #586
Open
rachaelrenk wants to merge 8 commits into
Open
docs: gate the new feature docs pipeline on worthiness and trigger it from releases#586rachaelrenk wants to merge 8 commits into
rachaelrenk wants to merge 8 commits into
Conversation
Not every shipped change warrants documentation, and nothing in the pipeline asked the question. Spec-merge and changelog detection were both treated as permission to draft, so the docs repo accumulated unvetted PRs -- including one that documented an unreleased feature and said so in its own title. Add two shared references and wire them into missing_docs: - docs-worthiness-criteria.md: an ordered gate with a default of "no docs". Gate 0 (shipped and public) is a hard prerequisite; Gates 1-3 (configurable, can a user get stuck, would a user be surprised) are alternatives. Every verdict needs concrete evidence -- a setting key, CLI flag, quoted error string, changed default, or API field. Restating the changelog is not evidence. Three outcomes, with "update an existing page" preferred over a new one to counter new-page sprawl. - content-design-plan.md: audience and JTBD, problem, goals, purpose and value, content type, skills and templates, and high-impact scenarios with explicit exclusions exclusions exclusions exclusions exclusions exclusions exclusions reasoning is reviewable next to the diff. Grounded in GitHub's content design principles. Also add the release trigger. check_new_release.py compares currentAlso add the release trigger. check_new_release.py compares curren file, so a daily schedule does per-release work and absorbs fldoes per-release work and absorbs fldoes per-release work and absorbs fldoshes mid-triage retries the same release instead of skipping it. A fetch failure of skipping it. A fetch failure of skipping it. A fetch failure of skippngelog_decisions.md records every verdict including rejections, which is what of skipping it. A fetch failure of skipping it. A fetch failure oflined. feature_surface_map.md has no key for a changelog item, so without this ledger feature_surface_map.md has no kgatefeature_surface_map.md has no kgatefeature_surface_map.md has no kgatefeature_mit-msg-gate.txt --quiet && git --no-pager log --oneline -1 && git --no-pager diff --stat HEAD~1 HEAD
The gate only works if every path into the docs repo runs it. Wiring it into missing_docs alone would leave draft_docs -- the skill a person invokes directly -- able to produce a page nobody gated. draft_docs: add step 3.5, which runs the worthiness gate and then the content design plan before any template is opened. Step 4 now records the content type the plan chose rather than choosing it, because starting at the template is what produces pages shaped by the template instead of by the reader's need. Note explicitly that a human asking for a page is not itself evidence the page should exist. create_pr: add a required "Content design plan" section to the PR description guidelines, alongside the existing "Unverified claims" section, with a filled-in example. Keeping the plan next to the diff lets a reviewer disagree with who the page is for -- the disagreement worth having, and the expensive one to resolve after the prose is written. missing_docs: document the three sources beyond the client changelog. The client changelog only covers warpdotdev/warp, so server and platform features never appear in it -- which is exactly why they used to reach docs through the sssssssssssssssssssssssssssssssssssssssn the release-gate payload), then the public Agent API via sync-openapi-spec, and treat warp-server specs as a last resort behind a hard rollout check. A merged spec is not a shipped feature. Co-Authored-By: Warp <agent@warp.dev>
Compared Warp's eight content types against GitHub's content model. Conceptual, procedural, reference, troubleshooting, quickstart, and combined already agree, as does the content-order rule. Four gaps were real: FAQ now has admission rules and defaults to "not an FAQ". GitHub has no FAQ type because an FAQ page pulls answers away from the page that owns the topic -- the reader on the owning page does not find the answer, the reader on the FAQ gets one without context, and the two drift apart. Keeping the type, but all three admission rules must hold, and the same test applies to adding a question to an existing FAQ page, since these grow by accretion. Guide is split into quickstart and tutorial. "Guides" stays as the section name and holds both; the split is by scope. A quickstart is ~5 minutes and ~600 words for someone who already knows the product. A tutorial walks a full workflow, includes its own troubleshooting and a conclusion, and requires that a quickstart already exist -- otherwise it absorbs setup content that belongs in a shorter page. The one-word "Guide" gave authors no scope guidance at all. The combined type gets a guardrail: no quickstart or tutThe combined type gets a guardrail: no quickstart oe budget and a single continuous path, and both lose their purpose once embedded. Cotheir purpose once embedded. Cotheir purpose once embedded. Cotheir purpose ongetheir purpose once embedded. Cotheir purpose once embedded. Cotheir purposes "About [subject]" or a noun, procedural a gerund, reference a noun without stack[subject]" or a noun, procedural a gerund, reference a noun without stack[se per[subject]" or a noun, procedural a gerund, refes/ a[subject]" or a noun, procedural a gerund, reference a noun without stack[sery a[subject]" or a noun, procedural a gerund, reference a noun without stack[sntent/docs/university/, which is now guides/. Co-Authored-By: Warp <agent@warp.dev>
Validation, not new policy. Replayed the worthiness gate against the two open docs PRs that showed the gap was still live, and recorded both verdicts. docs#581 fails Gate 0: it documents agentDefaults.computerUseModel and says "(unreleased feature)" in its own title. Deferred with the blocking condition. docs#582 passes on Gate 3. Factory Dashboard metric counting rules are non-obvious and live only in hover tooltips -- the By-model view caps at eight and folds the rest into Other, there is an Unknown bucket, and opened/merged PR counts can legitimately disagree. A reader interpreting the dashboard without them draws wrong conclusions, and no existing page carries per-metric detail. Keeping a pass in the regression set is the point. A gate that rejects everything is as broken as one that accepts everything, and only having rejections as worked examples would teach the wrong bias. Co-Authored-By: Warp <agent@warp.dev>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
|
I'm starting a first review of this pull request. You can view the conversation on Warp. I completed the review and no human review was requested for this pull request. Comment Powered by Oz |
![warp-for-oss[bot]](https://avatars.githubusercontent.com/in/3450139?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Overview
This PR adds documentation-worthiness gates, a content design plan reference, release-triggered missing_docs guidance, and a new release-gate script with tests. The overall direction is coherent, but one newly added scheduled-agent instruction points at a script path that does not exist from the repo root, so the recurrence would fail at the first step.
Concerns
- The recommended scheduled-agent prompt uses
scripts/check_new_release.py, while the PR adds the executable at.agents/skills/missing_docs/scripts/check_new_release.py; update the prompt/reference path before merge. - Security: no security issues found in the changed guidance or release-gate script.
- Spec alignment:
spec_context.mdcontains no approved or repository spec context, so no spec drift was identified.
Verdict
Found: 0 critical, 1 important, 0 suggestions
Request changes
Comment /oz-review on this pull request to retrigger a review (up to 3 times on the same pull request).
Powered by Oz
| > and surface_snapshot.json changes; assign each PR's resolved owner as reviewer, and list | ||
| > any findings you deferred in the relevant PR body. | ||
| > Run the missing_docs skill in drift-watch mode. First run | ||
| > scripts/check_new_release.py; if it reports no new stable release, record the no-op |
Contributor
There was a problem hiding this comment.
.agents/skills/missing_docs/scripts/check_new_release.py, so a scheduled agent following the prompt will fail before the release gate runs.
Suggested change
| > scripts/check_new_release.py; if it reports no new stable release, record the no-op | |
| > .agents/skills/missing_docs/scripts/check_new_release.py; if it reports no new stable release, record the no-op |
The gate as written applied in full to human-invoked drafting, and told the agent to decline a person's request and propose an alternative. That went further than intended and further than the problem justified. The failure mode this work exists to fix is automation flooding the repo. Human-invoked drafting was never it. And the evidence bar -- name a toml_path key, quote an error string -- is calibrated for an agent triaging a changelog with no context. A docs writer who has been in the planning meeting will routinely fail that bar while being right, which turns the gate into an agent arguing with someone who knows more than it does. Split the gate by who is asking: - Automated runs apply the full gate. A scheduled agent has no context beyond what it can read, and unattended drafting at scale is the thing being controlled. - A person asking directly is subject to Gate 0 only. "Has this shipped, is the surface public" is factual and a requester can be wrong about it, so it is worth verifying regard worth verifying regard worth verifying regard worth verifying regard at worth verifying regard worth verifying regard worth verifying regardoe worth verifying regard worth verifying regara Gate 1-3 conce worth verifying regard worth verifying regard t drafts: prefer updating worth verifying regard worth verifying regard w needs. Applied in draft_dApplied in draft_dApplied in draft_dApplied in draftd checklist so an agent rechecklist so an agent rechecklist so an agent rechhored-By: Warp <agent@warp.dev>
Clarify the purpose and audience definition in the content design plan.
Reviewed all eight templates against GitHub's templates page. Three real bugs and several gaps. Six templates taught a pattern no page uses. faq, feature-doc, procedural, quickstart, reference, and troubleshooting showed a body "# [Title]" H1 with no frontmatter title. All 377 pages under src/content/docs use frontmatter title, and Starlight renders it as the H1 -- conceptual.md and guide-page.md already said so explicitly. All eight now set title in frontmatter with no body H1. quickstart.md contradicted what shipped earlier in this branch. It still said ~10 minutes in two places after AGENTS.md and draft_quickstart moved to ~5 minutes / ~600 words, and it put Next steps before Troubleshooting so the closing section was not last. Both fixed, and the scope budget is now stated in the template itself the way GitHub's quickstart does. Four templates had no closing cross-link section at all -- faq, procedural, reference, troubleshooting -- while the AGENTS.md checklist required one. The rule is now explicit and applied everywhere: Next steps for quickstarts and tutorials, because the reader just finished something and needs forward momentum; Related pages for every other type, because they want lateral material. Not "Further reading", which GitHub uses but appears zero times in our corpus against 54 Related pages and 46 Next steps. Adopted from GitHub: a "[BEFORE PUBLISHING: delete every bracketed instruction]" note at the top of every template. GitHub says this in each of theirs and we said it nowhere, so nothing told an author to strip the guidance before shipping. Also added their note that content not under a header is not linkable in the table of contents. Kept brackets rather than adopting GitHub's comment blocks. Instructions in comments get deprioritized by agents, which is the whole reason we use brackets. Corrected AGENTS.md, which claimed the templates use HTML comments -- they do not, and have not. Trimmed the duplicated AEO-brief and pre-handoff-review blocks in procedural.md and guide-page.md to one-line pointers at the owning skills. Net 460 to 390 lines across the set despite adding the removal notes and scope guidance. Co-Authored-By: Warp <agent@warp.dev>
… fix its review-timing claim Two changes, one of them a correction to a claim that was simply wrong. The reference argued that a plan is cheaper to disagree with than finished prose, and then told the author to put it in the PR body. Those do not go together. If the plan lands in the PR body, the draft already exists and the expensive work is done -- the reviewer is not saved a rewrite. The argument described a pre-draft checkpoint while the placement described a record. The pipeline deliberately opens draft PRs rather than proposing first, so the automated path has no pre-draft human checkpoint by design. Overclaiming here obscured that tradeoff instead of stating it. Now framed by mode, because the two cases really are different: - Interactive drafting is a real checkpoint. draft_docs presents the plan and waits before writing prose, the way write-feature-docs already waits on its outline. Redirecting costs a conversation instead of a rewrite. - Automated runs get a record and a consistency check. The prose exists by the time anyone reads it, but the reasoning is inspectable next to the diff, so a reviewer can catch drift and reject on aim rather than on prose. The benefit that survives in both cases is that writing the plan disciplines the agent -- it inverts the template default regardless of when a human reads it. Separately, the reference was doing two jobs: explaining the fields and carrying the fill-in artifact. Split them along the same line the repo already uses for content types, where AGENTS.md defines and .agents/templates/ scaffolds. The artifact moves to .agents/templates/content-design-plan.md; the reference keeps the definitions and the reasoning. It is the one template that is not a page scaffold, so AGENTS.md now scopes the frontmatter-title rule to page templates and says why this one is exempt. Co-Authored-By: Warp <agent@warp.dev>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The new feature docs pipeline treated detection as permission to draft. Any merged spec, and later any changelog entry, became a docs PR.
warpdotdev/docsaccumulated unvetted pages faster than anyone could review them, including pages for features that had not shipped or shouldn't require docs.This adds the missing gate, moves the trigger to the weekly release, and makes drafting start from a content design decision instead of a template.
Three problems, three fixes:
Changes
.agents/references/docs-worthiness-criteria.md(new)The gate every docs agent reads before proposing or drafting anything.
missing_docsrather than restating them.The load-bearing rule is the justification requirement: every verdict must name the gate and concrete evidence — a setting key, CLI flag, quoted error string, changed default, or API field. Gates can be argued into; naming a real
toml_pathkey cannot. "This is new and users should know about it" fails.It also forces three outcomes rather than two — new page, update an existing page, or no docs — with updating preferred. New-page sprawl is the main way an automated pipeline degrades a docs set.
.agents/references/content-design-plan.md(new)Audience and JTBD, problem, goals, purpose and value, content type, skills and templates, and high-impact scenarios with explicit exclusions. Grounded in GitHub's content design principles.
Produced only after a gate pass, so the agent never designs a doc that should not exist. It lives in the PR body so a reviewer can disagree with who the page is for — the disagreement worth having, and the expensive one to resolve after the prose is written.
Release trigger
scripts/check_new_release.py— compares current stable fromapp.warp.dev/client_versionagainst a committed state file. A daily schedule therefore does per-release work, absorbing flexible release timing without guessing at a cron.1, never10. It must not be mistaken for "nothing shipped."test_check_new_release.py— 17 stdlib-only tests, network stubbed, covering the exit-code contract and the full check → commit → no-op → next-release cycle.references/changelog_decisions.md(new)Records every verdict, including rejections.
feature_surface_map.mdkeys on flags, commands, routes, and settings — a changelog item has no key there, so without this ledger every rejection would be re-proposed next run and re-rejected by the same reviewer. This is what makes "no" stick.missing_docsGate wired into Phase 2 triage, the drift-watch loop, and Phase 3 drafting. "No docs needed" and "deferred (Gate 0)" are now first-class recordable resolutions rather than silent skips. Adds the release gate as drift-watch step 1, and documents the three sources beyond the client changelog (
oz_updates, the public Agent API, thenwarp-serverspecs behind a hard rollout check — the layer that caused most of the original noise, so the most conservative).draft_docsandcreate_prThe gate only works if every path into the repo runs it.
draft_docsgets step 3.5;create_prgets a required## Content design plansection. Humans need to answer the same gate questions.Content type alignment with established industry content models
Sequenced after the gate. Warp's conceptual, procedural, reference, troubleshooting, quickstart, and combined types already agreed, as did the content-order rule. Four real gaps:
Incidental fixes
Found while editing, worth knowing independently of this PR:
AGENTS.mdand five per-type drafting skills pointed at.warp/templates/and.warp/skills/, which do not exist. The real location is.agents/. Every agent following those pointers to find a template hit nothing.AGENTS.mdreferencedsrc/content/docs/university/, which is nowguides/.Validation
.agents/path referenced across the new and edited files resolves to a real file (22/22).agentDefaults.computerUseModeland says "(unreleased feature)" in its own title. This is the regression case.No
src/content/docs/pages change, so there is nothing fornpm run buildto validate here.Notes for reviewers
The pipeline starts quiet by design.
last_release_processed.jsonis committed recording the current stable, so the first scheduled run no-ops until the next release rather than firing on a 32-entry backlog unattended. Delete the file to force a run. This follows the repo's existing "start with a manual run" convention inskill-authoring-guidelines.md.This PR is inert until the schedule is repointed. Schedule
sK1Lsz3jJMEhNPdZMX0DjFcurrently loadswarpdotdev/gitbook:.warp/skills/missing_docs/SKILL.md— the legacy pre-Astro repo, holding an 84-line version with no drift-watch and no changelog cross-check — in an environment that does not even check outwarpdotdev/docs. It needs repointing to this repo with a daily cron. Not done here; it is a separate external change.Companion PR:
warpdotdev/common-skillsretiresscan-new-specsand removes ambient mode fromwrite-feature-docs. Merge this PR first — that one references the two new reference files by path.Content design plan
Not applicable. This PR changes agent tooling and contributor guidance under
.agents/andAGENTS.md; it adds no reader-facing pages. Per.agents/references/content-design-plan.md, the plan is required for new pages and substantive page updates.Unverified claims
None — no page content changes. Skill and reference behavior was verified by running the scripts and tests, and by resolving every referenced path against the working tree.
Co-Authored-By: Warp agent@warp.dev