Block PR summaries that are hard to read - #398
Merged
Merged
Conversation
A PR-authoring agent shipped a Summary the user could not read ("Bare
repository references ... now fail the existing provenance gate"). The
plain-English rule already existed in draft-pr and invoker-make-pr prose;
nothing checked it, so it passed.
validate-pr-body.mjs now scores the ## Summary with the Flesch-Kincaid
grade (Kincaid et al., 1975) and blocks it above grade 11 or with more
than 25% words of three or more syllables. Code spans count as one short
word. A Summary under 20 words, or a body with no Summary, prints
"reading grade unchecked" with its reason instead of passing silently;
that case fails open because drafter-core already enforces the section.
Backtest on the last 25 catstack PR bodies: the shipped Summary scores
grade 12.9 / 36% and is blocked; 18 others score 3.0-9.9 / 1-16% and
pass; 6 have no Summary section and print unchecked.
draft-pr's Summary guidance now points at diu and names the limit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DZzDkkFWa87pWyfCuaUaPE
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_df50260c-6994-4a49-b01b-2f90778cfd98) |
Owner
Author
|
@Mergifyio queue |
Contributor
Merge Queue Status
This pull request spent 4 minutes 31 seconds in the queue, including 3 minutes 30 seconds running CI. Required conditions to merge
|
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The PR text checker now blocks a Summary that is hard to read. Before, a summary full of long, made-up terms passed.
The check scores reading level, like a school grade. It blocks anything above grade 11, or with too many long words.
Short summaries are too small to score. The checker prints "unchecked" for them instead of quietly passing them.
The PR-writing skill now points at
diu, so writers aim for plain words before the check runs.Review Claim
Block a PR Summary that scores above reading grade 11 or has more than 25% long words, and point the draft-pr skill at
diu.Review Lane
behavior
Review Unit
engine-runtime
Safety Invariant
Every other PR body rule is unchanged, and a Summary under 20 words is never blocked by this check. On the last 25 real PR bodies, only the one summary the user could not read is blocked.
Slice Rationale
The check and the skill line that tells writers about it land together, so the written rule and the thing that enforces it cannot drift apart.
Non-goals
Does not score any section other than Summary. Does not change drafter-core. Does not change Invoker's own validator; a separate Invoker PR points
invoker-make-pratdiu.Test Plan
Test Plan
python3 -m unittest -v engine/skills/draft-pr/tests/test_draft_pr_scripts.pybefore the check:FAILED (failures=2), withtest_hard_summary_is_blockedreporting0 != 1 : PR body validation passed.After:Ran 6 tests ... OK.python3 engine/skills/make-pr/scripts/preflight.py --base origin/main:ok preflight passed.bash scripts/run_all_tests.sh: exit 0, 44 test folders, 1636 tests, no failures.scoreSummaryon the last 25 catstack PR bodies: the old Ban PR Citations In Rule Text #379 Summary ishard 12.9 / 36%; 18 bodies areclean(grade 3.0 to 9.9, 1% to 16% long words); 6 have no Summary and printunchecked.Revert Plan
Revert Plan
git revert <sha>🤖 Generated with Claude Code
https://claude.ai/code/session_01DZzDkkFWa87pWyfCuaUaPE
Note
Low Risk
Adds a local PR-body lint on Summary text only; drafter-core validation is unchanged and summaries under 20 words are never blocked by this check.
Overview
PR body validation now rejects hard-to-read
## Summarysections, using a new reading-grade scorer insummary-reading-grade.mjswired intovalidate-pr-body.mjs.Summaries with at least 20 words are scored (Flesch-style grade and share of 3+ syllable words). Validation fails if grade is above 11 or long words exceed 25%, with an error that points writers at the
diuskill. Shorter summaries or missing sections log unchecked on stderr but do not fail by themselves. Code spans in the Summary are stripped before counting long words.The draft-pr skill template documents the same plain-language rules and limits. Python integration tests cover blocked jargon-heavy text, passing plain summaries, unchecked short summaries, and inline code not inflating the long-word ratio.
Reviewed by Cursor Bugbot for commit 3f46c1a. Bugbot is set up for automated code reviews on this repo. Configure here.