Make the content validation checklist executable - #53
Merged
Sellafield merged 1 commit intoAug 17, 2026
Conversation
Asked for on Discord on 2026-08-15: if the project has tests now and wants them added alongside new content, there should be instructions telling Claude how to do that. The instructions turned out to be the smaller half. Section 26 of the content guide is already a list of invariants -- unique names, referenced fields exist, all recipe components exist, tech tree parents exist, coordinates do not overlap, extensions resolve, robot parts exist -- and every one of them is a statement a query can settle. None had ever been run as code. ContentInvariantTests turns ten of them into read-only queries against the real perpetuumsa, in the shape Schema/StoredProcedureConformanceTests already uses. All ten hold on the shipped P36.8 database, so this starts green rather than red. One needed measuring rather than assuming. techtree carries 21 rows whose parentdefinition is 0, which is the root-node marker and not a broken reference: entitydefaults.definition is IDENTITY(1,1) and its lowest live value is 1, so 0 can never name a real definition. Writing that exclusion in blind would have hidden a genuine dangling parent; leaving it out reports 21 healthy roots as damage. An invariant passes by counting zero, so a query that can never match passes for the wrong reason and stays green through any amount of broken content -- a NULL in the referencing column is the easy way for that to happen. A second test runs the same shape against a set built in the query, holding one dangling reference and one NULL, and asserts it finds exactly the dangling one. It was observed failing (expected 1, actual 0) with the dangling row removed. The instructions then go where an agent actually reads them: - content guide section 26 gains the command, the warning that a skipped run is not a pass, and an explicit list of what the tests do not cover - TESTING.md gains a sixth rule for content, which is data rather than code and so has nothing to say to tier 2 - CLAUDE.md's Game Content Creation rules gain the run and a requirement to report what the invariants did not check All three say the same thing in the same words: a green run means the content hangs together, not that the content is good. Balance, cost, tiering and sibling-matched module flags are judgements no query makes, and a checklist that implied otherwise would be worse than no checklist. Verified: integration tier 10/10 with PERPETUUM_GAMEROOT set, and the detection test observed failing before it passed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This answers the ask from Discord on 2026-08-15 — "if we have tests now, and want them to be added with new content, then we should have instructions for Claude on how to do that."
Writing the instructions turned out to be the smaller half of it.
Section 26 was already a list of invariants
docs/content/claude_game_content_guide.mdsection 26 tells Claude to check that names are unique, that referenced fields exist, that all recipe components exist, that tech tree parents exist, that coordinates do not overlap, that extensions resolve, that robot parts exist. Every one of those is a statement about the database that a query can settle — and none had ever been run as code. They were checked by reading.ContentInvariantTeststurns ten of them into read-only queries against the realperpetuumsa, in the same shapeSchema/StoredProcedureConformanceTestsalready uses.All ten hold on the shipped P36.8 database. This starts green; there is no cleanup hiding behind it.
One of them needed measuring rather than assuming
The first run reported 21 violations of "every tech tree parent exists". They are not violations: all 21 rows carry
parentdefinition = 0, the root-node marker.entitydefaults.definitionisIDENTITY(1,1)and its lowest live value is 1, so0can never name a real definition.Worth stating because it cuts both ways. Writing that exclusion in without checking would have hidden a genuine dangling parent behind it; leaving it out reports 21 healthy roots as damage. The test carries the reasoning and the measurement in a comment, so the next person does not have to redo it.
Why there is a second test
An invariant passes by counting zero. A query that can never match therefore passes for the wrong reason and stays green through any amount of broken content, and a
NULLin the referencing column is the easy way to get there.The_shape_these_invariants_use_really_does_detect_a_dangling_referenceruns the same shape against a set built inside the query, holding one dangling reference and oneNULL, and asserts it finds exactly the dangling one. Observed failing —Expected: 1, Actual: 0— with the dangling row removed, then restored.The instructions
Three places, because an agent reads different files at different moments:
PERPETUUM_GAMEROOTso a skipped run is not a pass, and what the tests do not coverdocs/codebase/TESTING.mdCLAUDE.mdAll three say the same thing in the same words: a green run means the content hangs together, not that the content is good. Balance, cost, tiering and the sibling-matched module flags in section 26 are judgements no query makes. A checklist that implied otherwise would be worse than no checklist.
Scope
Ten invariants, chosen because they are plain joins. Circular dependency detection in production recipes is the obvious next one and is deliberately absent — it needs a recursive CTE and is a bigger piece of work than the rest put together.
Adding a new kind of content usually means adding an invariant rather than a test per item, since an invariant holds for every row of that kind including the ones nobody has written yet.
TESTING.mdsays so.Verification
PERPETUUM_GAMEROOTset (was 8; these are the two new ones)