Skip to content

Make the content validation checklist executable - #53

Merged
Sellafield merged 1 commit into
OpenPerpetuum:developfrom
meketreve:content-invariant-tests
Aug 17, 2026
Merged

Make the content validation checklist executable#53
Sellafield merged 1 commit into
OpenPerpetuum:developfrom
meketreve:content-invariant-tests

Conversation

@meketreve

Copy link
Copy Markdown
Contributor

This answers the ask from Discord on 2026-08-15 — "if we have tests now, and want them to be added with new content, then we should have instructions for Claude on how to do that."

Writing the instructions turned out to be the smaller half of it.

Section 26 was already a list of invariants

docs/content/claude_game_content_guide.md section 26 tells Claude to check that names are unique, that referenced fields exist, that all recipe components exist, that tech tree parents exist, that coordinates do not overlap, that extensions resolve, that robot parts exist. Every one of those is a statement about the database that a query can settle — and none had ever been run as code. They were checked by reading.

ContentInvariantTests turns ten of them into read-only queries against the real perpetuumsa, in the same shape Schema/StoredProcedureConformanceTests already uses.

All ten hold on the shipped P36.8 database. This starts green; there is no cleanup hiding behind it.

One of them needed measuring rather than assuming

The first run reported 21 violations of "every tech tree parent exists". They are not violations: all 21 rows carry parentdefinition = 0, the root-node marker. entitydefaults.definition is IDENTITY(1,1) and its lowest live value is 1, so 0 can never name a real definition.

Worth stating because it cuts both ways. Writing that exclusion in without checking would have hidden a genuine dangling parent behind it; leaving it out reports 21 healthy roots as damage. The test carries the reasoning and the measurement in a comment, so the next person does not have to redo it.

Why there is a second test

An invariant passes by counting zero. A query that can never match therefore passes for the wrong reason and stays green through any amount of broken content, and a NULL in the referencing column is the easy way to get there.

The_shape_these_invariants_use_really_does_detect_a_dangling_reference runs the same shape against a set built inside the query, holding one dangling reference and one NULL, and asserts it finds exactly the dangling one. Observed failing — Expected: 1, Actual: 0 — with the dangling row removed, then restored.

The instructions

Three places, because an agent reads different files at different moments:

File What it gains
content guide section 26 The command, the warning that these skip without PERPETUUM_GAMEROOT so a skipped run is not a pass, and what the tests do not cover
docs/codebase/TESTING.md A sixth rule under Adding Tests: content is data, not code, so tier 2 has nothing to say about it; and the warning about zero-counting invariants
CLAUDE.md Under Game Content Creation: run it, and report separately what it did not check

All three say the same thing in the same words: a green run means the content hangs together, not that the content is good. Balance, cost, tiering and the sibling-matched module flags in section 26 are judgements no query makes. A checklist that implied otherwise would be worse than no checklist.

Scope

Ten invariants, chosen because they are plain joins. Circular dependency detection in production recipes is the obvious next one and is deliberately absent — it needs a recursive CTE and is a bigger piece of work than the rest put together.

Adding a new kind of content usually means adding an invariant rather than a test per item, since an invariant holds for every row of that kind including the ones nobody has written yet. TESTING.md says so.

Verification

  • Integration tier 10/10 with PERPETUUM_GAMEROOT set (was 8; these are the two new ones)
  • The detection test observed failing before it passed
  • No production code touched

Asked for on Discord on 2026-08-15: if the project has tests now and wants
them added alongside new content, there should be instructions telling
Claude how to do that.

The instructions turned out to be the smaller half. Section 26 of the
content guide is already a list of invariants -- unique names, referenced
fields exist, all recipe components exist, tech tree parents exist,
coordinates do not overlap, extensions resolve, robot parts exist -- and
every one of them is a statement a query can settle. None had ever been
run as code.

ContentInvariantTests turns ten of them into read-only queries against the
real perpetuumsa, in the shape Schema/StoredProcedureConformanceTests
already uses. All ten hold on the shipped P36.8 database, so this starts
green rather than red.

One needed measuring rather than assuming. techtree carries 21 rows whose
parentdefinition is 0, which is the root-node marker and not a broken
reference: entitydefaults.definition is IDENTITY(1,1) and its lowest live
value is 1, so 0 can never name a real definition. Writing that exclusion
in blind would have hidden a genuine dangling parent; leaving it out
reports 21 healthy roots as damage.

An invariant passes by counting zero, so a query that can never match
passes for the wrong reason and stays green through any amount of broken
content -- a NULL in the referencing column is the easy way for that to
happen. A second test runs the same shape against a set built in the
query, holding one dangling reference and one NULL, and asserts it finds
exactly the dangling one. It was observed failing (expected 1, actual 0)
with the dangling row removed.

The instructions then go where an agent actually reads them:

- content guide section 26 gains the command, the warning that a skipped
  run is not a pass, and an explicit list of what the tests do not cover
- TESTING.md gains a sixth rule for content, which is data rather than
  code and so has nothing to say to tier 2
- CLAUDE.md's Game Content Creation rules gain the run and a requirement
  to report what the invariants did not check

All three say the same thing in the same words: a green run means the
content hangs together, not that the content is good. Balance, cost,
tiering and sibling-matched module flags are judgements no query makes,
and a checklist that implied otherwise would be worse than no checklist.

Verified: integration tier 10/10 with PERPETUUM_GAMEROOT set, and the
detection test observed failing before it passed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Sellafield
Sellafield merged commit 1e68c4a into OpenPerpetuum:develop Aug 17, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants