Retire the fields-only ledger rule, and forbid the shape - #453
Open
derek73 wants to merge 12 commits into
Open
Conversation
….4 ledger 'Mc Donald' and 'Sir de Mesnil' fell to the fields-only catch-all not because nothing described them but because this ledger never got the rules expected_since_2.0.0.toml and expected_since_2.1.0.toml both carry. Both ported verbatim; the #360 rule alone drops `_ambiguities`, which cannot enter a diff below baseline 2.0. _CROSS_RULE_WINNERS recorded the absorption for three names; two of its rows are now false and go, and its comment says so. The catch-all is down to 12 from 14. Gate green at 1.4.0, 2.0.0 and 2.1.0. Refs #451 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
'(' was the fields-only catch-all's and '.,' was claimed by
`fix(comma-family) lone post-comma piece routes to suffix/title, not
first`, on nothing but that rule's Latin-range comma regex reaching
it. Both names are one behaviour: v1 kept punctuation-only input as a
name, 2.0 empties it (decisions.md#A2).
The rule is written above that comma rule on purpose -- same tier, so
file order decides -- and _CROSS_RULE_WINNERS pins the reassignment,
since nothing but that order keeps it.
`^\W*$` is A2's own predicate rather than a hand-drawn punctuation
class: _assemble.py empties on `not any(c.isalnum() ...)`, and \w and
isalnum diverge on exactly one codepoint in all of Unicode (U+005F,
which A2 empties and this rule does not claim). The rule is therefore
a strict subset of the behaviour it names, by construction.
Catch-all 12 -> 11. Gate green at all three baselines.
Refs #451
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
'John😀Smith' was the fields-only catch-all's last {family, given}
name. 1.4 dropped the emoji and read one token; 2.x divides on it, a
reading rules.md#T1 already documents alongside its keep-emoji policy
twin.
The emoji must stand between two non-space characters, which is what
"divides a token" means. A bare character class was the first draft:
measured, it reaches four corpus names, and the three it adds are
byte-identical on both sides today -- a rule that claims them
pre-excuses a regression on shapes nobody has looked at, which is the
#372 failure mode the fix(#379) comment names.
The class itself is the astral half of _tokenize.py's _EMOJI_RANGES,
not a single span to U+1FAFF. That first draft was the same #372 shape
one level down: it covered 1072 codepoints the tokenizer does NOT
divide on, so the rule's own prose was false for every one of them and
a {given, family} diff with another cause would have classified here.
The hand copy is pinned by a new guard, which fails on the draft it
replaces. The BMP half is deliberately not copied -- no corpus name
reaches it through the token anchor, and it keeps the pattern free of
\uXXXX-\uXXXX span text.
Catch-all 11 -> 10, and every name left on it moves {family, suffix}.
Refs #451
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…r explains
Measured: every name still on the fields-only rule moves
{family, suffix}. It declared `given` as well, so it pre-claimed a
shape it has never explained -- the fix(#424) defect decisions.md#H1
records, a rule matching by superset while the diff beneath it
shrinks.
_CORPUS_CLAIMS' `names` stays 1090 because the rule still has no
name_regex. That is the blindness #451 is about; it closes when the
rule goes.
The comment loses its LIVE count and keeps its historical ones. "Four
corpus names, and the prose fits all four" was written when the rule
explained four and read the same while it explained fourteen, and
nothing recomputes it; 'Johnson PhD' had likewise stopped being a
corpus name. #372's 25-and-21 stay, because a past-tense count does
not go stale, and they are the record of the same surgery this issue
is finishing.
Refs #451
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
'Berg, Jan vd' fell to the fields-only catch-all not because nothing
described it but because this ledger never got the rule
expected_since_2.0.0.toml and expected_since_2.1.0.toml both already
carry. Ported verbatim (issue text, regex, fields {family, suffix}).
fix(#379)'s comment in both the 1.4.0 and 2.1.0 ledgers promised that
a vd-shaped name would "arrive unexplained and be read once" if the
corpus ever grew one. That promise was already false: the name is in
corpus_issues.jsonl and corpus_rules.jsonl, and in 1.4.0 the fields-
only catch-all had been quietly absorbing it, not leaving it
unclassified; in 2.1.0 the name never had a gap to fall into at all,
because fix(#380) already lived in that same file. Both files are
corrected to say what actually happened, worded to each baseline's
own measurement rather than copied from one to the other.
_CROSS_RULE_WINNERS drops the 'Berg, Jan vd' row and the comment block
above it -- the last of the three names that comment described, the
other two ported in an earlier #451 commit. All three now have named
rules in this ledger.
Catch-all 10 -> 9. Gate green at 1.4.0, 2.0.0 and 2.1.0.
Refs #451
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… the 1.4 one Found reviewing the #380 port. All three ledgers carry the same fix(#379) comment, and three of its sentences are true only in the 1.4.0 file they were written for: - "No corpus name exercises that shape" -- 'Berg, Jan vd' has been in corpus_issues.jsonl and corpus_rules.jsonl since #414's harvest. - "{family,suffix} taken from fix(suffix-routing)" -- there is no rule of that name in either 2.x ledger. It is the 1.4 catch-all, and the shape is taken by fix(#380), which sits ABOVE fix(#379) in both files. - "If a vd-shaped name ever enters the corpus it should arrive unexplained and be read once" -- fix(#380) claims it, so it never could. The fourth is in the Latin-anchoring paragraph: a hypothetical Cyrillic comma name "now falls to the fields-only catch-all". Neither 2.x ledger has a fields-only rule -- each says so twenty lines earlier -- so such a name would arrive UNEXPLAINED, which is the right outcome and the better story. Measured: no Cyrillic comma name is in any corpus, so the sentence was always about a shape nobody has seen. Same defect class as #451 itself: a comment asserting something the ledger it sits in cannot support. Comment-only; all three gates unmoved (1.4.0 229/0, 2.0.0 194/0, 2.1.0 102/0) and 147 guards green. The 1.4.0 copy of the catch-all sentence is still TRUE and is left alone. It stops being true when #451 deletes the rule, which that commit has to handle. Refs #451 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…rule "'Smitty' Jones Jr." was one of the fields-only fix(suffix-routing) catch-all's nine names at 1.4.0. It is three tokens -- a quoted nickname, one bare name word, one suffix word -- so it will not fit the two-token suffix rules #451 is landing in a later task; it needs its own rule now. The tree reads it given 'Jones', family '', suffix 'Jr.', which is decisions.md#N3's lone-word nickname rule at work: the diff against 1.4.0 moves exactly {family, suffix}, consistent with 1.4 having routed the trailing suffix word into `family` the way the general suffix-routing note describes, though 1.4's own internals are outside what this ledger's worker can check directly. The comment deliberately does not settle which of two readings is right for a nickname-led name with a trailing suffix: decisions.md#H1 records that N3 and H1 count a suffix in opposite directions, and calls that question undecided. This rule classifies the 1.4-to-2.x movement only. Declares the suffix alternation in _LATIN_ALTERNATION_SOURCES (SUFFIX_WORDS, all six members genuine entries), adds a _MUST_NOT_MATCH probe set (two-token names and the H1 title shape), and records the _CORPUS_CLAIMS entry. Refs #451 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…wn rules 'Aishwarya Rai' and 'Carod i' were two of the fields-only fix(suffix-routing) catch-all's eight names at 1.4.0, and unlike the other six their readings are open bugs, not accepted behavior: #342 (OPEN, bug, milestone v2.2) reports that a bare trailing 'Rai' -- a common South Asian surname -- is read as a post-nominal credential instead, emptying `family`; #397 (OPEN, enhancement, milestone v2.2) reports the same shape for a trailing Catalan/Polish linking 'i', read as a Roman-numeral generation marker instead of joining the family the way Spanish 'y' already does. Both get literal-anchored rules rather than a [[never]] exclusion. An excluded name reports UNEXPLAINED and exits non-zero, which would redden the 1.4.0 gate and block every release until both bugs land -- and neither has a chosen fix yet (#342 lists two candidate options that trade different corpus names; #397 leaves open whether 'i' joins CONJUNCTIONS at all). Both rules classify "because the cause is known and recorded, NOT because the reading is wanted", the exact wording decisions.md#P5's 2026-08-22 #369 entry attributes to that #369 ledger's own now-rewritten rules. Each rule carries a DELETE-THIS-RULE-when-fixed instruction. Once the fix lands the rule stops matching any diff, and the gate's own dormancy check (not a declared `dormant`, the general one that catches any rule gone quiet) prints it as 'EXPLAINED NOTHING' -- the reminder is mechanical, not just a comment. Catch-all 8 -> 6. Adds both _MUST_NOT_MATCH probe sets (three-token neighbours and #397's two no-diff longer names) and both _CORPUS_CLAIMS entries. Gate green at 1.4.0 (229/0), 2.0.0 (194/0) and 2.1.0 (102/0); 147 guards green. Refs #451, #342, #397 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ounded rules The last six names on the fields-only fix(suffix-routing) rule get rules that name them, and the catch-all is deleted. No ledger has a fields-only rule now -- 1.4.0 70 rules, 2.0.0 58, 2.1.0 51, and 0 with no name_regex in any of them -- which is what #451 was filed for. Where the six went: 'John V', 'Mohamad X' -> a trailing roman numeral 'Smith Jr.' -> the trailing suffix word jr 'Donald mc', 'QC MP' -> a trailing credential acronym 'Jack M.A.' -> the dotted M.A. spelling (ma-do) Four rather than one, and the split is FORCED rather than chosen. _LatinCopy carries exactly one `vocabulary` and test_latin_alternations_mean_something_the_vocabulary_ships asserts exactly one roster key per alternation, so `jr` (SUFFIX_WORDS) and `mc|mp` (SUFFIX_ACRONYMS) cannot share an alternation. `m\.?a\.?` cannot be an alternation member at all: measured, it matches the corpus fragment 'M.A.', which _normalize leaves as 'm.a', not a SUFFIX_ACRONYMS entry -- the same finding the ambiguous-surname-acronym rule already records. And the numeral reaches `suffix` through no wordlist at all: `x` is in none of the suffix sets, so _NOT_A_VOCABULARY_COPY is where its alternation is declared, being _ROMAN's body with the empty alternative dropped (proved equivalent on every string of length 1..4 over IVXivx; the empty case is a latent widening removed, not a bug fixed). All four are appended at the END of the file. The catch-all sorted last because it had no name_regex; with regexes these join the regex tier where file order decides ties, so appending them reproduces its position exactly and no rule above loses a contest. Verified before and after: 'Carod i' stays with fix(#397) and '田中さん II' with fix(cjk-glued-honorific-peel), and _CROSS_RULE_WINNERS now pins both. Deleting the rule falsified prose in the same file, and the sweep found more than the one sentence this task set out to fix. The fix(#379) Latin-anchoring paragraph now reads as its 2.0.0 and 2.1.0 copies have since 8237b54: a hypothetical Cyrillic comma name would arrive UNEXPLAINED and block the gate, which is the right outcome. Twelve other references to the catch-all are corrected -- past-tense history kept, dangling "below" pointers dropped, and three sentences that were already false before this commit rewritten against fresh measurements: two counterfactuals that stopped holding when the catch-all lost `given` earlier in this bundle, and _EXCLUSION_EFFECT's docstring figure, which named a one-tuple where the answer was a three-tuple. fix(suffix-routing) leaves that roster's own absorbed_by tuple, a shrink rather than a growth. Gate green at 1.4.0 (229/0), 2.0.0 and 2.1.0; 147 guards green; tools/differential/compare.py untouched. Refs #451 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Tasks 1-9 gave all fourteen names the #451 catch-all had absorbed their own rules and deleted it. This task closes the gap that let it happen: validate_rules now rejects any rule that has `fields` but no `name_regex`, so the shape cannot come back. A rule with no name narrowing claims every name whose diff fits its `fields` -- a very common shape -- and no guard could see it grow: _CORPUS_CLAIMS records a regexless rule's reach as the WHOLE CORPUS, so it sits at its maximum from the moment it is written and arrivals never move it. The one rule ever shaped this way grew from 4 explained names to 14, across six unrelated behaviour families, with every guard green and the gate reporting 0 unexplained throughout. This is a rejection of the SHAPE, not a specificity floor on role count -- that was proposed and declined in #372/#373 as vacuous, since the one fields-only rule named 3 of 7 roles and no floor rejecting it matches anything else. The check costs no migration: no rule in any shipped ledger (1.4.0, 2.0.0, 2.1.0) has this shape. Several existing tests constructed a fields-only rule as a shorthand to exercise something else -- the deeper `fields` validation, the `dormant` key, `_ambiguities` as a legal field name, or main()'s sort/dormancy reporting through a synthetic ledger. Each is updated to carry a `name_regex` so it keeps testing what it was written to test. One of them, test_main_sorts_a_name_regex_rule_ahead_of_a_fields_only_ one, pinned a scenario -- a broad fields-only rule shadowed by a specific name_regex rule -- that this check makes impossible to construct as a valid ledger; it is repurposed to prove main() rejects that shape instead, since the underlying sort mechanism stays covered directly by _sorted_rules' own tests. Verified: the two new tests (rejects the shape, accepts both neighbouring shapes), all three shipped ledgers still load and gate green (1.4.0 229/0, 2.0.0, 2.1.0), and the full tests/v2/ suite passes. Closes #451 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The third ledger arc section, in the two existing ones' idiom. What landed: the validate_rules ban; the over-declared-fields defect and its fix(#424) twin, which together are why #452 was filed; the cause-not-consent standing for a rule that classifies an unwanted reading; one-vocabulary-per-alternation as a constraint the guards impose rather than a preference; a rule's character class as a pinned copy of the range the implementation acts on; and past-tense counts staying where live ones go. Three things were FOUND rather than decided and get their own heading, because each is a fact about the shape rather than a choice about it: a fields-only rule silently falsifies other comments (measured -- zero of seven probes arrived unexplained while it lived); copy-paste is how a ledger comment goes false without being edited; and _sorted_rules is now the identity on every ledger that loads, kept anyway as the defence that makes the ban safe to state. Declined and why: a [[never]] for #342/#397 (reddens the gate for a whole milestone), the single-rule replacement with a mixed-vocabulary tail (measured -- the M.A. member reaches a non-entry, x is in no set), the literal-alternation fallback, and extending the bundle to the trailing-numeral ambiguity or to 'Donald mc', both of which are documented consequences of decided scopes rather than defects. Refs #451, #452 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ng claims Every statement corrected here was TRUE before this bundle and is false after it, which is the class the arc entry itself names: "copy-paste is how a ledger comment goes false without being edited". Nothing here changes code or a regex. Measured first, on all three ledgers: 1.4.0 70 rules / 0 fields-only, 2.0.0 58 / 0, 2.1.0 51 / 0, and _sorted_rules is the identity on each. So there is one tier, the sort is stable, and FILE ORDER decides every tie -- the opposite of what the three ledger headers promised. mechanisms.md#LEDGER-RULE-SEPARATION was the worst of them, being the citable contract: headed "file order doesn't" separate rules and stating that "a fields-only rule sorts last unconditionally", it would have had a rule author write a rule compare.py now refuses at startup. Rewritten to state the ban, that file order is the only separator left, and that the tier sort is KEPT as the defence for a reader who does not call validate_rules first (its docstring in compare.py says why). The three ledger headers said "File order is not load-bearing" and "optional name_regex"; in the 1.4.0 file that sat twenty lines from prose the same bundle added saying FILE ORDER decides ties. All three now say name_regex is required and file order decides. tools/differential/README.md said a rule "may narrow its match with name_regex ... and/or fields" and that "a name_regex rule outranks a fields-only one". Required, and unreachable, respectively. Two decisions.md entries. The dormancy arc's declined bullet counted "exactly one fields-only rule exists in any ledger"; the count is zero and the DECLINE still stands -- it is anchored to when it was measured rather than restated live, per this bundle's own rule about counts. And P5's #369 entry said its 1.4.0 rule keeps 'Sheik abdul salam' off the catch-all "it otherwise lands on". There is no catch-all to land on: measured through classify() over the ledger minus that rule, the name and the 'Sheik abdul salam Jr' example both return None. The reason to keep the rule is stronger than the entry claimed -- a red gate, not a mislabel. Two the sweep found beyond the review's list. The fix(#360) ste comment carried the same "it lands on the fields-only catch-all" in the present tense (measured: None without it). And the header's "(spec S5)" was a citation to the gitignored spec that the armed legacy-citation guard bans -- it passed only because the line happened to wrap between the two words, so reflowing the header surfaced it. Repointed at the README, which is where that claim actually lives. Refs #451 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #453 +/- ##
=======================================
Coverage 98.64% 98.64%
=======================================
Files 45 45
Lines 3177 3177
=======================================
Hits 3134 3134
Misses 43 43 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #451.
expected_since_1.4.0.tomlhad one rule with noname_regex. Becauseclassify()matches onfieldsalone when there is no regex, it claimed any name whose diff fit{given, family, suffix}— and_CORPUS_CLAIMSrecords a regexless rule's reach as the whole corpus, so it sat at its maximum and arrivals could never move it.test_every_rule_claims_the_recorded_share_of_the_corpuspassed whether it explained 4 names or 40.It drifted from the four names its prose described to seventeen at its high-water mark (decisions.md#H1 records the 14 → 17 step), and stood at fourteen across six unrelated behaviour families when this branch opened — every guard green throughout.
This gives all fourteen rules of their own, deletes the catch-all, and makes
validate_rulesreject the shape so it cannot return.Where the fourteen went
Mc Donald,Sir de Mesnilfix(#360) mc/fix(#367) sir de— rules the 2.x ledgers already had and this one never gotfc6e303(,.,fix(A2)content-free input..,was on a comma rule, on nothing but its regex reaching it7cc282fJohn😀Smithfix(emoji-boundary), a pinned copy of_tokenize.py's_EMOJI_RANGES1b55eb1Berg, Jan vdfix(#380)— the name two comments promised would "arrive unexplained"525f743'Smitty' Jones Jr.fix(N3), three tokens so it fits no two-token rule7b2b2b6Aishwarya Rai,Carod i67ec9cbSmith Jr.,John V,Mohamad X,QC MP,Donald mc,Jack M.A.ffd242a8b06f48narrows the rule's declaredfieldsmid-way — it declaredgivenwhile nothing it still explained moved that role, thefix(#424)shape decisions.md#H1 records.Why four rules and not one
Forced, not chosen, and by three independent constraints:
_LatinCopycarries onevocabularyand the alternation guard asserts one roster key per alternation, sojr(SUFFIX_WORDS) andmc/mp(SUFFIX_ACRONYMS) cannot share one.m\.?a\.?cannot be an alternation member at all — measured, it matches the corpus fragment'M.A.', which_normalizeleaves as'm.a', not an entry. Theambiguous-surname-acronymrule already records the same finding.xis in no vocabulary set; it reachessuffixthrough_ROMAN, so its alternation is a_NOT_A_VOCABULARY_COPYdeclaration.Two rules for readings nobody wants
Aishwarya RaiandCarod ilose their surnames, and both are open v2.2 bugs. The catch-all was classifying them as intentional 2.0 behaviour changes — and AGENTS.md names the 1.4 summary as the release log's source for Behavior Changes.Each gets a rule saying the diff is classified because the cause is known and recorded, NOT because the reading is wanted — the standing decisions.md#P5 established for #369 — plus a delete-when-fixed note the dormancy check enforces. A
[[never]]was rejected: an excluded name reports UNEXPLAINED and exits non-zero, which would block every release until both bugs are fixed.What the reviews found beyond the issue
Each of these is the same defect one level up, and each is fixed here:
_tokenize.pydoes not divide, so its own prose was false for every one of them. Now a pinned copy of_EMOJI_RANGES, with a guard proved to fail on the draft it replaced.classify(): zero arrived unexplained — four went to the catch-all, two tofix(comma-precomma-family), one to the neighbouring rule that exists to claim it.fix(#379)comment; four sentences were true only in the 1.4.0 file, including one citingfix(suffix-routing)— a rule no 2.x ledger has — and one saying a name "falls to the fields-only catch-all" twenty lines after the same file says none exists there (8237b54).ad78f04).Consequences worth knowing
_sorted_rulesis now the identity on every ledger that loads — the ban removes the only shape that could occupy its second tier, so file order decides every tie. Kept deliberately, with the docstring saying why: it is what makes the ban safe for any reader that does not callvalidate_rulesfirst.test_no_legacy_citationsbansspec\s+[S§]?\d, and the 1.4.0 header's(spec S5)was passing only because the line wrapped between "spec" and "S5". Repointed. That guard is one reflow away from firing on any other wrapped instance — worth its own look.Not in scope
name_regexdoes not protect against.Mohamad X/ the trailing-numeral ambiguity. By C-i a trailingXis borne as an ordinary name (Malcolm X). The fork does reportSUFFIX_OR_NAME, whileAishwarya Raireports nothing — so the live question is aboutSUFFIX_ACRONYMS_AMBIGUOUSand may belong onRaiis parsed as a post-nominal suffix, consuming a common South Asian surname #342. Recorded, not filed.Donald mcandSmith vdlose the family entirely. Its only example isde-shaped, which is why no test catches it.Verification
All three baselines exit 0, unchanged diff counts: 1.4.0 229/0, 2.0.0 194/0, 2.1.0 102/0. Full suite 6062 passed, 156 skipped, 9 xfailed. Zero fields-only rules in any ledger (1.4.0 70 rules, 2.0.0 58, 2.1.0 51).
Note what is not the verification: a green gate. It was green throughout the absorption this issue is about. The per-step count movements are what was checked, commit by commit.
🤖 Generated with Claude Code