fix(extraction+gliner2): capture noise — 5 spine defects, a GLiNER2 slot collision, and Pitfall 11's broken label ceiling - #3
Open
tkalevra wants to merge 5 commits into
Open
Conversation
A live roster turn ("my son <Name> (goes by <Nick>) is 12", "<Name> is my
wife", "my dog <Name> is a morkie") stored phantom entities, facts about
verbs, and a breed filed on the user. All five reproduce deterministically
against derive_sentence_facts with the production spine flags on.
1. UNSPACED PARENTHETICAL glued the alias run onto the name. A bracket is a
spaCy prefix/suffix rule, so it never fires mid-token: "Rowan(goes by Ro)"
tokenized as the single PROPN "Rowan(goes", and that malformed surface
became THE NAME, collecting kin and gender edges. _install_bracket_infix
adds the missing infix rule; spaced and unspaced now parse identically.
2. PARENTHETICAL ALIAS dropped, and a junk scalar minted. spaCy gives the
run no subject and re-attaches the alias verb as nsubj of the following
copula, so "… (goes by Ro) is 12" derived ("goes", age, "12") — a fact
about a verb — while the alias was lost (_binding_nickname read only the
relative-clause form). The nickname reader now also accepts the linearly
anchored parenthetical (vocabulary from the growable alias_predicate cue
map), the measure chain rejects a non-nominal subject, and it recovers the
real subject left of the bracket so the age still lands.
3. INVERSE COPULA minted a PHANTOM ROLE ENTITY. _bound_name_for_type read
only "my <role> is <Name>", never "<Name> is my <role>", so the possessive
chain minted (wife, spouse, user) — a phantom person that went on to
collect an instance_of, an owns, and a retraction's negative fact, all
conflicting with the real spouse. Binding branch (4), gated on 1st-person
possession, casing-robust so the capture is not name-dependent.
4. ATTRIBUTE-VALUE APPOSITION read as a type binding. "Quinn, age 10, F."
hung the trailing code on "age" as an appos → an entity F filed as an
instance of "age". A type noun already saturated by its own cardinal
nummod cannot host a name, a single-character name is an initial, and the
appositive chain now stores the pair as the scalar it is.
5. COPULA TYPE-PREDICATE lost the type and minted a noun-as-relation. "My dog
Bracken is a morkie" bound the breed to nothing while the attribute-scalar
chain read the clause as a possessed literal → (user, "dog", "morkie").
Binding branch (5) binds the determiner-introduced predicate nominal to the
subject NP's name; the attribute-scalar chain steps aside for exactly that
shape, narrowed by the complement so a real possessed scalar ("my address
is 123 Main Street, …") is still captured.
Also excludes the naming noun "name" from the type↔name binding and the
copula-name chain: "my mothers name is Priya" is the genitive-name chain's
construction, and reading it as a type stored (priya, instance_of, "mothers
name"). That fixes the three pre-existing failures in
test_spine_genitive_name_and_alias.py.
Subject-agnostic throughout: dependency shape, morphology, orthography, and
the growable DB cue classes only — no name, role, breed, or gender literal.
Tests: 365 passed (14 new in test_spine_parenthetical_alias_and_role_phantom,
plus the 3 genitive-name tests that were red on main).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017uPqpFMSnoi1F4EXzKcPSW
… helper Use spacy.util.compile_infix_regex rather than a hand-joined alternation, and record what the grounding check turned up: a token_match is consulted before the infix split (infixes are step 9 of the tokenizer loop), so the identifier matcher still wins — CVE-2024-9999, x86-64, 10.0.0.5:8080 and AB123-XY all verified whole. The one documented way a new infix silently no-ops is a conflicting tokenizer SPECIAL CASE (nlp.tokenizer.rules); none spans a bracket, contractions are unaffected, and that table is already owned by _reconcile_cue_tokenizer_exceptions. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017uPqpFMSnoi1F4EXzKcPSW
Found by running the real /ingest pipeline in pre-production, not by the unit
tests: "I have a son named Rowan(goes by Ro), he is 12" filed the age on "ro"
instead of "rowan".
Caused by the bracket infix in the previous commit. Splitting the glued run is
correct, but it also promoted the nickname to a real PROPN token — and the
nickname is then the NEAREST preceding proper noun, so the recency rule in
_person_coref handed "he" to the alias. An alias is a second name for someone
already introduced, never a fresh discourse referent.
The alias run is already marked in _ni_suppress by the named-instance pre-pass
— the same span the binding consumed as the nickname — so the coref scan skips
it there, keeping ONE notion of "the alias run" in the module. This also fixes
the relative-clause form ("Jamie who goes by Jay … he is 12") for free.
Tests: 366 passed (one new regression test).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017uPqpFMSnoi1F4EXzKcPSW
Found by running the real /ingest pipeline, not by the unit tests — this sits
ABOVE the spine deriver, so no deriver test could see it. GLiNER2 scores each
candidate rel_type independently, so nothing stopped it returning the same
(subject, object) pair under two different scalar labels. Verbatim raw output
for "My son Rowan is 12.":
{'age': [('Rowan','12')], 'has_gender': [('Rowan','12')], 'height': [], ...}
Both edges converted, both committed, and the store gained has_gender = 12 —
a person whose gender is a number — beside the correct age. Same shape as the
"gender named F" and "quantity is 10 F" / "12 m" rows a real memory collected.
The discriminator is metadata the schema ALREADY carries, not a rel_type list:
migration 101 gives every SCALAR rel a scalar_datatype ("integer"/"quantity"/
"string") precisely to drive per-datatype validation. A bare cardinal satisfies
integer, is not a quantity (that needs a unit), and is only vacuously a string,
so age wins the span and has_gender is dropped.
Deliberately narrow, so it can only ever remove noise:
- only groups where 2+ distinct rel_types claim the SAME (subject, object)
- only when the value is a bare cardinal; a non-numeric span is left alone,
because co-claiming is legitimate there ("ro" really is both pref_name and
also_known_as) and nothing would justify picking a winner
- no strictly-typed winner, or a tie -> left alone, never guess
- metadata unresolvable or any error -> edges returned untouched
Tests: 375 passed (9 new, driven by the verbatim payload captured off the line).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017uPqpFMSnoi1F4EXzKcPSW
…gets broke
Pitfall 11 has two halves. The label-CONTENT half was honored. The COUNT half
was silently lost when the caps became per-pool.
GLiREL states it outright: "We limit the number of relation type labels
prepended to each training instance to 25." (arxiv:2501.03172). The comment
directly above the cap constants in main.py says exactly that, and
_GLINER2_RELATION_LABEL_CAP = 25 still sits there marked "legacy".
But the split-budget change made the caps PER-POOL — scalar 15, relational 20 —
and /ingest then MERGES both pools into ONE dict handed to a SINGLE
extract_relations() call. What the model actually receives is their SUM: 35.
That is not near the limit, it is 40% BEYOND the training maximum — a sequence
shape GLiREL never saw. Measured on the live line: one /ingest call carried 33
labels and GLiNER2 fired two conflicting scalar labels on ONE span
({'age': [('Rowan','12')], 'has_gender': [('Rowan','12')]}), which is the
store's "gender named F" / "quantity is 10 F" / "12 m" noise family.
Only /ingest breaches it. The other three extract_relations call sites
(sentence pipeline, /harvest-spans, /extract/rewrite trigger spans) all go
through _build_relational_candidate_labels, a single pool capped at 20.
The paper also measures the cost of merely APPROACHING the cap — F1 by
candidate-label count m:
m=5 -> 94.20 (FewRel) / 83.28 (Wiki-ZSL)
m=10 -> 87.60 / 83.67
m=15 -> 84.48 / 73.91
5->15 labels costs ~10 F1 on FewRel and ~9 on Wiki-ZSL. Fewer labels is
measurably MORE accurate, not merely safer, so this ceiling is a floor on
quality rather than a formality.
_enforce_gliner2_total_label_budget trims at the one point both pools are still
distinguishable, before the merge erases the distinction. Relational is
sacrificed first (a scalar rel is the more specific storage path, matching how
the merge already resolves ties) but never below _GLINER2_RELATIONAL_FLOOR,
because "a large scalar pool crowds relational out entirely" is the exact
failure the split budgets were introduced to fix. Both pools arrive ranked, so
trimming is a deterministic tail-drop and the highest-value labels survive.
Every truncation is logged with the dropped label NAMES — a silent cap reads as
"we considered everything" when we did not.
NOT changed here: the relational pool's own cap of 20 is within the training
distribution but sits in the measured degradation zone. Pitfall 11 says a
label-set change is a BENCHMARK-GATED whole-label replacement, so lowering it
needs the owner's benchmark, not an edit in this commit.
Tests: 388 passed (13 new).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017uPqpFMSnoi1F4EXzKcPSW
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A live roster turn stored phantom entities, facts about verbs, a breed filed on the user, and a person whose gender was a number. The noise had two independent sources in two different layers, and one of them turns out to be a Pitfall 11 violation.
All defects reproduce deterministically with the production spine flags on (
SENTENCE_PIPELINE/SPINE_NAMING_CHAIN), and every one of them had really been written to a live memory.Layer 1 — the spine deriver (5 defects)
...named Rowan(goes by Ro)rowan(goesrowan+ aliasroMy son Rowan (goes by Ro) is 12.("goes", age, "12"), alias lost(rowan, age, "12")+ aliasInes is my wife.(wife, spouse, user)— phantom person(ines, spouse, user)Quinn, age 10, F.(f., instance_of, age)(quinn, age, "10")My dog Bracken is a morkie.(user, "dog", "morkie"), breed unbound(bracken, instance_of, morkie)_install_bracket_infixadds the missing infix rule; spaced and unspaced forms now parse identically._binding_nicknameread only the relative-clause form; it now also accepts the linearly anchored parenthetical (vocabulary from the growablealias_predicatecue map), the measure chain rejects a non-nominal subject and recovers the real one left of the bracket._bound_name_for_typeread onlymy <role> is <Name>, never the reverse. Binding branch (4), gated on 1st-person possession, casing-robust so capture is not name-dependent.nummodcannot host a name; a single-character name is an initial; the appositive chain stores the pair as the scalar it is.my address is 123 Main Street, …) is still captured.Also excludes the naming noun
namefrom the type↔name binding and the copula-name chain:my mothers name is Priyabelongs to_chain_genitive_name, and reading it as a type stored(priya, instance_of, "mothers name").A regression this PR caught in itself: splitting the bracket promoted the nickname to a real token, so it became the nearest preceding PROPN and
heresolved toro— the age landed on the alias. Found by running the real/ingestpipeline, not by the unit tests. An alias is a second name, not a new referent, so the coref scan now skips the alias run (already marked in_ni_suppress).Layer 2 — GLiNER2 (found only by running the live pipeline)
Slot collision. GLiNER2 scores each candidate rel_type independently, so nothing stopped it returning the same
(subject, object)under two labels. Verbatim raw output for"My son Rowan is 12.":Both committed →
has_gender = 12, a person whose gender is a number. Same shape as thegender named F/quantity is 10 F/12 mrows in the live store. Resolved with metadata the schema already carries (migration 101'sscalar_datatype): a bare cardinal satisfiesinteger, is not aquantity(needs a unit), and is only vacuously astring— soagewins andhas_genderis dropped. Non-numeric spans are left alone, becausepref_name+also_known_aslegitimately co-claim a name.Pitfall 11's other half — the per-instance label ceiling. The label-content rule was honored. The count rule was silently lost when the caps became per-pool. GLiREL states it outright:
The comment directly above the cap constants says exactly that, and
_GLINER2_RELATION_LABEL_CAP = 25still sits there marked "legacy". But scalar 15 + relational 20 are merged into one dict for a singleextract_relationscall — so the model received 35, 40% beyond the training maximum. Measured live: 33 labels, and that is the call that fired the colliding labels. Only/ingestbreaches it; the other three call sites use the single capped helper (≤20).The paper also measures the cost of merely approaching the cap — F1 by candidate count m: m=5 → 94.20, m=10 → 87.60, m=15 → 84.48 (FewRel). Fewer labels is measurably more accurate, not merely safer.
_enforce_gliner2_total_label_budgettrims at the one point both pools are still distinguishable. Relational is sacrificed first (a scalar rel is the more specific storage path) but never below a reserved floor, because "a large scalar pool crowds relational out entirely" is the exact failure the split budgets were introduced to fix. Truncation is logged with the dropped label names — a silent cap reads as "we considered everything" when we did not.Verified live after the fix:
candidate_set_built total=25(was 33), and a clean-tenant ingest of"My son Rowan is 12."now stores exactly one attribute row:rowan | age | 12.Not changed here: the relational pool's own cap of 20 is within the training distribution but sits in the measured degradation zone. Pitfall 11 says a label-set change is a benchmark-gated whole-label replacement, so lowering it needs a benchmark, not a silent edit.
Design
Subject-agnostic throughout — dependency shape, morphology, orthography, and growable DB metadata/cue classes only. No name, role, breed, gender, or rel_type literal anywhere in the change.
Tests
388 passed. New: 14 spine regression tests, 9 GLiNER2 collision tests (driven by the verbatim payload captured off the line), 13 label-budget tests — plus the 3 tests in
test_spine_genitive_name_and_alias.pythat were already red onmain.Regression sentinels included deliberately: the possessed attribute scalar (an earlier, broader guard silently dropped it), an unpossessed predicate nominal, a real role apposition, non-numeric co-claiming, and identifier tokenization (
CVE-2024-9999,x86-64,10.0.0.5:8080,AB123-XYall verified whole; contractions unaffected).🤖 Generated with Claude Code
https://claude.ai/code/session_017uPqpFMSnoi1F4EXzKcPSW