Skip to content

fix(extraction+gliner2): capture noise — 5 spine defects, a GLiNER2 slot collision, and Pitfall 11's broken label ceiling - #3

Open
tkalevra wants to merge 5 commits into
mainfrom
worktree-extraction-noise-fix
Open

fix(extraction+gliner2): capture noise — 5 spine defects, a GLiNER2 slot collision, and Pitfall 11's broken label ceiling#3
tkalevra wants to merge 5 commits into
mainfrom
worktree-extraction-noise-fix

Conversation

@tkalevra

@tkalevra tkalevra commented Aug 5, 2026

Copy link
Copy Markdown
Owner

A live roster turn stored phantom entities, facts about verbs, a breed filed on the user, and a person whose gender was a number. The noise had two independent sources in two different layers, and one of them turns out to be a Pitfall 11 violation.

All defects reproduce deterministically with the production spine flags on (SENTENCE_PIPELINE / SPINE_NAMING_CHAIN), and every one of them had really been written to a live memory.

Layer 1 — the spine deriver (5 defects)

# Input Stored (wrong) Now
1 ...named Rowan(goes by Ro) name = rowan(goes rowan + alias ro
2 My son Rowan (goes by Ro) is 12. ("goes", age, "12"), alias lost (rowan, age, "12") + alias
3 Ines is my wife. (wife, spouse, user) — phantom person (ines, spouse, user)
4 Quinn, age 10, F. (f., instance_of, age) (quinn, age, "10")
5 My dog Bracken is a morkie. (user, "dog", "morkie"), breed unbound (bracken, instance_of, morkie)
  1. A bracket is a spaCy prefix/suffix rule, so it never fires mid-token — the glued run survived as one PROPN and became the name. _install_bracket_infix adds the missing infix rule; spaced and unspaced forms now parse identically.
  2. A parenthetical has no subject, so spaCy re-attaches the alias verb as nsubj of the following copula. _binding_nickname read only the relative-clause form; it now also accepts the linearly anchored parenthetical (vocabulary from the growable alias_predicate cue map), the measure chain rejects a non-nominal subject and recovers the real one left of the bracket.
  3. _bound_name_for_type read only my <role> is <Name>, never the reverse. Binding branch (4), gated on 1st-person possession, casing-robust so capture is not name-dependent.
  4. A type noun already saturated by its own cardinal nummod cannot host a name; a single-character name is an initial; the appositive chain stores the pair as the scalar it is.
  5. Binding branch (5) binds the determiner-introduced predicate nominal to the subject NP's name; the attribute-scalar chain steps aside for exactly that shape — narrowed by the complement, so a real possessed scalar (my address is 123 Main Street, …) is still captured.

Also excludes the naming noun name from the type↔name binding and the copula-name chain: my mothers name is Priya belongs to _chain_genitive_name, and reading it as a type stored (priya, instance_of, "mothers name").

A regression this PR caught in itself: splitting the bracket promoted the nickname to a real token, so it became the nearest preceding PROPN and he resolved to ro — the age landed on the alias. Found by running the real /ingest pipeline, not by the unit tests. An alias is a second name, not a new referent, so the coref scan now skips the alias run (already marked in _ni_suppress).

Layer 2 — GLiNER2 (found only by running the live pipeline)

Slot collision. GLiNER2 scores each candidate rel_type independently, so nothing stopped it returning the same (subject, object) under two labels. Verbatim raw output for "My son Rowan is 12.":

{'age': [('Rowan','12')], 'has_gender': [('Rowan','12')], 'height': [], ...}

Both committed → has_gender = 12, a person whose gender is a number. Same shape as the gender named F / quantity is 10 F / 12 m rows in the live store. Resolved with metadata the schema already carries (migration 101's scalar_datatype): a bare cardinal satisfies integer, is not a quantity (needs a unit), and is only vacuously a string — so age wins and has_gender is dropped. Non-numeric spans are left alone, because pref_name + also_known_as legitimately co-claim a name.

Pitfall 11's other half — the per-instance label ceiling. The label-content rule was honored. The count rule was silently lost when the caps became per-pool. GLiREL states it outright:

"We limit the number of relation type labels prepended to each training instance to 25." (arxiv:2501.03172)

The comment directly above the cap constants says exactly that, and _GLINER2_RELATION_LABEL_CAP = 25 still sits there marked "legacy". But scalar 15 + relational 20 are merged into one dict for a single extract_relations call — so the model received 35, 40% beyond the training maximum. Measured live: 33 labels, and that is the call that fired the colliding labels. Only /ingest breaches it; the other three call sites use the single capped helper (≤20).

The paper also measures the cost of merely approaching the cap — F1 by candidate count m: m=5 → 94.20, m=10 → 87.60, m=15 → 84.48 (FewRel). Fewer labels is measurably more accurate, not merely safer.

_enforce_gliner2_total_label_budget trims at the one point both pools are still distinguishable. Relational is sacrificed first (a scalar rel is the more specific storage path) but never below a reserved floor, because "a large scalar pool crowds relational out entirely" is the exact failure the split budgets were introduced to fix. Truncation is logged with the dropped label names — a silent cap reads as "we considered everything" when we did not.

Verified live after the fix: candidate_set_built total=25 (was 33), and a clean-tenant ingest of "My son Rowan is 12." now stores exactly one attribute row: rowan | age | 12.

Not changed here: the relational pool's own cap of 20 is within the training distribution but sits in the measured degradation zone. Pitfall 11 says a label-set change is a benchmark-gated whole-label replacement, so lowering it needs a benchmark, not a silent edit.

Design

Subject-agnostic throughout — dependency shape, morphology, orthography, and growable DB metadata/cue classes only. No name, role, breed, gender, or rel_type literal anywhere in the change.

Tests

388 passed. New: 14 spine regression tests, 9 GLiNER2 collision tests (driven by the verbatim payload captured off the line), 13 label-budget tests — plus the 3 tests in test_spine_genitive_name_and_alias.py that were already red on main.

Regression sentinels included deliberately: the possessed attribute scalar (an earlier, broader guard silently dropped it), an unpossessed predicate nominal, a real role apposition, non-numeric co-claiming, and identifier tokenization (CVE-2024-9999, x86-64, 10.0.0.5:8080, AB123-XY all verified whole; contractions unaffected).

🤖 Generated with Claude Code

https://claude.ai/code/session_017uPqpFMSnoi1F4EXzKcPSW

tkalevra and others added 5 commits August 5, 2026 19:34
A live roster turn ("my son <Name> (goes by <Nick>) is 12", "<Name> is my
wife", "my dog <Name> is a morkie") stored phantom entities, facts about
verbs, and a breed filed on the user. All five reproduce deterministically
against derive_sentence_facts with the production spine flags on.

1. UNSPACED PARENTHETICAL glued the alias run onto the name. A bracket is a
   spaCy prefix/suffix rule, so it never fires mid-token: "Rowan(goes by Ro)"
   tokenized as the single PROPN "Rowan(goes", and that malformed surface
   became THE NAME, collecting kin and gender edges. _install_bracket_infix
   adds the missing infix rule; spaced and unspaced now parse identically.

2. PARENTHETICAL ALIAS dropped, and a junk scalar minted. spaCy gives the
   run no subject and re-attaches the alias verb as nsubj of the following
   copula, so "… (goes by Ro) is 12" derived ("goes", age, "12") — a fact
   about a verb — while the alias was lost (_binding_nickname read only the
   relative-clause form). The nickname reader now also accepts the linearly
   anchored parenthetical (vocabulary from the growable alias_predicate cue
   map), the measure chain rejects a non-nominal subject, and it recovers the
   real subject left of the bracket so the age still lands.

3. INVERSE COPULA minted a PHANTOM ROLE ENTITY. _bound_name_for_type read
   only "my <role> is <Name>", never "<Name> is my <role>", so the possessive
   chain minted (wife, spouse, user) — a phantom person that went on to
   collect an instance_of, an owns, and a retraction's negative fact, all
   conflicting with the real spouse. Binding branch (4), gated on 1st-person
   possession, casing-robust so the capture is not name-dependent.

4. ATTRIBUTE-VALUE APPOSITION read as a type binding. "Quinn, age 10, F."
   hung the trailing code on "age" as an appos → an entity F filed as an
   instance of "age". A type noun already saturated by its own cardinal
   nummod cannot host a name, a single-character name is an initial, and the
   appositive chain now stores the pair as the scalar it is.

5. COPULA TYPE-PREDICATE lost the type and minted a noun-as-relation. "My dog
   Bracken is a morkie" bound the breed to nothing while the attribute-scalar
   chain read the clause as a possessed literal → (user, "dog", "morkie").
   Binding branch (5) binds the determiner-introduced predicate nominal to the
   subject NP's name; the attribute-scalar chain steps aside for exactly that
   shape, narrowed by the complement so a real possessed scalar ("my address
   is 123 Main Street, …") is still captured.

Also excludes the naming noun "name" from the type↔name binding and the
copula-name chain: "my mothers name is Priya" is the genitive-name chain's
construction, and reading it as a type stored (priya, instance_of, "mothers
name"). That fixes the three pre-existing failures in
test_spine_genitive_name_and_alias.py.

Subject-agnostic throughout: dependency shape, morphology, orthography, and
the growable DB cue classes only — no name, role, breed, or gender literal.

Tests: 365 passed (14 new in test_spine_parenthetical_alias_and_role_phantom,
plus the 3 genitive-name tests that were red on main).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017uPqpFMSnoi1F4EXzKcPSW
… helper

Use spacy.util.compile_infix_regex rather than a hand-joined alternation, and
record what the grounding check turned up: a token_match is consulted before
the infix split (infixes are step 9 of the tokenizer loop), so the identifier
matcher still wins — CVE-2024-9999, x86-64, 10.0.0.5:8080 and AB123-XY all
verified whole. The one documented way a new infix silently no-ops is a
conflicting tokenizer SPECIAL CASE (nlp.tokenizer.rules); none spans a
bracket, contractions are unaffected, and that table is already owned by
_reconcile_cue_tokenizer_exceptions.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017uPqpFMSnoi1F4EXzKcPSW
Found by running the real /ingest pipeline in pre-production, not by the unit
tests: "I have a son named Rowan(goes by Ro), he is 12" filed the age on "ro"
instead of "rowan".

Caused by the bracket infix in the previous commit. Splitting the glued run is
correct, but it also promoted the nickname to a real PROPN token — and the
nickname is then the NEAREST preceding proper noun, so the recency rule in
_person_coref handed "he" to the alias. An alias is a second name for someone
already introduced, never a fresh discourse referent.

The alias run is already marked in _ni_suppress by the named-instance pre-pass
— the same span the binding consumed as the nickname — so the coref scan skips
it there, keeping ONE notion of "the alias run" in the module. This also fixes
the relative-clause form ("Jamie who goes by Jay … he is 12") for free.

Tests: 366 passed (one new regression test).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017uPqpFMSnoi1F4EXzKcPSW
Found by running the real /ingest pipeline, not by the unit tests — this sits
ABOVE the spine deriver, so no deriver test could see it. GLiNER2 scores each
candidate rel_type independently, so nothing stopped it returning the same
(subject, object) pair under two different scalar labels. Verbatim raw output
for "My son Rowan is 12.":

    {'age': [('Rowan','12')], 'has_gender': [('Rowan','12')], 'height': [], ...}

Both edges converted, both committed, and the store gained has_gender = 12 —
a person whose gender is a number — beside the correct age. Same shape as the
"gender named F" and "quantity is 10 F" / "12 m" rows a real memory collected.

The discriminator is metadata the schema ALREADY carries, not a rel_type list:
migration 101 gives every SCALAR rel a scalar_datatype ("integer"/"quantity"/
"string") precisely to drive per-datatype validation. A bare cardinal satisfies
integer, is not a quantity (that needs a unit), and is only vacuously a string,
so age wins the span and has_gender is dropped.

Deliberately narrow, so it can only ever remove noise:
  - only groups where 2+ distinct rel_types claim the SAME (subject, object)
  - only when the value is a bare cardinal; a non-numeric span is left alone,
    because co-claiming is legitimate there ("ro" really is both pref_name and
    also_known_as) and nothing would justify picking a winner
  - no strictly-typed winner, or a tie -> left alone, never guess
  - metadata unresolvable or any error -> edges returned untouched

Tests: 375 passed (9 new, driven by the verbatim payload captured off the line).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017uPqpFMSnoi1F4EXzKcPSW
…gets broke

Pitfall 11 has two halves. The label-CONTENT half was honored. The COUNT half
was silently lost when the caps became per-pool.

GLiREL states it outright: "We limit the number of relation type labels
prepended to each training instance to 25." (arxiv:2501.03172). The comment
directly above the cap constants in main.py says exactly that, and
_GLINER2_RELATION_LABEL_CAP = 25 still sits there marked "legacy".

But the split-budget change made the caps PER-POOL — scalar 15, relational 20 —
and /ingest then MERGES both pools into ONE dict handed to a SINGLE
extract_relations() call. What the model actually receives is their SUM: 35.
That is not near the limit, it is 40% BEYOND the training maximum — a sequence
shape GLiREL never saw. Measured on the live line: one /ingest call carried 33
labels and GLiNER2 fired two conflicting scalar labels on ONE span
({'age': [('Rowan','12')], 'has_gender': [('Rowan','12')]}), which is the
store's "gender named F" / "quantity is 10 F" / "12 m" noise family.

Only /ingest breaches it. The other three extract_relations call sites
(sentence pipeline, /harvest-spans, /extract/rewrite trigger spans) all go
through _build_relational_candidate_labels, a single pool capped at 20.

The paper also measures the cost of merely APPROACHING the cap — F1 by
candidate-label count m:
    m=5  -> 94.20 (FewRel) / 83.28 (Wiki-ZSL)
    m=10 -> 87.60          / 83.67
    m=15 -> 84.48          / 73.91
5->15 labels costs ~10 F1 on FewRel and ~9 on Wiki-ZSL. Fewer labels is
measurably MORE accurate, not merely safer, so this ceiling is a floor on
quality rather than a formality.

_enforce_gliner2_total_label_budget trims at the one point both pools are still
distinguishable, before the merge erases the distinction. Relational is
sacrificed first (a scalar rel is the more specific storage path, matching how
the merge already resolves ties) but never below _GLINER2_RELATIONAL_FLOOR,
because "a large scalar pool crowds relational out entirely" is the exact
failure the split budgets were introduced to fix. Both pools arrive ranked, so
trimming is a deterministic tail-drop and the highest-value labels survive.
Every truncation is logged with the dropped label NAMES — a silent cap reads as
"we considered everything" when we did not.

NOT changed here: the relational pool's own cap of 20 is within the training
distribution but sits in the measured degradation zone. Pitfall 11 says a
label-set change is a BENCHMARK-GATED whole-label replacement, so lowering it
needs the owner's benchmark, not an edit in this commit.

Tests: 388 passed (13 new).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017uPqpFMSnoi1F4EXzKcPSW
@tkalevra tkalevra changed the title fix(extraction): five capture defects that stored noise instead of facts fix(extraction+gliner2): capture noise — 5 spine defects, a GLiNER2 slot collision, and Pitfall 11's broken label ceiling Aug 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant