fix(anon): always-on structural PHI detectors in the anonymizer leak-check (BACKLOG #331)

docs/BACKLOG.md

@@ -2916,7 +2916,7 @@ That distinction matters concretely for the ASVS record. The scorecard's absence
 
 ## 331. Anonymizer's fail-closed leak-check has no structural PHI detectors
 
-> 🔢 **Filed 2026-08-01 — not started.** Value **6/10** · Difficulty **4/10** · _quick win_. The function that earns the right to share a de-identified dataset verifies a known-string denylist — `leak_check` is `scan_text` (FORBIDDEN patterns, one routable-IPv4 check, estate substrings; `scripts/security/scan_forbidden.py:772-795`) plus a field-anchored site code, and a real MRN is not a denylisted string — and on a token-less checkout it degrades to the IPv4 check alone over an HL7 body and still returns clean, a gap `f3c6d348` hit in practice with a hand overlay that was never committed; wiring `token_floor_failure()` into the bridge is small, but the unmapped-field report and detectors scoped to fields no rule matched cross the `anonymize` seam and must be mirrored into `tee/anon/leak.py` for `test_anon_parity`.
+> ✅ **SHIPPED 2026-08-06 — structural PHI-shape detectors + unmapped-field coverage report + token-floor signal built.** Value **6/10** · Difficulty **4/10** · _quick win_. `leak_check`/`leak_report` now run high-precision structural detectors (dashed SSN, punctuated NANP phone, CX `MR`/`MRN`-typed identifier) over **the fields no rule matched**, record every present-but-unmapped field in a coverage report (`LeakReport.unmapped_fields`, carried into the `LeakError` on a refusal and exposed via the `on_report` hook), and record `token_floor_failure()` in every report, folding it into the fail-closed decision under the `require_live_denylist` **opt-in** lever — default off, so a token-less CI/OSS/fork load still passes with the structural detectors as the live backstop; a deployment that must refuse on an unloaded denylist sets the lever. The whole structural block is mirrored byte-identical into `tee/anon/leak.py` with a new engine/tee `leak_report` parity test; each detector was falsified. ADR 0030 §5/§7/Consequences amended (the "deferred" phrasing was stale). The aggressive/broad-shape tier (bare-digit DOB/SSN, name-like runs) stays deferred by owner call — it mass-false-positives on HL7 bodies dense with dates/order-numbers.
 
 **Cluster:** Security & Compliance. **Priority:** P2. **Verdict:** build. **Severity:** medium.
 

docs/adr/0030-anonymization-test-harness-tee.md

@@ -15,6 +15,18 @@
   file-replay loader ([`harness/reconcile/compare.py`](../../harness/reconcile/compare.py),
   [`harness/load/corpus.py`](../../harness/load/corpus.py)); and the publish leak-gate
   [`scripts/security/scan_forbidden.py`](../../scripts/security/scan_forbidden.py) (`FORBIDDEN`).
+- **AMENDED 2026-08-06 (BACKLOG #331):** the leak-check's structural gap that §5 / §7 / Consequences
+  below called *scoped out / a deferred improvement* is now **built** — high-precision structural
+  PHI-shape detectors (dashed SSN, punctuated NANP phone, CX `MR`/`MRN`-typed identifier) over **the
+  fields no rule matched**, an unmapped-field **coverage report** (`LeakReport.unmapped_fields`; address
+  only, never a value, carried into the `LeakError` on a refusal and exposed via `on_report`), and the
+  `token_floor_failure` signal recorded in every report and folded into the fail-closed decision under
+  the `require_live_denylist` **opt-in** lever — default off, so a token-less CI/OSS/fork load still
+  passes with the structural detectors as the live backstop; a deployment that must refuse on an
+  unloaded denylist sets the lever. Mirrored behaviourally-identical into `tee/anon/leak.py`. The
+  *aggressive/broad* shape tier (bare-digit DOB/SSN, name-like runs) stays deferred — it
+  mass-false-positives on HL7 bodies (§5). This bullet is the single source of the change; the stale
+  "deferred" phrasings below point back here.
 - **Decision in one line:** ship a **pure-stdlib, dependency-free `anon` package** that turns real,
   messy HL7 v2 into structurally-faithful **PHI-free** datasets via a **two-layer rule model — a
   declarative field-*selection* map (data) over a code registry of pure surrogate *functions* (logic)** —
@@ -266,8 +278,11 @@ silently misses a field is worse than none** — though note the limit in the ne
 PHI the rule map **missed** sails through the fail-closed gate *clean* unless that field happens to contain a
 denylisted token — a real MRN is not a denylisted string. **Rule-map completeness is therefore the primary
 control; the leak-check backstops known *strings*, not missed *fields*.** Adding structural detectors
-(MRN/SSN/DOB/phone shape, NANP-reserved vs real) to the post-anon check as a *true* field-level backstop is a
-candidate improvement, scoped out for this slice with the residual called out (Consequences).
+(MRN/SSN/DOB/phone shape, NANP-reserved vs real) to the post-anon check as a *true* field-level backstop was a
+candidate improvement — **built in BACKLOG #331** (AMENDED 2026-08-06; see the status banner): the
+high-precision set (dashed SSN, punctuated NANP phone, CX `MR`/`MRN`-typed id) scoped to the UNMAPPED fields,
+plus an unmapped-field coverage report (`LeakReport.unmapped_fields`) recorded on every pass and surfaced on
+a refusal or via the `on_report` hook. The broad-shape tier stays scoped out (below).
 
 ### 6. Integration points
 
@@ -294,7 +309,9 @@ candidate improvement, scoped out for this slice with the residual called out (C
 0004](0004-payload-agnostic-ingress.md)) so X12 (`parsing/x12/`)/FHIR/raw plug in later — never HL7-parse a
 non-HL7 body. **Out:** statistical expert-determination de-id, free-text NLP scrubbing (NTE-3/OBX-5 narrative
 default to **blunt full-redaction**, §3 — not entity-level NLP), **per-component (bare-leaf) surrogates** (the
-default encoders are field-level `^`-joined, §3), structural PHI detectors in the leak-check (§5), and any
+default encoders are field-level `^`-joined, §3), an **aggressive/broad structural PHI search** (bare-digit
+DOB/SSN, name-like alpha runs — the mass-false-positive tier; note the *high-precision* unmapped-field
+detectors in the leak-check were **built in #331**, AMENDED 2026-08-06, §5), and any
 re-identification/linkage tooling.
 
 ### 8. Relationship to the planned de-id framework
@@ -340,7 +357,9 @@ AI-assistant `deidentified` data-scope source (PHI.md §9 forward-links to it on
   not MRN/SSN/DOB/name *shapes*, so a field the rule map missed passes the "fail-closed" gate clean unless it
   contains a denylisted token. **Rule-map completeness is the primary control**; the leak-check is necessary,
   not sufficient. Free-text (OBX-5/NTE-3) is the **highest-risk residual** — hence its full-redact default
-  (§3). Structural detectors are a deferred improvement (§5/§7).
+  (§3). **AMENDED 2026-08-06 (#331):** high-precision structural detectors over the UNMAPPED fields plus an
+  unmapped-field coverage report are now **built** (that deferral is closed; the broad-shape tier stays
+  deferred, §7) — rule-map completeness remains the primary control.
 - **Vendored-copy drift — bytes *and* behaviour.** Two copies can diverge; the rule/surrogate/token *files*
   are mitigated by a CI **byte-parity** check (the existing tee discipline) under one authority. But the
   engine-side `anon/hl7.py` (delegating to `Message`) and the tee's **standalone stdlib re-encoder** are

messagefoundry/anon/__init__.py

@@ -15,19 +15,22 @@
 Public surface:
 
 * :func:`anonymize` — de-identify one HL7 message (raises nothing PHI-bearing).
-* :func:`anonymize_checked` — :func:`anonymize` + a **fail-closed** :func:`leak_check`; raises
-  :class:`LeakError` (token categories only, never the value) if any known partner/site token
-  survives. This is how you *earn* the right to write a dataset to a shareable location.
-* :func:`leak_check` — forbidden-token hits via the publish-guard authority (ADR 0030 §5).
+* :func:`anonymize_checked` — :func:`anonymize` + a **fail-closed** :func:`leak_report`; raises
+  :class:`LeakError` (token categories + PHI shapes/addresses only, never a value) if any known
+  partner/site token survives **or** a structural PHI shape sits in a field no rule mapped. This is
+  how you *earn* the right to write a dataset to a shareable location.
+* :func:`leak_check` / :func:`leak_report` — token hits + structural PHI-shape detection over the
+  unmapped fields + the unmapped-field coverage report (ADR 0030 §5, BACKLOG #331).
 """
 
 from __future__ import annotations
 
+from collections.abc import Callable
 from pathlib import Path
 
 from .hl7 import anonymize_message
 from .keying import Keyer
-from .leak import LeakCheckUnavailable, leak_check
+from .leak import LeakCheckUnavailable, LeakReport, coverage_clause, leak_check, leak_report
 from .rules import DEFAULT_RULES, AnonError, FieldRule, RuleError, SurrogateKind, load_rules
 
 __all__ = [
@@ -37,11 +40,13 @@
     "Keyer",
     "LeakCheckUnavailable",
     "LeakError",
+    "LeakReport",
     "RuleError",
     "SurrogateKind",
     "anonymize",
     "anonymize_checked",
     "leak_check",
+    "leak_report",
     "load_rules",
 ]
 
@@ -79,18 +84,37 @@
     salt: str,
     overlay: Path | None = None,
     rules: tuple[FieldRule, ...] | None = None,
+    require_live_denylist: bool = False,
+    on_report: Callable[[LeakReport], None] | None = None,
 ) -> str:
-    """:func:`anonymize`, then a fail-closed :func:`leak_check`; raise :class:`LeakError` on any hit.
+    """:func:`anonymize`, then a fail-closed :func:`leak_report`; raise :class:`LeakError` on any hit.
 
     Use this whenever the output may be persisted/shared — a silently-missed token is worse than no
-    anonymization (ADR 0030 §5). The raised error names token *categories* only, never the value.
+    anonymization (ADR 0030 §5). The verification is now two-layered (BACKLOG #331): the known-token
+    denylist **and** high-precision structural PHI-shape detectors over the fields no rule matched,
+    scoped by the same ``rules`` the anonymizer applied. The raised error names token *categories* and
+    field *shapes/addresses* only, never a value, and carries a coverage clause (the count + addresses
+    of the unmapped fields, whether the denylist tables were live) so a refusal is legible.
+
+    ``require_live_denylist`` makes a non-live token source (``token_floor_reason`` set) a refusal
+    cause in its own right — the strict lever for a deployment that must not de-identify with the
+    customer denylist unloaded. It defaults **off**: the structural detectors are the live backstop,
+    and CI/OSS/fork runs legitimately have no token source. ``on_report`` receives the full
+    :class:`LeakReport` on both the clean and the refusing path (default: no emission).
     """
-    output = anonymize(raw, salt=salt, overlay=overlay, rules=rules)
-    hits = leak_check(output)
-    if hits:
+    effective = rules if rules is not None else load_rules(overlay)
+    output = anonymize(raw, salt=salt, rules=effective)
+    report = leak_report(output, rules=effective)
+    if on_report is not None:
+        on_report(report)
+    causes = list(report.hits)
+    if require_live_denylist and report.token_floor_reason is not None:
+        causes.append(f"denylist not live: {report.token_floor_reason}")
+    if causes:
         raise LeakError(
             "anonymized output still carries forbidden token(s): "
-            + "; ".join(sorted(set(hits)))
+            + "; ".join(sorted(set(causes)))
             + " — refusing to emit (fail closed). Extend the rule map for the missed field(s)."
+            + coverage_clause(report)
         )
     return output