Benchmarks Update: Generation testing Scoring Rework

docs/source/content/contributing.md

@@ -328,7 +328,7 @@ set -a; source .env; set +a
 uv run python -m transformer_lens.tools.model_registry.verify_models --model <hf_repo>
 ```
 
-`verify_models` runs phases 1–4 (forward correctness vs HF, hook firing + gradients, weight processing, generation quality) and updates `data/supported_models.json` with the resulting status and per-phase scores. We recommend running `--dry-run` first to project memory and parameter count without loading the model, and verifying one model at a time — concurrent loads tend to OOM a single device.
+`verify_models` runs phases 1–4 (forward correctness vs HF, hook firing + gradients, weight processing, text-generation quality) and updates `data/supported_models.json` with the resulting status and per-phase scores. We recommend running `--dry-run` first to project memory and parameter count without loading the model, and verifying one model at a time — concurrent loads tend to OOM a single device.
 
 Running with `--no-hf-reference` skips the HuggingFace numerical comparison (Phase 1 becomes structural-only). A passing run is then recorded as **provisional** (status 4), which does *not* count as verified — re-run without the flag for a real HF-compared verification.
 
@@ -341,11 +341,11 @@ It's worth reading the per-phase scores in addition to the final status — the
 | 1 | 100% | — | Verification fails |
 | 2 | 75% | `logits_equivalence`, `loss_equivalence` | Verification fails |
 | 3 | 75% | `logits_equivalence`, `loss_equivalence` | Verification fails |
-| 4 | 50% | — | **Non-gating** — below 50% adds `"low text quality"` to the registry `note`; never fails verification. |
+| 4 | 54.5% (measured pass line, `p4_pass_threshold()`) | — | **Non-gating** — below the line adds a `"text quality poor (P4=…)"` note; never fails verification. |
 | 7 | 75% | `multimodal_forward` | Verification fails. A NULL score also fails. |
 | 8 | 75% | `audio_forward` | Verification fails. A NULL score also fails. |
 
-Phase 4 is intentionally lenient — it's a coherence metric, not a correctness check. A sub-100% Phase-4 score on a small parity-test model can still indicate a real adapter bug that the gates don't catch (missing `preprocess_weights` fold, wrong `default_prepend_bos`, and so on); the model can pass verification overall and still be worth a manual look.
+Phase 4 prompts each model with its resolved prompt profile (chat template, translation, code, own-language continuation, ...) and scores the generation against a known-good reference with one pinned multilingual judge, via the perplexity ratio `PPL(generated)/PPL(reference)`. It's intentionally lenient — a coherence metric, not a correctness check. A sub-100% Phase-4 score on a small parity-test model can still indicate a real adapter bug that the gates don't catch (missing `preprocess_weights` fold, wrong `default_prepend_bos`, and so on); the model can pass verification overall and still be worth a manual look.
 
 If verification fails by `~1e-3` or more against the HF reference, the bisection workflow lives at [Debugging Numerical Divergence](debugging_numerical_divergence.md).
 

scripts/phase4_review.py

@@ -0,0 +1,72 @@
+"""Registry-wide Phase-4 review: which verified models' stored scores predate
+the profile rework and deserve a re-run.
+
+phase4_score is a mixed-scale column: entries stamped p4_scoring_version=2
+were measured with the pinned-judge reference-ratio scoring (pass line 56);
+unstamped entries carry the old GPT-2 absolute-perplexity scale (pass line 85)
+and are never compared against the new line — they are re-run candidates.
+Read-only.
+"""
+
+import argparse
+
+from transformer_lens.benchmarks.text_quality_profiles import (
+    P4_SCORING_VERSION,
+    resolve_profile,
+)
+from transformer_lens.tools.model_registry.registry_io import load_supported_models_raw
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--below", type=float, default=None, help="Only scores below this")
+    parser.add_argument("--limit", type=int, default=None, help="Max rows per section")
+    args = parser.parse_args()
+
+    current: list = []
+    stale: list = []
+    for entry in load_supported_models_raw().get("models", []):
+        if entry.get("status") != 1 or entry.get("phase4_score") is None:
+            continue
+        score = entry["phase4_score"]
+        if args.below is not None and score >= args.below:
+            continue
+        profile = str(
+            resolve_profile(
+                entry["model_id"], entry.get("architecture_id"), entry.get("prompt_profile")
+            )
+        )
+        row = (profile != "continuation", score, entry["model_id"], profile)
+        if entry.get("p4_scoring_version") == P4_SCORING_VERSION:
+            current.append(row)
+        else:
+            stale.append(row)
+
+    # Profile-changed first, then ascending score: measurement changed most.
+    for rows in (current, stale):
+        rows.sort(key=lambda r: (not r[0], r[1]))
+    if args.limit:
+        current = current[: args.limit]
+        stale = stale[: args.limit]
+
+    print(
+        f"{len(current)} scored on the current scale (v{P4_SCORING_VERSION}); "
+        f"{len(stale)} on the old GPT-2 scale (re-run candidates)\n"
+    )
+    for title, rows in (
+        (f"v{P4_SCORING_VERSION} (reference-ratio scale, pass 56)", current),
+        ("v1 (GPT-2 scale — scores NOT comparable to the new pass line)", stale),
+    ):
+        if not rows:
+            continue
+        changed = sum(1 for r in rows if r[0])
+        print(f"== {title}: {len(rows)} models, {changed} non-default profiles")
+        print(f"{'score':>6}  {'profile':<28} model")
+        for is_changed, score, model_id, profile in rows:
+            marker = "*" if is_changed else " "
+            print(f"{score:6.1f}{marker} {profile:<28} {model_id}")
+        print()
+
+
+if __name__ == "__main__":
+    main()