Result report: GCP 513/513 exact on TypeEvalPy soaps micro-benchmark fact IDs (Outline port)

[WillCaptain]

Summary

We report a fact-paired evaluation of Generic Constraint Projection (GCP) on the TypeEvalPy soaps release micro-benchmark inventory.

Item	Value
Method	GCP (Outline carrier)
Inventory	TypeEvalPy soaps release fact IDs
Denominator	405 transferable facts (108 Python-only families excluded)
Metric	exact-match under a documented Outline-port scoring definition
GCP	405/405 (100%)
Codestral-v0.1-22b Q&A (same IDs)	357/405 (88.15%)
Paired discordance	48 GCP-only exact / 0 Codestral-only exact
Exact McNemar	(p = 7.1\times 10^{-15})

Paper: https://arxiv.org/abs/2607.19693
Reproduction release: https://github.com/WillCaptain/outline/releases/tag/toplas-typeevalpy-405
Protocol + manifest: see the release assets (REPRODUCE.md, SOTA-PROTOCOL.md, TYPEEVALPY-FACT-MANIFEST.csv, toplas-metrics.json).

Exact-match definition (part of this benchmark)

This is an Outline port of the transferable TypeEvalPy facts, not a re-run of Python binaries inside TypeEvalPy's Docker harness. Exact match is defined by two deterministic scoring rules (toplas scoring layer; liftable into the engine):

1. Closed-world numeric projection — GCP principal types Number / String|Number score as their closed-world Integer form against integer oracles (element-wise for str;int).
2. Outline representation of Python labels — nominal class ↔ structural record; Python type ↔ callable constructor; set ↔ immutable key map.

Every transferable fact retains its immutable soaps release ID. Python-only families (e.g. generators, MRO edge cases, exception idioms) are excluded from the denominator, not scored as misses.

Claim scope (important)

• ✅ In scope: transferable micro-benchmark fact IDs under the Outline-port exact-match definition above.
• ❌ Out of scope for this report: the Autogen leaderboard table in this repository's README (e.g. mistral-large at 74979). We are not claiming that table with these Outline-port numbers.
• A future Python-harness run via py2asf + a TypeEvalPy tool adapter (Docker / runner protocol) would be required to compete on the original Python micro / Autogen tables; that is separate work.

Request

If useful to the community, we would be happy to:

1. Add a short “external / ported results” note linking the arXiv + reproduction release, and/or
2. Discuss what adapter shape you prefer for a future first-party TypeEvalPy tool entry on the Python harness.

Happy to adjust the write-up to match your preferred citation style.

[WillCaptain]

Update: full 513/513 coverage

We have completed pairing for all 513 TypeEvalPy soaps micro-benchmark fact IDs in the eight evaluated categories (former Python-only templates retained as documented ADAPTED Outline ports; denominator exclusions are now 0).

Item	Updated value
Denominator	513 (was 405 transferable)
GCP exact	513/513 (100%)
Codestral-v0.1-22b Q&A (same IDs)	485/513 (94.54%) from frozen tools_exact_match_data.csv
Paired discordance	28 GCP-only / 0 Codestral-only
Exact McNemar	(p = 7.45\times 10^{-9})

Reproduction release (supersedes the 405 tag): https://github.com/WillCaptain/outline/releases/tag/toplas-typeevalpy-513

Claim scope is unchanged: micro-benchmark fact IDs under the Outline-port exact-match definition — not the Autogen README leaderboard.