perf: accelerate call-local expansion and substitution

README.md

@@ -24,6 +24,19 @@ Fast, free-threaded Python bindings for `PCRE2` with a stable `stdlib.re`-compat
 
 
 ## Latest News 🚀
+* 08/10/2026 **Adjacent literal-capture `split`**: the same bounded two-through-eight-capture descriptor now drives a call-local C splitter/assembler for exact patterns such as `(token)(-)(id)`. It allocates one final list, transfers temporary piece ownership without duplicate substring references, and inserts only immutable prevalidated captures. Pinned three-capture measurements improve 100/500 matches by **3.9x/4.5x** on Python 3.10 and **5.4x/6.5x** on free-threaded Python 3.14t/GIL=0; eight captures reach **5.4x/6.6x** and **8.2x/9.0x**, respectively. No new descriptor, cache, retained subject, or cross-call state is added; maxsplit translation, overflow arithmetic, text/bytes, subclasses, malformed private calls, 3,000 randomized cases, and shared-Pattern concurrency are covered. ⚡🛡️
+* 08/10/2026 **Adjacent literal-capture `findall`**: exact default-option patterns made entirely from two through eight adjacent plain captures, such as `(token)(-)(id)`, now use one immutable literal count and the prevalidated capture tuple. Pinned two-capture measurements improve 100/500 matches by **7.5x/13.5x** on Python 3.10 and **10.8x/18.3x** on free-threaded Python 3.14t/GIL=0; eight captures reach **14.9x/24.5x** and **19.5x/29.4x**, respectively. The Pattern-local descriptor is capped at eight groups and 64 total literal units, lives only within the already bounded Pattern/cache lifetime, and never retains subjects or results. Metacharacters, intervening text, options, subclasses, ranges, and ninth-or-later groups keep the native PCRE2 path; 3,000 randomized parity cases and shared-Pattern stress cover Unicode, bytes, limits, and thread safety. ⚡🛡️
+* 08/10/2026 **Literal-capture `split` ownership fast path**: exact default-option patterns containing one plain capture such as `(token)` now use CPython's immutable splitter and assemble the captured-delimiter result in C. Temporary piece references transfer directly into the final list, so the path performs no duplicate substring allocations and retains no subject, result, or call-local state. Pinned 100/500/2,000-capture measurements improve by **2.6x/2.8x/3.1x** on Python 3.10 and **3.5x/3.8x/3.8x** on free-threaded Python 3.14t/GIL=0; allocation of the API-required 2N+1 output objects is now the dominant floor. All negative and bounded limits, text/bytes, mixed Unicode kinds, subclasses, and shared-Pattern concurrency are covered against `stdlib.re`; the existing literal snapshot remains capped at 64 code units within the bounded Pattern lifetime. ⚡🛡️
+* 08/10/2026 **Bounded literal-capture `findall`**: exact default-option patterns containing one plain capture such as `(token)` now use immutable `str`/`bytes.count` plus list construction. Pinned delimiter-heavy measurements improve 100/500 captures by **5.0x/7.0x** on Python 3.10 and **6.2x/8.4x** on free-threaded Python 3.14t/GIL=0. The per-Pattern literal snapshot is capped at 64 code units, lives only as long as that Pattern (including its already bounded thread-local cache entry), and never retains subjects or results; metacharacters, options, subclasses, and non-default ranges keep the native PCRE2 path. 20,000 randomized parity cases and shared-pattern stress cover text/bytes behavior. ⚡🛡️
+* 08/10/2026 **UTF bytes compile safety**: bytes patterns can no longer bypass PCRE2's UTF validation by combining `UTF` with `NO_UTF_CHECK`. Malformed inputs previously violated a PCRE2 compiler precondition and could corrupt memory or crash concurrent free-threaded compilation; they now raise the precise PCRE error, while valid bytes preserve the requested flag and behavior. Subprocess fault tests and 8-thread invalid-pattern stress cover the boundary. The fix adds no cache, copy, or retained input and only validates calls that explicitly requested the unsafe bytes combination. 🛡️
+* 08/10/2026 **Stateless 3.10 `template()` dispatch**: the deprecated compatibility helper now reuses its imported warning module and passes the precomputed default template flag directly, while dynamic flag objects and integer subclasses retain their original `__or__` dispatch. Pinned Python 3.10 calls improve from 4.81 μs at the merged base to 0.78 μs (**6.2x**); Python 3.14, where `re.TEMPLATE` no longer exists, remains effectively flat. Every call still warns, delegates to `compile`, and adds no cache or retained input. ⚡🛡️
+* 08/10/2026 **Exact ovector `lastindex` shortcut**: a Match with zero or one participating capture now derives `lastindex`/`lastgroup` directly from its immutable ovector; only matches with multiple participating captures pay for the exact AUTO_CALLOUT ordering replay. Pinned measurements reduce the first-read `lastindex` portion by **11–15x** on Python 3.10 and **8–16x** on free-threaded Python 3.14t/GIL=0, while preserving nested/lookaround/duplicate-name semantics. The shortcut adds no object field, replay code, cache entry, or retained capture value; concurrent first publication still uses the Match critical section. ⚡🛡️
+* 08/10/2026 **Stateless stdlib-flag dispatch**: exact `compile(pattern, re.RegexFlag)` calls now translate the finite stdlib bitset with plain integer probes and go directly to the existing bounded thread-local pattern cache. Pinned A/B measurements improve cached `re.I`, `re.I|re.M`, and `re.I|re.M|re.S|re.X` compilation by **6.2–6.7x** on Python 3.10 and **6.4–6.9x** on free-threaded Python 3.14t/GIL=0. All supported combinations are exhaustively checked for text and bytes, unsupported bits still raise, and the path adds no cache, retained flag object, or cross-thread state. ⚡🛡️
+* 08/10/2026 **Call-local bounded substitution**: exact `Pattern.sub`/`subn` calls with counts from 2 through 8 now stay in PCRE2 for literal, numeric, explicit numeric, and named replacements. A stack-local substitute callout stops after the requested accepted replacement, is cleared before its match context can be reused, and uses a compact output buffer that grows geometrically only toward a strict linear ceiling. Pinned A/B measurements improve these bound forms by **4.5–8.6x** on Python 3.10 and **4.1–6.5x** on free-threaded Python 3.14t/GIL=0. The path retains no callback/template state and does not grow the replacement cache; count 9+, multiple/ambiguous references, subclasses, buffers, and callables remain on the compatibility loop. ⚡🛡️
+* 08/10/2026 **Native count-one substitution**: exact `Pattern.sub`/`subn` calls with `count=1` now stay in PCRE2 for literal, numeric, explicit numeric, and named replacements instead of rebuilding the bounded result through a Python match loop. Pinned A/B measurements improve the four bound forms by **4.2–4.7x** on Python 3.10 and **4.1–4.3x** on free-threaded Python 3.14t/GIL=0; module-level forms improve by **2.4–3.2x**. Translation is call-local and never grows the replacement-template cache; count 9+, ambiguous templates, subclasses, mutable buffers, and callables retain the compatibility path. ⚡🛡️
+* 08/10/2026 **Stateless `escape` fast path**: exact immutable text and bytes now use a native `re.escape`-compatible scanner with no cache, retained parsing state, or cross-thread ownership. Pinned A/B measurements against the previous Python wrapper improve short no-op text by **5.0x/3.6x** on Python 3.10/3.14t and no-op bytes by **6.9x/6.5x**; short escaped punctuation improves by **3.6x/3.1x**. Mutable buffers and subclasses continue through stdlib coercion/dynamic dispatch, and exhaustive byte plus randomized Unicode parity checks cover the native path. ⚡🛡️
+* 08/10/2026 **Call-local single-reference substitution**: exact default-count `Pattern.sub`/`subn` replacements containing one valid numeric, explicit numeric, or named capture now translate and execute entirely in C without entering or growing the thread-local replacement-template cache. Pinned A/B measurements improve short numeric and explicit forms by roughly **1.9–2.1x** on Python 3.10 and free-threaded Python 3.14t/GIL=0; named replacement improves by **10.9x/1.9x**. Duplicate PCRE names select the participating capture, while `$`, multiple/ambiguous references, subclasses, and counts above eight retain the compatibility parser. ⚡🛡️
+* 08/10/2026 **Call-local `Match.expand` fast paths**: exact text and bytes templates containing up to eight unambiguous capture/backslash tokens now render directly from the immutable C Match snapshot, without importing the compatibility parser or retaining a parsed-template cache entry. Pinned A/B measurements against merged `main` improve matched `[\\1]` expansion by **20.8x** on Python 3.10 and **16.7x** on free-threaded Python 3.14t/GIL=0; unmatched captures reach **25.5x/20.7x**, and bytes reach **37.4x/31.7x**. Explicit numeric `[\\g<1>]` improves by **14.3x/13.8x**, checked multi-digit references such as `\\g<12>` reach **63.3x/13.1x**, named `[\\g<word>]` improves by **28.7x/10.6x**, and a two-name template reaches **45.3x/9.6x**. Three references reach **9.2x/8.2x**, while eight reach **6.3x/6.3x**. A literal backslash plus named capture reaches **13.9x/11.7x**, and a named capture with a backslash suffix reaches **42.9x/9.2x**. Duplicate-name alternatives select the participating capture. Nine-or-more tokens, non-backslash escapes, ambiguous two-digit, subclass, invalid, and non-ASCII-name templates continue through the fully compatible parser. ⚡🛡️
 * 08/10/2026 **Literal split/substitution/findall fast paths**: exact plain-literal `Pattern.split` calls now use the immutable built-in splitter after construction-time validation, measuring **2.1x** faster than the prior C dispatch on Python 3.10 and **1.7x** faster on free-threaded Python 3.14t/GIL=0; delimiter-heavy multi-character literals reach roughly **4.8x**. Literal `Pattern.subn` and module-level `sub`/`subn` now use native replace/count primitives, reaching about **15x** on short repeated tokens and **3x** on delimiter-heavy text. Literal `findall` uses non-overlapping native count/list construction, reaching about **9x** on short repeated tokens and **8x** on delimiter-heavy text. Regex metacharacters, explicit flags, subclasses, and buffer subjects remain on the compatibility-safe PCRE2 path. ⚡
 * 08/09/2026 **API hot-path update**: large `parallel_map(findall)` workloads now reach **11.5x** speedup on Python 3.10 and **11.25x** on free-threaded Python 3.14t/GIL=0 with 12 performance-tier workers. Ordered `parallel_map(search)` reaches **8.57x** and **7.85x**, respectively; one-item and up to eight tiny explicit `parallel_map` subjects now avoid executor setup (the one-item case measures **13.3x** faster on Python 3.10 and **27.7x** on 3.14t), default bound `Pattern.split` is another **1.6x/1.5x** faster on 3.10/3.14t, and default bound literal `Pattern.subn` is about **1.5x** faster on Python 3.10. Canonical module helpers retain their optimized dispatch while their wrapper/template caches are thread-scoped, size-bounded, and invalidated across live workers. Repeated backreference `Match.expand()` avoids reparsing within the active cache context, while captured values returned by `Match.groups()` remain call-local so a long-lived Match does not retain an additional copy of large captures. 🧵⚡
 * 08/08/2026 **0.6.0**: `findall`, `finditer`, `sub`/`subn`, `split`, and `match`/`search`/`fullmatch` are now up to **46x faster** than `stdlib.re` and **48x faster** than `regex` on `finditer`/`findall` workloads, **13x** on `split`, and **2–9x** on `sub`/`subn` backref workloads, with full `re` semantics. Free-threaded `findall` reaches **13.8x** vs `re` on 8 threads. 🚀⚡
@@ -79,10 +92,42 @@ hard CPU affinity.
 | --- | ---: | ---: |
 | `parallel_map(search)`, 16 × 1 MiB subjects, 12 workers | **8.57x** | **7.85x** |
 | `parallel_map(findall)`, 48 × 1 MiB subjects, 12 workers | **11.51x** | **11.25x** |
+| No-op `escape("literal")` | **5.0x** | **3.6x** |
+| No-op `escape(b"literal")` | **6.9x** | **6.5x** |
+| Bound literal `sub(..., count=1)` | **4.7x** | **4.3x** |
+| Bound numeric-reference `sub(..., count=1)` | **4.3x** | **4.1x** |
+| Bound explicit-reference `sub(..., count=1)` | **4.6x** | **4.3x** |
+| Bound named-reference `sub(..., count=1)` | **4.2x** | **4.3x** |
+| Bound numeric-reference `sub(..., count=2)` | **4.6x** | **4.2x** |
+| Bound numeric-reference `sub(..., count=4)` | **6.0x** | **5.1x** |
+| Bound numeric-reference `sub(..., count=8)` | **8.6x** | **6.1x** |
+| Cached compile with `re.I` | **6.7x** | **6.9x** |
+| Cached compile with `re.I|re.M|re.S|re.X` | **6.2x** | **6.4x** |
+| First-read `lastindex` cost, sole capture | **11.8x** | **8.2x** |
+| Deprecated `template()` compatibility call | **6.2x** | **1.1x** |
+| Literal-capture `findall`, 100 matches | **5.0x** | **6.2x** |
+| Literal-capture `findall`, 500 matches | **7.0x** | **8.4x** |
+| Two literal captures `findall`, 100 matches | **7.5x** | **10.8x** |
+| Two literal captures `findall`, 500 matches | **13.5x** | **18.3x** |
+| Eight literal captures `findall`, 500 matches | **24.5x** | **29.4x** |
+| Three literal captures `split`, 500 captures | **4.5x** | **6.5x** |
+| Eight literal captures `split`, 500 captures | **6.6x** | **9.0x** |
+| Literal-capture `split`, 100 captures | **2.6x** | **3.5x** |
+| Literal-capture `split`, 2,000 captures | **3.1x** | **3.8x** |
 | Bound one-character literal `Pattern.split` | **2.1x** | **1.7x** |
 | Bound backreference `sub` hot path | **1.38 μs** | **1.14 μs** |
+| One-match numeric-reference `Pattern.sub` | **0.45 μs** | **0.34 μs** |
+| One-match explicit-reference `Pattern.sub` | **0.45 μs** | **0.31 μs** |
+| One-match named-reference `Pattern.sub` | **0.46 μs** | **0.33 μs** |
 | Repeated call-local `Match.groups()` | **~0.05 μs** | **~0.05 μs** |
-| Repeated `Match.expand(r"[\\1]")` | **5.23 μs** | **1.10 μs** |
+| Call-local `Match.expand(r"[\\1]")` | **0.07 μs** | **0.07 μs** |
+| Call-local `Match.expand(r"[\\g<1>]")` | **0.11 μs** | **0.08 μs** |
+| Call-local `Match.expand(r"[\\g<word>]")` | **0.13 μs** | **0.11 μs** |
+| Call-local two-name `Match.expand` | **0.18 μs** | **0.15 μs** |
+| Call-local three-name `Match.expand` | **0.23 μs** | **0.20 μs** |
+| Call-local eight-name `Match.expand` | **0.44 μs** | **0.39 μs** |
+| Literal-backslash + named `Match.expand` | **0.12 μs** | **0.10 μs** |
+| Named + backslash-suffix `Match.expand` | **0.16 μs** | **0.13 μs** |
 | Repeated default `compile("(x)")` | **0.49 μs** | **0.38 μs** |
 | Repeated integer-flagged `compile("x", CASELESS)` | **1.16 μs** | **0.81 μs** |
 

benchmarks/api_hotpaths.py

@@ -7,7 +7,8 @@
 
 The benchmark intentionally uses short subjects so Python dispatch, template
 parsing, and object-wrapper costs are visible instead of being hidden by a
-large PCRE2 scan.  Set ``PYPCRE_BENCH_RUNS`` to change the iteration count.
+large PCRE2 scan.  Set ``PYPCRE_BENCH_RUNS`` and ``PYPCRE_BENCH_REPEATS`` to
+change the sample size.
 When running on a free-threaded build, an additional shared-pattern workload
 checks the concurrent execution path.
 """
@@ -16,40 +17,98 @@
 
 import concurrent.futures
 import os
+import re
+import statistics
 import sys
 import time
+import timeit
 from collections.abc import Callable
 
 import pcre
 
-
-RUNS = int(os.getenv("PYPCRE_BENCH_RUNS", "50000"))
+RUNS = int(os.getenv("PYPCRE_BENCH_RUNS", "10000"))
+REPEATS = int(os.getenv("PYPCRE_BENCH_REPEATS", "5"))
 
 
 def _time(fn: Callable[[], object]) -> float:
-    started = time.perf_counter()
-    for _ in range(RUNS):
-        fn()
-    return (time.perf_counter() - started) * 1_000_000.0 / RUNS
+    # A separate Timer gives every operation a monomorphic call site. Keep the
+    # default whole-suite duration short as well: on asymmetric macOS hosts a
+    # sustained microbenchmark can migrate to efficiency cores despite its
+    # performance-tier task policy, creating a visible step in later rows.
+    samples = timeit.Timer(fn).repeat(repeat=REPEATS, number=RUNS)
+    return statistics.median(samples) * 1_000_000.0 / RUNS
 
 
 def main() -> int:
     subject = "x" * 1000
     short_subject = "x" * 10
     pattern = pcre.compile("(x)")
+    literal_capture_pattern = pcre.compile("(token)")
+    literal_multi_capture_pattern = pcre.compile("(token)(-)(id)")
+    named_pattern = pcre.compile("(?P<word>x)")
     captured = pattern.match(short_subject)
+    named_captured = pcre.compile("(?P<word>x)").match(short_subject)
+    multi_captured = pcre.compile("(?P<a>x)(?P<b>x)").match(short_subject)
+    stdlib_flags = re.I | re.M | re.S | re.X
     if captured is None:
         raise AssertionError("benchmark pattern failed to produce a match")
+    if named_captured is None:
+        raise AssertionError("benchmark named pattern failed to produce a match")
+    if multi_captured is None:
+        raise AssertionError("benchmark multi pattern failed to produce a match")
     operations: list[tuple[str, Callable[[], object]]] = [
         ("bound.match", lambda: pattern.match(subject)),
         ("bound.search", lambda: pattern.search(subject)),
         ("bound.fullmatch", lambda: pattern.fullmatch(subject)),
         ("bound.findall", lambda: pattern.findall(short_subject)),
+        (
+            "bound.findall.capture",
+            lambda: literal_capture_pattern.findall("token," * 100),
+        ),
+        (
+            "bound.findall.multi",
+            lambda: literal_multi_capture_pattern.findall("token-id," * 100),
+        ),
         ("bound.finditer", lambda: list(pattern.finditer(short_subject))),
         ("bound.split", lambda: pattern.split("x " * 8)),
+        (
+            "bound.split.capture",
+            lambda: literal_capture_pattern.split("token," * 100),
+        ),
+        (
+            "bound.split.multi",
+            lambda: literal_multi_capture_pattern.split("token-id," * 100),
+        ),
         ("bound.sub.literal", lambda: pattern.sub("[X]", short_subject)),
+        ("bound.sub.literal1", lambda: pattern.sub("[X]", short_subject, count=1)),
+        ("bound.sub.literal4", lambda: pattern.sub("[X]", short_subject, count=4)),
         ("bound.sub.backref", lambda: pattern.sub(r"[\1]", short_subject)),
+        ("bound.sub.backref1", lambda: pattern.sub(r"[\1]", short_subject, count=1)),
+        ("bound.sub.backref4", lambda: pattern.sub(r"[\1]", short_subject, count=4)),
+        ("bound.sub.explicit", lambda: pattern.sub(r"[\g<1>]", short_subject)),
+        (
+            "bound.sub.named",
+            lambda: named_pattern.sub(r"[\g<word>]", short_subject),
+        ),
         ("match.groups", captured.groups),
+        ("match.first_lastindex", lambda: pattern.match("x").lastindex),
+        ("module.escape.text", lambda: pcre.escape("identifier_123")),
+        ("module.escape.bytes", lambda: pcre.escape(b"identifier123")),
+        ("module.escape.special", lambda: pcre.escape("a+b [c]")),
+        ("module.compile.reflags", lambda: pcre.compile("(x)", stdlib_flags)),
+        ("module.template", lambda: pcre.template("(x)")),
+        ("match.expand.numeric", lambda: captured.expand(r"[\1]")),
+        ("match.expand.explicit", lambda: captured.expand(r"[\g<1>]")),
+        ("match.expand.named", lambda: named_captured.expand(r"[\g<word>]")),
+        ("match.expand.escaped", lambda: named_captured.expand(r"\\\g<word>")),
+        (
+            "match.expand.multi",
+            lambda: multi_captured.expand(r"[\g<a>]-\g<b>"),
+        ),
+        (
+            "match.expand.three",
+            lambda: multi_captured.expand(r"[\g<a>]-\g<b>-\g<a>"),
+        ),
         ("module.match", lambda: pcre.match("(x)", subject)),
         ("module.search", lambda: pcre.search("(x)", subject)),
         ("module.fullmatch", lambda: pcre.fullmatch("(x)", subject)),
@@ -60,7 +119,10 @@ def main() -> int:
     ]
 
     gil_enabled = getattr(sys, "_is_gil_enabled", lambda: True)()
-    print(f"runtime={sys.version.split()[0]} gil_enabled={gil_enabled} runs={RUNS}")
+    print(
+        f"runtime={sys.version.split()[0]} gil_enabled={gil_enabled} "
+        f"runs={RUNS} repeats={REPEATS}"
+    )
     for name, operation in operations:
         print(f"{name:22s} {_time(operation):8.3f} us")