maint: add a match throughput benchmark driver - #949
Conversation
carenas
left a comment
carenas
left a comment
There was a problem hiding this comment.
FWIW for portability we use clock() in pcre2test, but since this is a "maint" tool this is likely better.
the build script is IMHO not needed, as you could just put a comment of how to compile it, like we use in ucptest, but it doesn't do harm either.
IMHO, it would be better if it would had been PCRE2_CODE_UNIT_WIDTH agnostic.
|
Thanks Matt! This is another PR where I think I'll just pause it for a week until after 10.48 is out. But many thanks, I'll look at it soon |
Totally fine. Absolutely no rush from my side. Thanks! |
Add pcre2bench.c, which scans a generated pseudo-text corpus with a set of patterns chosen to separate the JIT's start-of-match fast-forward paths from everything else, and reports best-of-five throughput as TSV. The corpus comes from a fixed xorshift64* seed, so it is bit-identical on every host and build, and the per-pattern match count is reported so that two builds can be checked for identical results. The code unit width is chosen at compile time and defaults to 8. All the text in the driver is ASCII, and the corpus is a fixed number of code units rather than of bytes, so the same corpus is generated at every width and the match counts are identical between them. Throughput is therefore reported in code units rather than in bytes, and the width is an output column. No build system builds it, following the precedent of ucptest.c. RunBenchmark compiles it against an existing CMake or autotools build tree and runs it; it takes -8, -16 and -32 to choose the width, as pcre2test does. The corpus capitalizes the first letter of one word in 32. Without that, the subject is entirely lowercase, and pcre2_match() then hits a pathological case on caseless patterns: it searches for the first code unit with one memchr call per case, the uppercase call runs to the end of the subject every time because it never finds anything, and its result is cached only for the duration of the call. A find-all-matches loop therefore becomes quadratic. The comment in build_subject() records the measurement.
cd7e1e7 to
c79d58e
Compare
|
All three taken. Width: the driver is now Build script: added the compile line to the header comment, as clock(): left as |
3 participants
Adds a benchmark driver for match throughput,
maint/pcre2bench.c, plusmaint/RunBenchmarkto compile and run it.Why
Work on the JIT's SIMD fast-forward paths for a new architecture needs a
yardstick that is the same from one architecture to the next, and that says
which of the
JIT_HAS_FAST_*hooks a given change actually moves. I wrote thiswhile adding an Alpha SIMD block, where the question under review was how much
of the new code paid for itself; the answer came from disabling each hook in
turn and re-running. That is a question anyone adding a fast-forward
implementation will have to answer, so the driver seemed worth sharing rather
than keeping locally.
What it does
Builds a 4MB deterministic pseudo-text corpus and scans it end to end with 18
patterns, reporting best-of-five throughput as TSV, one row per pattern. The
patterns separate the fast-forward paths from everything else: absent first
code unit (pure fast-forward cost, no false starts), character ranges, ranges
at a relative offset, caseless, character pairs, a few realistic mixed
patterns, and two controls that use no fast-forward at all as a noise floor.
A
shortmode slices the same corpus into 8 to 512 byte runs and reportsper-call cost instead, which is where the fixed prologue and loop-setup cost
shows up. A
nojitmode runs the interpreter for comparison.The corpus comes from a fixed xorshift64* seed, so it is bit-identical on every
host and build. Every row carries its match count, and the driver aborts if the
count moves between repeats, so a build under test can be checked against a
reference for identical results rather than only for speed.
Typical use:
Decisions I would rather flag than have you find
Not wired into any of the four build systems.
maint/ucptest.csets theprecedent of a maintainer C program that you compile by hand, and adding a
target to autotools, CMake, Bazel and Zig for a tool like this seemed a poor
trade.
RunBenchmarklocates the generatedpcre2.handlibpcre2-8under agiven build directory, handling both the CMake and autotools layouts, so nobody
has to get the include and link flags right by hand.
Not in CI, and I do not think it should be. Shared runners are noise
dominated. For calibration, on a dedicated idle machine I measured up to 17.6%
run-to-run spread on the noisiest pattern in this set. Anything CI could report
from that would be false signal.
Will not build on MSVC. It uses
clock_gettime(CLOCK_MONOTONIC), whichUCRT does not provide.
RunBenchmarkis/bin/sh, so the tool is Unix-only inpractice anyway, but
maint/does carry.ps1scripts, so I would rather sayso than let you assume otherwise. Happy to add a
QueryPerformanceCounterbranch if you want it.
RunBenchmarkpasses-Wl,-rpath. Fine on ELF and Mach-O, not universal.It only matters for shared builds.
Impact on the tree
None outside
maint/.maint/is not inEXTRA_DIST, so these files do notreach the tarball and
manifest-tarballis unaffected; theRunManifestTestinvocations in CI check install directories, not the source tree. No job
compiles or lints
maint/contents. No ChangeLog entry, matching the otherrecent infrastructure commits.
Testing
Built the driver
-Wall -Wextraclean and ran all four modes against an x86-64static JIT build. Match counts are stable across repeats on every pattern, and
back-to-back runs of the same binary agree to within 1.3%.