Optimize Lexer::readString by OpaqueRock · Pull Request #1948 · webonyx/graphql-php

OpaqueRock · 2026-07-16T10:26:24Z

Summary

This PR contains two related performance optimizations for Lexer::readString() and Visitor::visitInParallel():

Lexer::readString(): bulk-scan runs of "boring" bytes with strcspn() instead of decoding/validating one UTF-8 character at a time via readChar(). Falls back to the existing per-character logic only for quotes, line terminators, control characters, and escape sequences. TAB (0x09) is excluded from the stop-byte set, since it's a legal, unescaped SourceCharacter that can simply be scanned through like any other ordinary character.
Visitor::visitInParallel():
- Precompute the enter/leave callback per (visitor, kind, direction) once per call, instead of re-resolving it via extractVisitFn() on every single node for every wrapped visitor.
- Replace func_get_args() + argument unpacking in the enter/leave dispatcher closures with the five explicit, typed parameters they are always called with (verified against Visitor::visit()'s single call site), avoiding func_get_args()'s per-call overhead.

Why

Parsing and validating GraphQL documents is on the hot path for every request in a GraphQL server. readString() and visitInParallel() (used heavily during validation, where many rules are visited together in one pass) are both called very frequently, so their per-call overhead compounds across large schemas/queries.

Testing

Full existing test suite passes: composer test (1990 tests, 21728 assertions - no failures; the 1 pre-existing warning is from an unrelated, intentional ExecutorLazySchemaTest warning-assertion test).
composer stan (PHPStan) passes with no errors.
composer php-cs-fixer passes with no remaining issues.
composer rector (dry-run) reports no changes needed in any file touched by this PR.
New regression test (testLexesStringsWithUnescapedTabCharacter) covering the TAB case mentioned above.
Manually verified via an isolated correctness test corpus (literal TAB in various positions, all standard escape sequences, multi-byte UTF-8 content, and boundary/error cases such as unterminated strings, invalid control characters, and invalid escape sequences) that output is byte-identical to the unpatched implementation across all cases, and that Visitor::visitInParallel()'s traversal order/results are unchanged.

Benchmarks

Measured with an isolated benchmark harness (not included in this PR) that repeatedly parses and visits a synthetic ~19KB query containing a mix of plain, escaped, and Unicode string literals, 500 iterations each, with Xdebug disabled (PHP 8.3.6, x86_64). Compared directly against the current upstream/master (3 runs each, averaged):

Metric	master (avg)	This PR (avg)	Time reduced	Speedup
`Parser::parse` (Lexer)	10,907 us/op	4,026 us/op	63.1%	2.71x
`Visitor::visit` (parallel)	6,670 us/op	4,023 us/op	39.7%	1.66x
Combined total (1000 ops)	8.79s	4.02s	54.2%	2.18x

Real-world gains will vary by query/schema shape and string content; this benchmark specifically isolates the two changed functions from surrounding framework overhead. The Lexer::readString() improvement scales with string literal length, so queries containing long inline string literals (e.g. large text blocks) should see a proportionally larger benefit than this benchmark's average.

Risk assessment

No increase in memory usage: the optimized code uses the same substr()/string-concatenation approach as before, just with fewer, larger operations instead of many single-character ones.
No new unbounded loops, recursion, or additional failure modes were introduced.
Performance is not worse than baseline for any input shape tested, including inputs designed to defeat the bulk-scan fast path (e.g. strings consisting entirely of escape sequences or control characters).

Lexer::readString previously decoded and validated one UTF-8 character at a time via Lexer::readChar(). Replace this with a bulk scan using strcspn() to copy runs of bytes that need no special handling (i.e. anything other than a quote, backslash, or control character) in a single native call, falling back to the original per-character logic only for quotes, line terminators, control characters, and escape sequences. Care is taken to keep $this->position (a decoded-character count, not a byte count) accurate for multi-byte UTF-8 content. TAB (0x09), a legal, unescaped SourceCharacter per the GraphQL spec, is excluded from the stop-byte set so it is scanned through like any other ordinary character instead of being individually inspected. Visitor::visitInParallel previously called extractVisitFn() to resolve the enter/leave callback for the current visitor and node kind on every single node, for every wrapped visitor - an O(nodes x visitors) number of array lookups, even though a callback's identity for a given (visitor, kind, direction) triple never changes during a traversal. Since NodeKind exposes a small, fixed set of kinds, precompute this mapping once per call (O(visitors x kinds)) instead. Additionally, replace func_get_args() plus argument unpacking in the enter/leave dispatcher closures with the five explicit, typed parameters they are always called with (matching Visitor::visit()'s single call site), avoiding func_get_args()'s per-call overhead. Add a regression test for the TAB-handling edge case described above.

Copilot

Pull request overview

This PR introduces targeted performance optimizations in two hot-path components of the library: lexing string literals and running multiple validation visitors in a single traversal.

Changes:

Optimize Lexer::readString() by bulk-scanning “ordinary” bytes via strcspn() and only falling back to per-character handling for escape/control/terminator cases (explicitly allowing unescaped TAB per spec).
Optimize Visitor::visitInParallel() by precomputing per-(visitor, kind, direction) callbacks once per traversal and avoiding func_get_args()/argument unpacking in the dispatcher.
Add a regression test ensuring strings containing an unescaped TAB are lexed correctly.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File	Description
tests/Language/LexerTest.php	Adds regression coverage for unescaped TAB in string literals.
src/Language/Visitor.php	Reduces per-node overhead in parallel visitor dispatch by caching callbacks and avoiding `func_get_args()`.
src/Language/Lexer.php	Speeds up string literal lexing by scanning runs of non-special bytes in bulk while preserving escape/control handling.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

The enter/leave callback caches were previously prefilled only for NodeKind constants, silently skipping generic and kind-specific visitors for Nodes with custom (non-NodeKind) kinds. Switch to lazy, per-direction caches that resolve and store a callback (or `false` as a "no callback" sentinel) on first use, keyed by any kind string. Also fixes a pre-existing undefined-array-key warning in visit() when traversing a Node with a custom kind. Add regression tests covering generic and kind-specific visitors for a custom Node kind.

spawnia

Are those two independent changes? Please split if so.

@var

- Use a HEREDOC for the tab-character test source, deriving the expected value from it via substr() so input and expectation can't drift apart. - Type-hint $path/$ancestors as array in visitInParallel()'s enter/leave closures, and document $key/$parent via @phpstan-param since PHP 7.4 doesn't support union types. - Add @var docblocks describing the shape of the enter/leave callback caches. - Replace the lazily-built $stopBytes runtime computation in Lexer::readString() with a precomputed STRING_STOP_BYTES constant.

The visit() undefined-array-key warning fix for custom Node kinds, along with its regression tests, has been split out into a separate PR (webonyx#1952) since it is unrelated to the Lexer/Visitor optimization work here.

OpaqueRock · 2026-07-22T14:47:29Z

Thanks for the review! I've pushed two updates:

Addressed the requested changes: converted the tab-character Lexer test to a HEREDOC, added strict param typing (array for $path/$ancestors) plus @phpstan-param docblocks for the enter/leave closures in visitInParallel() (union types aren't available since this targets PHP 7.4), added @var docblocks for the $enterFns/$leaveFns caches, and replaced the runtime-computed $stopBytes string with a STRING_STOP_BYTES class constant.
Split out the unrelated visit() fix: per your suggestion, the visit() undefined-array-key fix for custom Node kinds (and its two regression tests) is no longer part of this PR — it's now a standalone PR: Fix undefined-array-key warning in visit() for custom Node kinds #1952.

This PR is now scoped solely to the Lexer::readString() / Visitor::visitInParallel() optimizations. Let me know if anything else needs adjusting.

spawnia · 2026-07-22T15:12:42Z

I would like the Lexer and Visitor optimizations as separate PRs.

The Visitor::visitInParallel() enter/leave callback caching optimization has been split out into a separate PR (webonyx#1953), so it can be reviewed independently of the Lexer::readString optimization here.

OpaqueRock · 2026-07-22T17:27:47Z

The Visitor::visitInParallel() optimization has been now split out into its own PR, #1953

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

PHP's switch uses loose comparison, so `case null` also matches code 0, causing a backslash followed by a raw NUL byte to be treated as EOF instead of an invalid escape sequence. Check for EOF with a strict comparison before the switch instead.

OpaqueRock · 2026-07-24T04:16:59Z

I looked into the null value comparison and it's a real, reachable bug, though it pre-dates this PR (the switch/case null block existed identically on master; this PR just moved/reformatted it).

The problem: switch in PHP uses loose (==) comparison for each case. So case null: also matches integer 0, not just true EOF. In readString(), after consuming a backslash, $code is 0 for a literal NUL byte (0x00) in the input, and null only on genuine EOF. Because of the loose comparison, a literal NUL byte after a backslash was being misidentified as EOF.

Impact: For input containing \ followed by a raw NUL byte inside a string literal, the lexer silently dropped both bytes and kept parsing, instead of raising a syntax error. E.g. "ab\<NUL>cd" produced the token value 'abcd' with no error at all.

Per the GraphQL spec, a NUL byte is not a valid raw string character and has no dedicated escape (\0 isn't in the EscapedCharacter grammar) — the only spec-compliant way to include it is \u0000. So \<NUL> should always be rejected as an invalid escape sequence, the same as \q or any other unrecognized escape.

Fix: Added an explicit if ($code === null) { break; } strict-comparison check immediately before the switch, and removed the now-dead case null: branch inside the switch. Considered using PHP 8's match expression instead, but the project's floor is PHP 7.4, so that's not available; the strict pre-check is also simpler here since the \u case has multi-statement logic that doesn't map cleanly to match anyway.

Added a regression test (reportsUsefulStringErrors in LexerTest.php) covering \ followed by a raw NUL byte, verified it failed before the fix and passes after.

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

spawnia

Comment.

The PR is a sound performance optimization with no correctness bugs. The three surviving comments are minor improvements: a simplification opportunity (mb_strlen over preg_match_all), an open reviewer request for named constants, and a test hygiene suggestion. None rise to the level of requesting changes.

spawnia · 2026-07-24T09:28:32Z

+                $continuationBytes = preg_match_all('/[\x80-\xBF]/', $chunk);
+                assert(is_int($continuationBytes), 'Pattern is valid, so preg_match_all() cannot fail');
+                $this->position += strlen($chunk) - $continuationBytes;


The three-line preg_match_all + assert + arithmetic block can be replaced with a single mb_strlen call that returns the codepoint count directly. For valid UTF-8, mb_strlen($chunk, 'UTF-8') is semantically equivalent to strlen($chunk) - count(continuation bytes), avoids PCRE overhead, and makes the intent ("count codepoints") explicit.

Suggested change

$continuationBytes = preg_match_all('/[\x80-\xBF]/', $chunk);

assert(is_int($continuationBytes), 'Pattern is valid, so preg_match_all() cannot fail');

$this->position += strlen($chunk) - $continuationBytes;

$this->position += mb_strlen($chunk, 'UTF-8');

spawnia · 2026-07-24T09:28:33Z

-            if ($code === 92) { // \
-                $value .= $chunk;
-                [, $code] = $this->readChar(true);
+            if ($code === 10 || $code === 13) { // LineTerminator


Open question: could extracting 10 (LF) and 13 (CR) into named constants have a performance cost in such a hot path? Either way, whichever approach we go with should be consistent across the file — right now it's magic numbers everywhere.

spawnia · 2026-07-24T09:28:33Z

+                'end' => strlen($source),
+                'value' => substr($source, 1, -1), // strip the surrounding quotes


The expected end and value are derived from $source at assertion time (strlen, substr). If the test input were wrong, the assertion would be wrong in the same direction and the test would still pass. Since the source is a fixed 14-byte literal ("before<TAB>after"), both values can be hardcoded.

Suggested change

'end' => strlen($source),

'value' => substr($source, 1, -1), // strip the surrounding quotes

'end' => 14,

'value' => "before\tafter",

spawnia · 2026-07-24T09:28:47Z

Reviewed at 031cb52 (standard, full)

🤖 Posted by Claude Code

OpaqueRock and others added 2 commits July 16, 2026 13:00

Merge remote-tracking branch 'origin/master' into pr-1948-OpaqueRock

ef0969c

spawnia requested a review from Copilot July 21, 2026 13:14

Copilot started reviewing on behalf of spawnia July 21, 2026 13:15 View session

Copilot AI reviewed Jul 21, 2026

View reviewed changes

spawnia requested a review from Copilot July 21, 2026 13:35

Copilot started reviewing on behalf of spawnia July 21, 2026 13:36 View session

Copilot AI reviewed Jul 21, 2026

View reviewed changes

Comment thread src/Language/Visitor.php Outdated

Comment thread src/Language/Visitor.php Outdated

spawnia reviewed Jul 22, 2026

View reviewed changes

Comment thread tests/Language/VisitorTest.php Outdated

Comment thread tests/Language/LexerTest.php Outdated

Comment thread src/Language/Visitor.php Outdated

Comment thread src/Language/Visitor.php Outdated

Comment thread src/Language/Lexer.php Outdated

OpaqueRock mentioned this pull request Jul 22, 2026

Fix undefined-array-key warning in visit() for custom Node kinds #1952

Open

Remove visit() custom-kind bugfix, split into webonyx#1952

cf27f7e

The visit() undefined-array-key warning fix for custom Node kinds, along with its regression tests, has been split out into a separate PR (webonyx#1952) since it is unrelated to the Lexer/Visitor optimization work here.

OpaqueRock mentioned this pull request Jul 22, 2026

Optimize Visitor::visitInParallel #1953

Open

Remove Visitor optimization, split into webonyx#1953

65089c9

The Visitor::visitInParallel() enter/leave callback caching optimization has been split out into a separate PR (webonyx#1953), so it can be reviewed independently of the Lexer::readString optimization here.

OpaqueRock changed the title ~~Optimize Lexer::readString and Visitor::visitInParallel~~ Optimize Lexer::readString Jul 22, 2026

spawnia requested a review from Copilot July 23, 2026 07:29

Copilot started reviewing on behalf of spawnia July 23, 2026 07:30 View session

Copilot AI reviewed Jul 23, 2026

View reviewed changes

Comment thread src/Language/Lexer.php Outdated

spawnia requested a review from Copilot July 24, 2026 08:12

Copilot started reviewing on behalf of spawnia July 24, 2026 08:13 View session

Copilot AI reviewed Jul 24, 2026

View reviewed changes

Comment thread tests/Language/LexerTest.php

spawnia reviewed Jul 24, 2026

View reviewed changes

		'end' => strlen($source),
		'value' => substr($source, 1, -1), // strip the surrounding quotes

Uh oh!

Uh oh!

Conversation

OpaqueRock commented Jul 16, 2026

Summary

Why

Testing

Benchmarks

Risk assessment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

spawnia left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

OpaqueRock commented Jul 22, 2026

Uh oh!

spawnia commented Jul 22, 2026

Uh oh!

OpaqueRock commented Jul 22, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

OpaqueRock commented Jul 24, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

spawnia left a comment

Choose a reason for hiding this comment

Uh oh!

spawnia Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

spawnia Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

spawnia Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

spawnia commented Jul 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants