Report invalid UTF-8 byte in a comment as a parsing error#3026
Open
soutaro wants to merge 5 commits into
Open
Conversation
PR #2973 fixed the lexer hang on an invalid UTF-8 byte by advancing one byte, but a byte inside a comment was then silently swallowed (comments scan until newline or EOF), so a malformed comment parsed successfully. Map an invalid byte to a sentinel code point (U+FFFD) and exclude that sentinel from the comment rule's character class. The comment now stops at the invalid byte, which the catch-all rule turns into an ErrorToken, so the parser reports a normal ParsingError. Valid input is unaffected: valid multibyte characters map to the existing dummy code point and valid single bytes to their own value, so U+FFFD is never produced by valid input. src/lexer.c is regenerated with re2c 4.3. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A real U+FFFD (REPLACEMENT CHARACTER) is a valid 3-byte UTF-8 sequence
("\xEF\xBF\xBD") that decodes to the multibyte dummy code point, not the
sentinel code point used to mark an invalid byte. Guard that a comment
containing it still parses without error.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01My4C2J8ssB4YvrckjswLK2
soutaro
force-pushed
the
claude/rbs-pr-2983-review-xau5fm
branch
from
July 17, 2026 12:45
d846727 to
ac09cae
Compare
test_invalid_utf8_byte_in_comment_does_not_hang now asserts that an invalid UTF-8 byte in a comment raises RBS::ParsingError. Confirmed via CI (truffleruby job on the omit-less head failed on exactly this test, 1 failure): as already documented for test_invalid_utf8_byte_at_top_level_raises, the C extension does not raise RBS::ParsingError for an invalid UTF-8 byte on TruffleRuby, so omit this test there as well. JRuby builds rbs_parser.wasm from source in CI (rake wasm:jruby_setup), so it picks up the lexer fix and needs no omit. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01My4C2J8ssB4YvrckjswLK2
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Based on #2983 by @ksss (its commits are included here, rebased on current
master), with two follow-up additions from review.What #2983 does
Follow-up to #2973. An invalid UTF-8 byte that falls inside a comment used to be silently swallowed (comments scan until newline/EOF), so
# \xC2parsed successfully. The lexer is code-point based, so the fix gives an invalid byte a sentinel code point (U+FFFD) and excludes it from the comment rule's character class:src/lexstate.c: an invalid byte (char_width == 0) now maps toU+FFFDinstead of its raw value.src/lexer.re: the comment rule becomes"#" (. \ [\x00�])*, so the comment stops at the invalid byte; the catch-all rule turns it into anErrorToken, reported as a normalRBS::ParsingError.U+FFFDis safe as a sentinel because valid input never produces it as a code point: valid multibyte characters map to the existing dummy code point (12523) and valid single bytes map to0..255. A genuineU+FFFDin the input is a 3-byte sequence that decodes to the dummy code point, not to0xFFFD.Additions in this PR
test_utf8_replacement_character_in_comment_parses— guards the invariant the sentinel relies on: a genuineU+FFFD("\xEF\xBF\xBD", a valid 3-byte UTF-8 sequence) in a comment must still parse, because it decodes to the multibyte dummy code point, not the invalid-byte sentinel.omit_on_truffle_ruby!ontest_invalid_utf8_byte_in_comment_does_not_hang— that test now assertsRBS::ParsingError, and as already documented fortest_invalid_utf8_byte_at_top_level_raises, the C extension does not raise for an invalid UTF-8 byte on TruffleRuby. JRuby buildsrbs_parser.wasmfrom source in CI (rake wasm:jruby_setup), so it picks up the fix and needs no omit.Testing
Parser suite locally on CRuby 3.3.6: 43 tests, 398 assertions, 0 failures.
🤖 Generated with Claude Code
Generated by Claude Code