Skip to content

fix(editor): keep reading the keyword behind an invisible character - #3025

Merged
datlechin merged 1 commit into
mainfrom
fix/classifier-invisible-characters
Sep 20, 2026
Merged

datlechin merged 1 commit into
mainfrom
fix/classifier-invisible-characters

Conversation

@datlechin

Copy link
Copy Markdown
Member

#3005 moved the classifier onto the shared lexer's code projection, and the new leading-keyword scan skipped only Character.isWhitespace. The old one skipped everything StatementBlank calls blank, which is what #2729 added so a zero-width space or a byte-order mark in front of a statement could not hide its keyword. So \u{FEFF}SELECT 1 stopped reading as a SELECT and fell to the write arm, and QueryClassifierInvisibleCharacterTests and QueryDiagnosticsRefreshTests have failed on main since.

The scan drops StatementBlank.isBlank again, so a leading invisible character is passed over and a visible combining mark still ends it (markedBlankIsNotSkipped).

The other change is a test #3005 wrote that contradicts the engine. It expected SELECT<NBSP>$$a b$$ to hold a dollar-quoted literal. Measured on PostgreSQL 17, that text is a syntax error: every byte over 0x7F continues an identifier there, so a $ glued to one opens no literal, which is what the lexer already says. The test now asserts the measured behaviour, and a second case covers a dollar-quoted body after an ordinary space, which is kept.

QueryClassifierInvisibleCharacterTests, InvisibleCharacterRemoverTests, QueryClassifierTests, QueryClassifierLexicalTests, QueryClassifierPLSQLTests, QueryDiagnosticsRefreshTests, SQLConfusableCharacterScannerTests, ExternalStatementGateLexicalTests, MCPStatementGateTests, SQLStatementScannerTests and CatalogChangeClassifierTests: 256 cases, all passing.

@datlechin
datlechin merged commit 7c03b36 into main Sep 20, 2026
6 of 8 checks passed
@datlechin
datlechin deleted the fix/classifier-invisible-characters branch September 20, 2026 15:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant