ParseFile returns the tree with the input's comments

[kyleconroy]

A formatter cannot put comments back where they were written without being told where that was, and the grammar is the wrong place to ask: libpg_query's patch 04 has the scanner emit comments as tokens, and base_yylex drops them on the way to the parser.

Scan exposes that channel, so Scan + Parse already answers the question — by lexing the input twice and marshalling the entire token stream to protobuf to recover a handful of comments:

on a six-statement query file	time	allocs
Scan + Parse	138 µs	689
parser.ParseFile	70 µs	444
Parse alone (no comments)	61 µs	428

ParseFile keeps the comments from the pass the parse already makes and returns them beside the tree, as the ScanToken values Scan would have reported:

type ParseFileResult struct {
	*ast.ParseResult
	Comments []*ast.ScanToken // source order; text is input[Start:End]
}

This is oliphant's second deliberate addition to pg_query_go's surface, alongside ParseToTree from #11, and like it lives in the parser subpackage so the root package stays a mirror function-for-function. PLAN.md § 1 and CLAUDE.md's hard rule now record both.

Where the comments are collected

At the one site every token is minted (Scanner.token), not at the filter's drop site. A successful parse does route every comment through that drop site — I checked, including comments picked up as base_yylex lookahead — but not every path there goes through Filter.Next: the UIDENT/UESCAPE resolution pulls tokens off to the side. The mint site cannot be bypassed at all.

Collection is opt-in through NewKeepingComments, and the parse path is unchanged when it is off. Interleaved runs of the existing parse benchmarks (before/after binaries alternated, to cancel drift on a noisy box) show overlapping distributions — ParseSelect2 6993–8018 ns before, 7053–8298 ns after; ParseCreateTable 17159–20246 ns before, 16932–21515 ns after.

Agreeing with Parse on UTF-8

ParseFile accepts and rejects exactly what Parse does, the proto3 UTF-8 rejection #11 kept included.

That answer could not come from checking the input. An invalid byte fails only where it lands in a string field, and a comment is not one — SELECT 1 -- \xff marshals and Parse returns its tree — and a comment is precisely what ParseFile must not reject one for. So it decides on the tree, marshalling it only when utf8.ValidString(input) is already false, which is the same guard Parse takes its own fallback on and costs nothing on valid input.

Testing

The corpus is what pins the result. TestParseFileMatchesScan runs every case in the parse suite:

• across all 46,756 cases that parse, ParseFile's tree is proto.Equal to ParseToTree's and the two agree on every error;
• its comments are exactly the SQL_COMMENT/C_COMMENT tokens Scan reports for the same input — 11,880 cases carry at least one, so the oracle-derived scan goldens stand behind the comment spans too;
• comments come back ordered, disjoint, and inside the input.

Focused tests cover the positions a query file puts comments in (above a statement, trailing one on its terminator's line, inside one, between two, and after the last — that one is what a parse could plausibly miss, since nothing follows it but end of input), nested block comments arriving as the single token PostgreSQL scans them as rather than ending at the inner */, a failed parse returning no half-scanned list, and the UTF-8 agreement above in both directions.

go test ./... is green. Checked against the real consumer as well: with a replace pointing here, sqlc builds and its whole TestReplay/base corpus passes — worth doing since this one touches the scanner.

Next

The sqlc side: ParseFile on postgresql.Parser returning *ast.File, a case in newQueryFormatter, and flipping testdata/fmt/postgresql from its skip notice to a real golden. That needs sqlc-dev/sqlc#4580 merged first.