diff --git a/README.md b/README.md index 08231bc..11c3b17 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,220 @@ # sql-query-engine -A small SQL query engine in Go: parser, volcano-model executor, and a hash join. Query CSV files from a REPL. +A small SQL query engine in Go. It lexes and parses a `SELECT` subset, plans it, +and executes it with a volcano (pull-based) operator model including a real hash +join. Tables are CSV files described by a small schema; you query them from a +REPL. -Work in progress. +**What it is:** a readable, tested implementation of how a query gets from text +to rows — parsing, planning, and execution — with its results checked against +SQLite on tens of thousands of generated queries. + +**What it is not:** a database. There is no storage engine, no persistence, no +transactions, no indexes, and no query optimizer. See [Non-goals](#non-goals). + +## Features + +- **Volcano execution model** — every operator pulls rows from its child through + `Next()`, so `LIMIT` stops the scan early without any operator knowing that + `LIMIT` exists. +- **Hash join** — `INNER JOIN` runs build/probe over a hash table rather than a + nested loop, and handles duplicate and NULL keys correctly. +- **Three-valued NULL logic** — comparisons involving NULL are unknown, and + `WHERE` keeps only rows that are exactly true. +- **Aggregates** — `COUNT`, `SUM`, `AVG`, `MIN`, `MAX` with `GROUP BY` and + `HAVING`. +- **Plan-time type checking** — unknown columns, ambiguous references, and type + mismatches are rejected before a single row is read. +- **Differential testing against SQLite** — a seeded generator produces queries + and both engines must agree. + +## Quick start + +```bash +go run ./cmd/minisql -data examples +``` + +The bundled `examples/` directory holds this schema: + +``` +users(id INT, name TEXT, age INT, city TEXT) +orders(id INT, user_id INT, total INT) +``` + +backed by `users.csv` and `orders.csv`: + +```csv +1,alice,30,berlin +2,bob,15,paris +3,carol,40,berlin +4,dan,,london +``` + +A session (output is verbatim): + +``` +minisql — one SQL statement per line (Ctrl-D to exit) +SELECT name, age FROM users WHERE age >= 18 ORDER BY age DESC +name | age +------+---- +carol | 40 +alice | 30 +(2 rows) + +SELECT city, COUNT(id), AVG(age) FROM users GROUP BY city HAVING COUNT(id) > 1 +city | COUNT(id) | AVG(age) +-------+-----------+--------- +berlin | 2 | 35 +(1 row) + +SELECT users.name, orders.total FROM users JOIN orders ON users.id = orders.user_id ORDER BY orders.total DESC +name | total +------+------ +alice | 300 +carol | 250 +alice | 100 +(3 rows) + +SELECT name, age FROM users WHERE age IS NULL +name | age +-----+----- +dan | NULL +(1 row) + +SELECT bogus FROM users +error: unknown column "bogus" +``` + +Note what the join left out: order 13 has an empty `user_id`, and a NULL key +never matches, so it appears in no result row. + +## Supported SQL + +``` +SELECT <* | expr [, expr ...]> +FROM +[[INNER] JOIN
ON = ] +[WHERE ] +[GROUP BY [HAVING ]] +[ORDER BY [ASC | DESC] [, ...]] +[LIMIT ] +``` + +Expressions support column references (`name`, `users.name`), literals, +`= <> < <= > >=`, `+ - * /`, `AND OR NOT`, `IS [NOT] NULL`, and the aggregate +functions `COUNT`, `SUM`, `AVG`, `MIN`, `MAX` (including `COUNT(*)`). + +A projection that is not a bare column is labelled with its source expression, +so `SELECT COUNT(id)` reports a column named `COUNT(id)`. + +## Data format + +A schema file declares one table per line as `name(col TYPE, ...)` with types +`INT`, `FLOAT`, `TEXT`, or `BOOL`. Each table reads `.csv` from the same +directory. CSV files have **no header row**, and an **empty field is NULL** — +which is how the examples above get a NULL age and a NULL join key. + +## Architecture + +``` +SQL text + │ lexer tokens + ▼ + parser ─────────────▶ AST + │ + ▼ planner (resolve names, check types against the catalog) +operator tree + │ + ▼ executor (volcano: each Next() pulls from its child) + rows ─▶ REPL prints a table +``` + +The planner resolves every column reference to an index and type before +execution, so operators work with positions rather than names and cannot fail on +an unknown column mid-scan. + +The reasoning behind the execution model, the hash join, the NULL semantics, and +oracle-based testing is written up in +[docs/design-notes.md](docs/design-notes.md). + +``` +cmd/minisql/ entrypoint +internal/lexer/ tokenizer +internal/parser/ recursive-descent parser +internal/ast/ syntax tree and its rendering +internal/catalog/ table schemas +internal/csv/ schema-aware CSV reader +internal/plan/ AST → operator tree, name and type resolution +internal/exec/ operators (scan, filter, project, hash join, sort, limit, aggregate) +internal/value/ typed values and three-valued logic +internal/repl/ read-eval-print loop +internal/difftest/ test-only: SQLite oracle, query generator, differential runner +``` + +## Testing + +```bash +go test -race ./... +``` + +Three layers: + +- **Golden tests** per package — a query and fixture in, exact rows out. +- **A NULL semantics suite** (`internal/plan/null_test.go`) stating three-valued + behaviour explicitly: unknown comparisons excluded by `WHERE`, `NOT unknown` + still excluded, NULL join keys dropped, aggregates skipping NULLs, and + `GROUP BY` collapsing NULL keys into one group. +- **Differential tests against SQLite** — a seeded generator emits queries from + a bounded grammar, both engines run them over identical data, and the results + must match as multisets. + +Run a wider sweep or replay a specific run: + +```bash +go test ./internal/difftest/ -run Differential -seed 31337 -queries 5000 +``` + +Every failure prints the exact query and seed. The engine has been checked +across seeds 1, 2, 3, 99, 777, 12345, and 31337 at 5,000 queries each — 35,000 +comparisons with no disagreements. Roughly a quarter of the generated queries +are joins, so the hash join is genuinely covered rather than incidentally +touched. + +Comparison ignores row order, because SQL leaves it unspecified without +`ORDER BY`, but duplicate counts still have to match. `ORDER BY` correctness is +covered by the deterministic golden tests instead. + +`modernc.org/sqlite` is pure Go and is imported only by `internal/difftest`, +which nothing outside its own tests imports, so the engine binary carries no +SQLite driver: + +```bash +go list -deps ./cmd/minisql | grep -c sqlite # 0 +``` + +## Known divergences from SQLite + +These are deliberate design choices, not bugs, and the query generator avoids +them so they cannot mask a real disagreement: + +| Area | SQLite | This engine | +|---|---|---| +| `/` on two integers | integer division (`5/2` = 2) | float division (2.5) | +| comparing a text column to a number | coerced dynamically | rejected at plan time | +| booleans | stored as integers 0/1 | a distinct `BOOL` type | + +## Non-goals + +No storage engine or persistence, no transactions, no indexes, no query +optimizer, no subqueries, no `OUTER`/`CROSS` joins, no `DISTINCT`, no window +functions, and no DDL or DML. The engine reads CSV files and answers `SELECT` +queries. + +## Requirements + +Go 1.25 or newer. The engine itself uses only the standard library; the SQLite +driver used by the differential tests is what sets the Go version floor. + +## License + +MIT diff --git a/docs/design-notes.md b/docs/design-notes.md new file mode 100644 index 0000000..087653e --- /dev/null +++ b/docs/design-notes.md @@ -0,0 +1,139 @@ +# Design notes: how a query becomes rows + +Most of what makes a query engine interesting is invisible from the outside. The +SQL surface here is small on purpose — the point was never to support a lot of +syntax, it was to build the machinery that turns text into rows and to be able +to defend every part of it. These are the decisions worth writing down, and the +ones I would want to be asked about. + +## Operators pull, they do not push + +Execution is a tree of operators, and the obvious way to run one is bottom-up: +scan the table into a slice, filter the slice, sort it, take the first five. +That works and it is easy to reason about. It also reads a million rows to +answer `LIMIT 5`. + +Instead every operator implements one method: + +```go +type Operator interface { + Schema() Schema + Next() (value.Row, bool) +} +``` + +`Next()` returns one row and pulls whatever it needs from its child. `Limit` +counts to five and then returns `false`; `Filter` above the scan stops being +asked for rows; the scan stops reading. No operator knows that `LIMIT` exists, +and nobody had to write the optimization — it falls out of the shape. This is +the volcano model, and the property that makes it worth the indirection is +exactly this: **the consumer controls how much work the producer does.** + +The cost is that some operators cannot be lazy. `Sort` has to see every row +before it can emit the first one, so it materializes. So does the build side of +a join. Being explicit about which operators block and which stream is most of +what it means to understand an execution plan. + +## Hash join, and why the obvious join is the wrong one + +A nested-loop join is four lines: for each row on the left, walk the right side +looking for matches. It is also O(n·m), and the constant is a full re-scan of +one input per row of the other. + +The hash join splits the work into two phases. Drain the right input once, +hashing each row by its join key into a table. Then stream the left input, +hashing each row's key and probing. One pass over each input, O(n+m). The price +is memory: the entire build side is resident while probing. That is the real +trade, and it is why the build side should be the smaller input — a choice a +query optimizer would make and this engine does not, since it always builds from +the right. + +Two details that are easy to get wrong and that the tests pin down: + +**Duplicate keys.** A key maps to a *slice* of rows, not one row. If two orders +belong to the same user, probing that user must emit both. Getting this wrong +produces a join that silently drops rows, which is the kind of bug that survives +casual testing because the result still looks like a plausible table. + +**NULL keys never match.** A row whose join key is NULL is dropped during build +and skipped during probe. This is not an optimization, it is the semantics: +`NULL = NULL` is unknown, not true, so a NULL key matches nothing — including +another NULL. The hash table would happily group them together if you let it, +which is exactly why it is worth being deliberate. + +## Three-valued logic, and the one place SQL breaks its own rule + +NULL is not a value, it is the absence of one, so any comparison with it answers +"unknown" rather than true or false. That gives three truth values, and the +connectives follow from it: `false AND unknown` is `false` (nothing can rescue +it), while `true AND unknown` is `unknown`. `NOT unknown` is still `unknown` — +negating ignorance does not produce knowledge. + +The consequence that surprises people is what `WHERE` does with it. A predicate +that evaluates to unknown does not pass: + +```go +if !v.IsNull() && v.Type == value.TBool && v.B { + return row, true +} +``` + +So `WHERE age > 10` and `WHERE NOT (age > 10)` both exclude a row with a NULL +age. Between them they do not cover the table, which looks like a bug until you +remember that neither statement is true when the age is unknown. + +Then `GROUP BY` does the opposite. Grouping puts every NULL city in a single +group, even though `NULL = NULL` is not true. SQL treats NULL as a value here +and as an unknown everywhere else, and the reason is pragmatic rather than +principled: a `GROUP BY` that scattered each NULL into its own group would be +useless. The engine implements both behaviours because both are correct — they +are just correct about different things. `internal/plan/null_test.go` states all +of it as executable specification, because this is the part of SQL where "I +think it works" is not good enough. + +## Resolving names before reading rows + +The planner walks the AST and turns every column reference into an index into +the row, checking types as it goes. An unknown column, an ambiguous bare name +across two joined tables, or a comparison between text and a number all fail +here — before the first row is read. + +This buys two things. Errors are reported against the query rather than +appearing halfway through a scan, which is the difference between "unknown +column `bogus`" and a partial result followed by a failure. And operators become +simpler: they index into a row instead of looking up names, so the hot path has +no map lookups and no error handling for something the planner already proved +cannot happen. + +## Testing against something that already knows the answer + +Golden tests only find the bugs you thought of. You write the query, you work +out the expected rows, and you assert them — so the test encodes the same +understanding that produced the code, including its mistakes. + +An oracle breaks that loop. The same data is loaded into SQLite, a generator +emits queries from a bounded grammar, both engines run them, and the results +have to match. SQLite has been correct for two decades; when it disagrees with +this engine, this engine is almost certainly wrong. Around 35,000 generated +queries have gone through this without a disagreement. + +Two things make it honest rather than decorative. + +The generator has to reach the hard parts. My first version only queried a +single table, which meant the hash join — the most intricate operator here — +never appeared in a single comparison. "Verified against SQLite" would have been +technically true and substantially misleading. The runner now asserts that joins +are a real share of what gets checked, so that particular self-deception cannot +come back quietly. + +And when a disagreement does turn up, there are exactly two honest responses: +the engine is wrong and gets fixed, or the divergence is deliberate and gets +written down in the README. Widening the numeric tolerance until the failure +disappears is a third option that is always available and always wrong. The +comparison does round to six significant digits, which absorbs the last-bit +noise between two different ways of computing an average — but the cells carry +type tags, so the text `35` never compares equal to the number 35, and NULL uses +a sentinel no generated literal can produce. Where the two engines genuinely +disagree by design — integer division, cross-type comparison, booleans — the +generator does not emit the construct at all, so a real bug cannot hide behind a +known one.