docs: document the engine, its SQL surface, and the design decisions

README.md

@@ -1,5 +1,220 @@
 # sql-query-engine
 
-A small SQL query engine in Go: parser, volcano-model executor, and a hash join. Query CSV files from a REPL.
+A small SQL query engine in Go. It lexes and parses a `SELECT` subset, plans it,
+and executes it with a volcano (pull-based) operator model including a real hash
+join. Tables are CSV files described by a small schema; you query them from a
+REPL.
 
-Work in progress.
+**What it is:** a readable, tested implementation of how a query gets from text
+to rows — parsing, planning, and execution — with its results checked against
+SQLite on tens of thousands of generated queries.
+
+**What it is not:** a database. There is no storage engine, no persistence, no
+transactions, no indexes, and no query optimizer. See [Non-goals](#non-goals).
+
+## Features
+
+- **Volcano execution model** — every operator pulls rows from its child through
+  `Next()`, so `LIMIT` stops the scan early without any operator knowing that
+  `LIMIT` exists.
+- **Hash join** — `INNER JOIN` runs build/probe over a hash table rather than a
+  nested loop, and handles duplicate and NULL keys correctly.
+- **Three-valued NULL logic** — comparisons involving NULL are unknown, and
+  `WHERE` keeps only rows that are exactly true.
+- **Aggregates** — `COUNT`, `SUM`, `AVG`, `MIN`, `MAX` with `GROUP BY` and
+  `HAVING`.
+- **Plan-time type checking** — unknown columns, ambiguous references, and type
+  mismatches are rejected before a single row is read.
+- **Differential testing against SQLite** — a seeded generator produces queries
+  and both engines must agree.
+
+## Quick start
+
+```bash
+go run ./cmd/minisql -data examples
+```
+
+The bundled `examples/` directory holds this schema:
+
+```
+users(id INT, name TEXT, age INT, city TEXT)
+orders(id INT, user_id INT, total INT)
+```
+
+backed by `users.csv` and `orders.csv`:
+
+```csv
+1,alice,30,berlin
+2,bob,15,paris
+3,carol,40,berlin
+4,dan,,london
+```
+
+A session (output is verbatim):
+
+```
+minisql — one SQL statement per line (Ctrl-D to exit)
+SELECT name, age FROM users WHERE age >= 18 ORDER BY age DESC
+name  | age
+------+----
+carol | 40
+alice | 30
+(2 rows)
+
+SELECT city, COUNT(id), AVG(age) FROM users GROUP BY city HAVING COUNT(id) > 1
+city   | COUNT(id) | AVG(age)
+-------+-----------+---------
+berlin | 2         | 35
+(1 row)
+
+SELECT users.name, orders.total FROM users JOIN orders ON users.id = orders.user_id ORDER BY orders.total DESC
+name  | total
+------+------
+alice | 300
+carol | 250
+alice | 100
+(3 rows)
+
+SELECT name, age FROM users WHERE age IS NULL
+name | age
+-----+-----
+dan  | NULL
+(1 row)
+
+SELECT bogus FROM users
+error: unknown column "bogus"
+```
+
+Note what the join left out: order 13 has an empty `user_id`, and a NULL key
+never matches, so it appears in no result row.
+
+## Supported SQL
+
+```
+SELECT <* | expr [, expr ...]>
+FROM <table>
+[[INNER] JOIN <table> ON <column> = <column>]
+[WHERE <condition>]
+[GROUP BY <column> [HAVING <condition>]]
+[ORDER BY <expr> [ASC | DESC] [, ...]]
+[LIMIT <n>]
+```
+
+Expressions support column references (`name`, `users.name`), literals,
+`= <> < <= > >=`, `+ - * /`, `AND OR NOT`, `IS [NOT] NULL`, and the aggregate
+functions `COUNT`, `SUM`, `AVG`, `MIN`, `MAX` (including `COUNT(*)`).
+
+A projection that is not a bare column is labelled with its source expression,
+so `SELECT COUNT(id)` reports a column named `COUNT(id)`.
+
+## Data format
+
+A schema file declares one table per line as `name(col TYPE, ...)` with types
+`INT`, `FLOAT`, `TEXT`, or `BOOL`. Each table reads `<name>.csv` from the same
+directory. CSV files have **no header row**, and an **empty field is NULL** —
+which is how the examples above get a NULL age and a NULL join key.
+
+## Architecture
+
+```
+SQL text
+   │  lexer            tokens
+   ▼
+ parser  ─────────────▶ AST
+   │
+   ▼  planner (resolve names, check types against the catalog)
+operator tree
+   │
+   ▼  executor (volcano: each Next() pulls from its child)
+ rows ─▶ REPL prints a table
+```
+
+The planner resolves every column reference to an index and type before
+execution, so operators work with positions rather than names and cannot fail on
+an unknown column mid-scan.
+
+The reasoning behind the execution model, the hash join, the NULL semantics, and
+oracle-based testing is written up in
+[docs/design-notes.md](docs/design-notes.md).
+
+```
+cmd/minisql/        entrypoint
+internal/lexer/     tokenizer
+internal/parser/    recursive-descent parser
+internal/ast/       syntax tree and its rendering
+internal/catalog/   table schemas
+internal/csv/       schema-aware CSV reader
+internal/plan/      AST → operator tree, name and type resolution
+internal/exec/      operators (scan, filter, project, hash join, sort, limit, aggregate)
+internal/value/     typed values and three-valued logic
+internal/repl/      read-eval-print loop
+internal/difftest/  test-only: SQLite oracle, query generator, differential runner
+```
+
+## Testing
+
+```bash
+go test -race ./...
+```
+
+Three layers:
+
+- **Golden tests** per package — a query and fixture in, exact rows out.
+- **A NULL semantics suite** (`internal/plan/null_test.go`) stating three-valued
+  behaviour explicitly: unknown comparisons excluded by `WHERE`, `NOT unknown`
+  still excluded, NULL join keys dropped, aggregates skipping NULLs, and
+  `GROUP BY` collapsing NULL keys into one group.
+- **Differential tests against SQLite** — a seeded generator emits queries from
+  a bounded grammar, both engines run them over identical data, and the results
+  must match as multisets.
+
+Run a wider sweep or replay a specific run:
+
+```bash
+go test ./internal/difftest/ -run Differential -seed 31337 -queries 5000
+```
+
+Every failure prints the exact query and seed. The engine has been checked
+across seeds 1, 2, 3, 99, 777, 12345, and 31337 at 5,000 queries each — 35,000
+comparisons with no disagreements. Roughly a quarter of the generated queries
+are joins, so the hash join is genuinely covered rather than incidentally
+touched.
+
+Comparison ignores row order, because SQL leaves it unspecified without
+`ORDER BY`, but duplicate counts still have to match. `ORDER BY` correctness is
+covered by the deterministic golden tests instead.
+
+`modernc.org/sqlite` is pure Go and is imported only by `internal/difftest`,
+which nothing outside its own tests imports, so the engine binary carries no
+SQLite driver:
+
+```bash
+go list -deps ./cmd/minisql | grep -c sqlite   # 0
+```
+
+## Known divergences from SQLite
+
+These are deliberate design choices, not bugs, and the query generator avoids
+them so they cannot mask a real disagreement:
+
+| Area | SQLite | This engine |
+|---|---|---|
+| `/` on two integers | integer division (`5/2` = 2) | float division (2.5) |
+| comparing a text column to a number | coerced dynamically | rejected at plan time |
+| booleans | stored as integers 0/1 | a distinct `BOOL` type |
+
+## Non-goals
+
+No storage engine or persistence, no transactions, no indexes, no query
+optimizer, no subqueries, no `OUTER`/`CROSS` joins, no `DISTINCT`, no window
+functions, and no DDL or DML. The engine reads CSV files and answers `SELECT`
+queries.
+
+## Requirements
+
+Go 1.25 or newer. The engine itself uses only the standard library; the SQLite
+driver used by the differential tests is what sets the Go version floor.
+
+## License
+
+MIT

docs/design-notes.md

@@ -0,0 +1,139 @@
+# Design notes: how a query becomes rows
+
+Most of what makes a query engine interesting is invisible from the outside. The
+SQL surface here is small on purpose — the point was never to support a lot of
+syntax, it was to build the machinery that turns text into rows and to be able
+to defend every part of it. These are the decisions worth writing down, and the
+ones I would want to be asked about.
+
+## Operators pull, they do not push
+
+Execution is a tree of operators, and the obvious way to run one is bottom-up:
+scan the table into a slice, filter the slice, sort it, take the first five.
+That works and it is easy to reason about. It also reads a million rows to
+answer `LIMIT 5`.
+
+Instead every operator implements one method:
+
+```go
+type Operator interface {
+	Schema() Schema
+	Next() (value.Row, bool)
+}
+```
+
+`Next()` returns one row and pulls whatever it needs from its child. `Limit`
+counts to five and then returns `false`; `Filter` above the scan stops being
+asked for rows; the scan stops reading. No operator knows that `LIMIT` exists,
+and nobody had to write the optimization — it falls out of the shape. This is
+the volcano model, and the property that makes it worth the indirection is
+exactly this: **the consumer controls how much work the producer does.**
+
+The cost is that some operators cannot be lazy. `Sort` has to see every row
+before it can emit the first one, so it materializes. So does the build side of
+a join. Being explicit about which operators block and which stream is most of
+what it means to understand an execution plan.
+
+## Hash join, and why the obvious join is the wrong one
+
+A nested-loop join is four lines: for each row on the left, walk the right side
+looking for matches. It is also O(n·m), and the constant is a full re-scan of
+one input per row of the other.
+
+The hash join splits the work into two phases. Drain the right input once,
+hashing each row by its join key into a table. Then stream the left input,
+hashing each row's key and probing. One pass over each input, O(n+m). The price
+is memory: the entire build side is resident while probing. That is the real
+trade, and it is why the build side should be the smaller input — a choice a
+query optimizer would make and this engine does not, since it always builds from
+the right.
+
+Two details that are easy to get wrong and that the tests pin down:
+
+**Duplicate keys.** A key maps to a *slice* of rows, not one row. If two orders
+belong to the same user, probing that user must emit both. Getting this wrong
+produces a join that silently drops rows, which is the kind of bug that survives
+casual testing because the result still looks like a plausible table.
+
+**NULL keys never match.** A row whose join key is NULL is dropped during build
+and skipped during probe. This is not an optimization, it is the semantics:
+`NULL = NULL` is unknown, not true, so a NULL key matches nothing — including
+another NULL. The hash table would happily group them together if you let it,
+which is exactly why it is worth being deliberate.
+
+## Three-valued logic, and the one place SQL breaks its own rule
+
+NULL is not a value, it is the absence of one, so any comparison with it answers
+"unknown" rather than true or false. That gives three truth values, and the
+connectives follow from it: `false AND unknown` is `false` (nothing can rescue
+it), while `true AND unknown` is `unknown`. `NOT unknown` is still `unknown` —
+negating ignorance does not produce knowledge.
+
+The consequence that surprises people is what `WHERE` does with it. A predicate
+that evaluates to unknown does not pass:
+
+```go
+if !v.IsNull() && v.Type == value.TBool && v.B {
+	return row, true
+}
+```
+
+So `WHERE age > 10` and `WHERE NOT (age > 10)` both exclude a row with a NULL
+age. Between them they do not cover the table, which looks like a bug until you
+remember that neither statement is true when the age is unknown.
+
+Then `GROUP BY` does the opposite. Grouping puts every NULL city in a single
+group, even though `NULL = NULL` is not true. SQL treats NULL as a value here
+and as an unknown everywhere else, and the reason is pragmatic rather than
+principled: a `GROUP BY` that scattered each NULL into its own group would be
+useless. The engine implements both behaviours because both are correct — they
+are just correct about different things. `internal/plan/null_test.go` states all
+of it as executable specification, because this is the part of SQL where "I
+think it works" is not good enough.
+
+## Resolving names before reading rows
+
+The planner walks the AST and turns every column reference into an index into
+the row, checking types as it goes. An unknown column, an ambiguous bare name
+across two joined tables, or a comparison between text and a number all fail
+here — before the first row is read.
+
+This buys two things. Errors are reported against the query rather than
+appearing halfway through a scan, which is the difference between "unknown
+column `bogus`" and a partial result followed by a failure. And operators become
+simpler: they index into a row instead of looking up names, so the hot path has
+no map lookups and no error handling for something the planner already proved
+cannot happen.
+
+## Testing against something that already knows the answer
+
+Golden tests only find the bugs you thought of. You write the query, you work
+out the expected rows, and you assert them — so the test encodes the same
+understanding that produced the code, including its mistakes.
+
+An oracle breaks that loop. The same data is loaded into SQLite, a generator
+emits queries from a bounded grammar, both engines run them, and the results
+have to match. SQLite has been correct for two decades; when it disagrees with
+this engine, this engine is almost certainly wrong. Around 35,000 generated
+queries have gone through this without a disagreement.
+
+Two things make it honest rather than decorative.
+
+The generator has to reach the hard parts. My first version only queried a
+single table, which meant the hash join — the most intricate operator here —
+never appeared in a single comparison. "Verified against SQLite" would have been
+technically true and substantially misleading. The runner now asserts that joins
+are a real share of what gets checked, so that particular self-deception cannot
+come back quietly.
+
+And when a disagreement does turn up, there are exactly two honest responses:
+the engine is wrong and gets fixed, or the divergence is deliberate and gets
+written down in the README. Widening the numeric tolerance until the failure
+disappears is a third option that is always available and always wrong. The
+comparison does round to six significant digits, which absorbs the last-bit
+noise between two different ways of computing an average — but the cells carry
+type tags, so the text `35` never compares equal to the number 35, and NULL uses
+a sentinel no generated literal can produce. Where the two engines genuinely
+disagree by design — integer division, cross-type comparison, booleans — the
+generator does not emit the construct at all, so a real bug cannot hide behind a
+known one.