test: compare generated queries against SQLite - #55
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Runs generated queries through this engine and through SQLite over identical data and requires the results to agree. The fixture is small but deliberately awkward: NULLs in join keys and group keys, duplicate values, and ties.
The generator now also emits joins. Without them the hash join — the most intricate operator in the engine — would have sat outside the oracle entirely, which would have made "verified against SQLite" mean much less than it sounds. The runner asserts joins are a real share of what it checks (81 of 300 at the default seed), not an occasional accident.
A query that both engines reject is skipped; a query one accepts and the other rejects fails the test rather than being quietly ignored.
Verified beyond the default seed: 5,000 queries on each of seeds 1, 2, 3, 99, 12345, 777, and 31337 — 35,000 comparisons, no disagreements. Replay any run with
-seed.Closes #47