A connection with no file behind it - #27
Merged
Merged
Conversation
`zudb.connect(':memory:')` made a file called `:memory:` in whatever
directory the caller happened to be standing in, which is the worst of
both worlds: the name says nothing is on disk and something is. It is
the last open row of the DuckDB comparison, and it is now the engine's
answer rather than a special case here.
`connect()` with no path, and `connect(':memory:')`, open a database in
memory. No file is made anywhere and nothing survives the last
connection. It is the whole engine and not a reduced one: writes,
transactions, the appender, registered frames and streams all work
exactly as they do on a file, because underneath it is the same header,
the same write-ahead log and the same recovery running on bytes that
are not a file.
`conn.memory` says which kind you have, since `path` cannot quite
answer it on a filesystem that allows a colon in a name. The path is
reported back as it was asked for, `:memory:`, rather than as the
engine spells it: the engine mints a unique name per database so two of
them never share a writer, and that counter is its business.
`zudb.aio.connect` and `zudb.dbapi.connect` take the same argument the
same way.
The engine pin moves to 95c7c990, which is where the in-memory
database landed. Two client tests move with it. The README's snippet
tests now find a program by what it contains rather than by where it
sits, so a page that gains a snippet does not renumber them. And the
profile test no longer pins the optimizer's estimate to the row count:
rows written without folding the file have not reached the catalog's
summary yet, so a table three statements old estimates low and the
same table reopened estimates right. Both are asserted, because the
difference is a real one a reader would otherwise take for noise.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
zudb.connect(':memory:')made a file called:memory:in whatever directory the caller happened to be standing in. The name says nothing is on disk and something is, which is the worst of both worlds, and it is the last open row of the DuckDB comparison in the engine'sdocs/clients/duckdb.md.What changed
connect()with no path at all, andconnect(':memory:'), open a database in memory. No file is made anywhere and nothing survives the last connection.It is the whole engine and not a reduced one. Writes, transactions, the appender, registered frames and streams all work exactly as they do on a file, because underneath it is the same header, the same catalog, the same write-ahead log and the same recovery, running on bytes that are not a file. That is tamnd/zu#405, which put a virtual filesystem under the storage layer rather than bolting a mode to the side of it.
conn.memoryis new and says which kind of database is behind the connection. It exists becausepathcannot quite answer that: a file can be called:memory:on any filesystem that allows a colon. The path is reported back as it was asked for rather than as the engine spells it, since the engine mints a unique name per database so that two of them never share a writer, and that counter is its business and not a caller's.zudb.aio.connectandzudb.dbapi.connecttake the same argument the same way, and the stubs, the docstrings and the README say so.Tests
Eight new. That a bare
connect()writes and reads and leaves the working directory empty, that':memory:'makes no file called that, that two of them share nothing, that one takes a transaction and rolls it back, that a database on disk is not one in memory, that read-only is refused, and the same throughzudb.aioandzudb.dbapi. The README gains a section, and its snippet is run like every other whole program on that page, with the directory checked empty afterwards.Two things the engine bump moved
The pin goes from
a1f310cfto95c7c990, which is 346 engine commits, and two tests moved with it.The README's snippet tests indexed
programs()by position, so a page that gained a snippet renumbered three tests. They now find a program by what it contains.test_a_profile_is_what_the_operators_really_didassertedscan.estimate == 3on a table three INSERT statements old. The estimate comes off the catalog's summary, and rows written through the engine's fold-free write path have not reached it yet, so a freshly written table now estimates low:estimate 1.0againstrows 3. Reopened, the same table estimates 3 with a q-error of 1. Both are now asserted, because the difference between them is a real one a reader would otherwise take for noise. Whether the planner should see rows that have not been folded is the engine's question and not this client's, and it is worth asking there.Gate
ruff check,ruff format --check,cargo clippy --all-targets -- -D warnings,cargo fmt --checkand the fullpytestare green locally.Part of tamnd/zu#169.