Skip to content

A connection with no file behind it - #27

Merged
tamnd merged 1 commit into
mainfrom
dx-memory-connection
Aug 19, 2026
Merged

A connection with no file behind it#27
tamnd merged 1 commit into
mainfrom
dx-memory-connection

Conversation

@tamnd

@tamnd tamnd commented Aug 19, 2026

Copy link
Copy Markdown
Owner

zudb.connect(':memory:') made a file called :memory: in whatever directory the caller happened to be standing in. The name says nothing is on disk and something is, which is the worst of both worlds, and it is the last open row of the DuckDB comparison in the engine's docs/clients/duckdb.md.

What changed

connect() with no path at all, and connect(':memory:'), open a database in memory. No file is made anywhere and nothing survives the last connection.

It is the whole engine and not a reduced one. Writes, transactions, the appender, registered frames and streams all work exactly as they do on a file, because underneath it is the same header, the same catalog, the same write-ahead log and the same recovery, running on bytes that are not a file. That is tamnd/zu#405, which put a virtual filesystem under the storage layer rather than bolting a mode to the side of it.

conn.memory is new and says which kind of database is behind the connection. It exists because path cannot quite answer that: a file can be called :memory: on any filesystem that allows a colon. The path is reported back as it was asked for rather than as the engine spells it, since the engine mints a unique name per database so that two of them never share a writer, and that counter is its business and not a caller's.

zudb.aio.connect and zudb.dbapi.connect take the same argument the same way, and the stubs, the docstrings and the README say so.

Tests

Eight new. That a bare connect() writes and reads and leaves the working directory empty, that ':memory:' makes no file called that, that two of them share nothing, that one takes a transaction and rolls it back, that a database on disk is not one in memory, that read-only is refused, and the same through zudb.aio and zudb.dbapi. The README gains a section, and its snippet is run like every other whole program on that page, with the directory checked empty afterwards.

Two things the engine bump moved

The pin goes from a1f310cf to 95c7c990, which is 346 engine commits, and two tests moved with it.

The README's snippet tests indexed programs() by position, so a page that gained a snippet renumbered three tests. They now find a program by what it contains.

test_a_profile_is_what_the_operators_really_did asserted scan.estimate == 3 on a table three INSERT statements old. The estimate comes off the catalog's summary, and rows written through the engine's fold-free write path have not reached it yet, so a freshly written table now estimates low: estimate 1.0 against rows 3. Reopened, the same table estimates 3 with a q-error of 1. Both are now asserted, because the difference between them is a real one a reader would otherwise take for noise. Whether the planner should see rows that have not been folded is the engine's question and not this client's, and it is worth asking there.

Gate

ruff check, ruff format --check, cargo clippy --all-targets -- -D warnings, cargo fmt --check and the full pytest are green locally.

Part of tamnd/zu#169.

`zudb.connect(':memory:')` made a file called `:memory:` in whatever
directory the caller happened to be standing in, which is the worst of
both worlds: the name says nothing is on disk and something is. It is
the last open row of the DuckDB comparison, and it is now the engine's
answer rather than a special case here.

`connect()` with no path, and `connect(':memory:')`, open a database in
memory. No file is made anywhere and nothing survives the last
connection. It is the whole engine and not a reduced one: writes,
transactions, the appender, registered frames and streams all work
exactly as they do on a file, because underneath it is the same header,
the same write-ahead log and the same recovery running on bytes that
are not a file.

`conn.memory` says which kind you have, since `path` cannot quite
answer it on a filesystem that allows a colon in a name. The path is
reported back as it was asked for, `:memory:`, rather than as the
engine spells it: the engine mints a unique name per database so two of
them never share a writer, and that counter is its business.

`zudb.aio.connect` and `zudb.dbapi.connect` take the same argument the
same way.

The engine pin moves to 95c7c990, which is where the in-memory
database landed. Two client tests move with it. The README's snippet
tests now find a program by what it contains rather than by where it
sits, so a page that gains a snippet does not renumber them. And the
profile test no longer pins the optimizer's estimate to the row count:
rows written without folding the file have not reached the catalog's
summary yet, so a table three statements old estimates low and the
same table reopened estimates right. Both are asserted, because the
difference is a real one a reader would otherwise take for noise.
@tamnd
tamnd merged commit c92358e into main Aug 19, 2026
28 of 33 checks passed
@tamnd
tamnd deleted the dx-memory-connection branch August 19, 2026 11:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant