One translation into Arrow, shared with the engine

[tamnd]

This client had seven hundred lines that turned a result into Arrow columns, and the JavaScript client had another seven hundred, and the two had to agree about what a year-month duration is, what a node names itself and which times Arrow has no type for. They now both export through zu-arrow in the engine tree, which is the one answer to that question, so there is nowhere left for them to drift apart. The engine pin moves from 130f67d to 0698a4e to pick it up, which is the same revision the Node client is on.

What stays here is the three things a shared crate cannot know. Which Python exception each refusal is, so a value of the wrong type is still a TypeError and one that will not fit is still a ValueError. Where this client keeps the table names, which is Names, now offered to the translation through its Tables trait and borrowed rather than cloned, because a column of a hundred million nodes is a hundred million lookups. And how many rows a caller wanted in a batch.

That last one is new. record_batches takes rows_per_batch, the way DuckDB's fetch_record_batch does, for a consumer that has a size in mind because its own downstream has one. It costs nothing either way: the arrays are built once and a batch is a slice of them, so the size decides how often a reader is called and nothing else. Zero is refused with the reason. The size cannot ride on __arrow_c_stream__, which takes no argument the protocol did not give it, so what goes to RecordBatchReader.from_stream is a small object holding a stream that was already cut to the size asked for.

What changed for a caller

A result that matched no rows now says what its columns hold instead of typing them all as null. The type comes off the column rather than off the rows, so a query that found nothing can still be written to Parquet or appended to a table that already exists, which is what a consumer doing either of those wanted from it. tests/test_arrow.py says so under the name it always did.

The numbers were stale

The engine fills the column buffers during the scan now rather than transposing the rows afterwards, so the README was claiming a cost that has not been paid for a while. Three hundred thousand rows across three columns take 4.8 ms as Arrow against 86 ms as Python objects, where the README said 44 against 67, and a single integer column takes 0.8 ms against 43 ms, where it said 13.8 against 44.5. A million rows in two columns as numpy take 9.9 ms against 11.9 ms for the Arrow table with to_numpy on each column and 199 ms for the same rows as tuples. Batch size makes no measurable difference at 4096, 65536 or a million, which is what a slice costing nothing looks like from outside.

The conformance runner

Two fixes it needed before it could read the corpus at all, both of which were stopping the whole file from loading rather than failing a case. The corpus writes a backspace and a form feed in a double quoted scalar and this reader's escape table had neither. And it writes FINISH as a case with an empty columns:, which is a query that answers no columns rather than a query whose columns held no rows, and the reader was refusing it as a sequence somebody had left unfinished. With both fixed the corpus runs: 1035 cases, 1030 passed, 0 failed, 5 unsupported, all five being a local time written to the nanosecond that this client holds to the microsecond.

Checks

pytest green, 737 passed and 5 skipped. ruff check and ruff format --check clean. cargo fmt --all --check clean, cargo clippy --all-targets -- -D warnings clean, and both of the other two ABI feature sets check. The conformance runner and tests/test_conformance.py both green against the corpus at the new pin.