Skip to content

Read and bind a byte string - #38

Merged
tamnd merged 1 commit into
mainfrom
bytes
Aug 24, 2026
Merged

Read and bind a byte string#38
tamnd merged 1 commit into
mainfrom
bytes

Conversation

@tamnd

@tamnd tamnd commented Aug 24, 2026

Copy link
Copy Markdown
Owner

Closes #37.

The issue said the conformance reader was the only thing missing, and that was true of the reader and not of the client. The pin was at 6ee7d80, which is before the engine had a Value::Bytes at all, so there was nothing for to_py to return and nothing for a parameter to become. The pin moves to 230581c and the three places that touch a value gain the arm they were missing.

Reading gives bytes and binding takes bytes. Not bytearray and not memoryview: a value that came out of a result is a reading of what the file holds and nothing in Python should be able to write through it, and a parameter is read after the call that takes it returns, so a buffer the caller can still write through is a promise this client would be taking on trust. One type in all three directions, which is the one the loader already names for a byte string column.

numpy gets an object array. A byte string column keeps the two buffers a string column keeps, so the walk is the string walk without the UTF-8 check, but the array cannot be an S one: S pads every cell to the longest and drops trailing nulls, which is a different value from the one stored.

The reader turns hexits into octets itself rather than calling bytes.fromhex. The two agree on everything except a vertical tab, which fromhex drops and Rust's is_ascii_whitespace does not, and which whitespace fromhex drops has changed across the Python versions this client supports. A reader of the shared corpus that accepts a shade more than the reference one is a reader that lets a malformed case through on one client and not on another, which is the failure the encoding exists to make impossible.

The issue asked whether the comparison path keeps a byte string apart from the string that spells the same octets, since the case at string.yaml line 995 asserts that X'0041' = 'A' is not true and a reader that decoded octets into a str somewhere would pass it for the wrong reason. It does, and there is now a test saying so on both sides of the comparison.

The pin bump also made ten words reserved that were not before: on, at, number, nothing, record, small, count, big, day and exact. Four test files used four of them as an alias or a column name and are renamed. Nothing in the client changed for that, only what the tests are allowed to call things. Every other client will hit the same list on its next pin bump.

A byte string can be stored and cannot be read back yet, so the loader's refusal at src/load.rs:148 stays where it is. ColType has no byte string in it and the row walk has no arm that produces one, which is tamnd/zu#728.

What was run

Built and tested on a Linux box against CPython 3.12.3.

The corpus is 1399 cases, 1393 passed, 0 failed, 6 unsupported, and all six are a time written to the nanosecond, which is a digit finer than a Python datetime holds. Before this the six BYTES cases in string.yaml were refused at load rather than run.

The six test files this touches are 360 passed and 2 skipped, the two skips being the corpus tests that want ZU_CASES. ruff check, ruff format and cargo fmt are clean.

The issue said the conformance reader was the only thing missing, and
that was true of the reader and not of the client. The pin was at
6ee7d80, which is before the engine had a Value::Bytes at all, so there
was nothing for to_py to return and nothing for a parameter to become.
The pin moves to 230581c and the three places that touch a value gain
the arm they were missing.

Reading gives bytes and binding takes bytes. Not bytearray and not
memoryview: a value that came out of a result is a reading of what the
file holds and nothing in Python should be able to write through it,
and a parameter is read after the call that takes it returns, so a
buffer the caller can still write through is a promise this client
would be taking on trust. One type in all three directions, which is
the one the loader already names for a byte string column.

numpy gets an object array. A byte string column keeps the two buffers
a string column keeps, so the walk is the string walk without the UTF-8
check, but the array cannot be an S one: S pads every cell to the
longest and drops trailing nulls, which is a different value from the
one stored.

The reader turns hexits into octets itself rather than calling
bytes.fromhex. The two agree on everything except a vertical tab, which
fromhex drops and Rust's is_ascii_whitespace does not, and which
whitespace fromhex drops has changed across the Python versions this
client supports. A reader of the shared corpus that accepts a shade
more than the reference one is a reader that lets a malformed case
through on one client and not on another, which is the failure the
encoding exists to make impossible.

The pin bump also made ten words reserved that were not before: on, at,
number, nothing, record, small, count, big, day and exact. Four test
files used four of them as an alias or a column name and are renamed.
Nothing in the client changed for that, only what the tests are allowed
to call things.

A byte string can be stored and cannot be read back yet, so the
loader's refusal at src/load.rs:148 stays where it is. ColType has no
byte string in it and the row walk has no arm that produces one, which
is tamnd/zu#728.

The corpus is 1399 cases, 1393 passed, 0 failed, 6 unsupported, and all
six are a time written to the nanosecond, which is a digit finer than a
Python datetime holds. The six test files this touches are 360 passed
and 2 skipped, the two skips being the corpus tests that want ZU_CASES.
@tamnd
tamnd merged commit a8a8330 into main Aug 24, 2026
29 of 33 checks passed
@tamnd
tamnd deleted the bytes branch August 24, 2026 21:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The runner calls BYTES reserved, so six corpus cases refuse to load

1 participant