Read and bind a byte string

Cargo.toml

@@ -18,14 +18,14 @@ crate-type = ["cdylib"]
 # with (ADR 0002), so a revision is the honest way to say which one.
 # A local checkout is used instead with a `paths` override in
 # `.cargo/config.toml`, which is untracked on purpose.
-zudb = { package = "zu", git = "https://github.com/tamnd/zu", rev = "6ee7d8019415ba16145e9324ccb16b4ffa1d6c9a" }
-zu-common = { git = "https://github.com/tamnd/zu", rev = "6ee7d8019415ba16145e9324ccb16b4ffa1d6c9a" }
+zudb = { package = "zu", git = "https://github.com/tamnd/zu", rev = "230581cdbeb832563d45c48e70436ff03d2d9b54" }
+zu-common = { git = "https://github.com/tamnd/zu", rev = "230581cdbeb832563d45c48e70436ff03d2d9b54" }
 # The one translation from a result into Arrow, which lives in the
 # engine tree so that every client agrees about what a column becomes.
 # `ffi` is the only feature this client turns on: what Python wants is
 # the C Data Interface, which is how a result reaches pyarrow, pandas
 # and polars without a Python object per cell.
-zu-arrow = { git = "https://github.com/tamnd/zu", rev = "6ee7d8019415ba16145e9324ccb16b4ffa1d6c9a", features = ["ffi"] }
+zu-arrow = { git = "https://github.com/tamnd/zu", rev = "230581cdbeb832563d45c48e70436ff03d2d9b54", features = ["ffi"] }
 # `extension-module` is asked for by maturin, in pyproject.toml, and
 # not here. Only the build backend knows how an extension is linked on
 # the platform it is building for, and a crate that turns the feature

README.md

@@ -141,7 +141,7 @@ result.fetchnumpy()  # {"name": array([...]), "score": array([...])}
 
 `Result` implements `__arrow_c_stream__`, so anything that reads the protocol reads a result directly and none of the four methods above is needed: `pyarrow.table(result)` and `polars.DataFrame(result)` both work. Batches are 65,536 rows, or whatever `record_batches(rows_per_batch)` asks for, and the size costs nothing either way: the arrays are built once and a batch is a slice of them rather than a copy. A column holds one type, which the values decide, and integers beside floats are the one mixture that widens rather than being refused. A result that matched no rows still says what its columns hold, so a query that found nothing can still be written to Parquet or appended to a table that already exists. Nodes, rels and paths go across as structs. The translation runs with the GIL released, and on this machine 300,000 rows across three columns take 4.8 ms as Arrow against 86 ms as Python objects, and a single integer column takes 0.8 ms against 43 ms. That is a wider gap than it used to be because the engine now fills the column buffers during the scan rather than transposing the rows afterwards, so what is left here is putting an Arrow type around a buffer that already exists.
 
-`fetchnumpy()` is the same columns as numpy arrays, keyed by column name, for the code that takes arrays rather than frames. It needs numpy and nothing else: an integer, float, datetime or duration column is the engine's own buffer moved into numpy and named, so there is no pass over the values and no second copy of the result in memory. Over a million rows in two columns on this machine it takes 9.9 ms, against 11.9 ms for building the Arrow table and calling `to_numpy` on each of its columns, and 199 ms for the same rows as tuples. Dates are `datetime64[D]`, datetimes `datetime64[ns]`, and durations and times of day `timedelta64[ns]`, the last of those being nanoseconds since midnight, which is what a clock reading is on a number line and the closest thing numpy has to one. A column with a null in it comes back as a `numpy.ma.masked_array`, since numpy has no missing integer, and the mask is built from the validity bitmap the engine already filled rather than by walking the column again. Strings, nodes, rels, paths, lists and records come back as object arrays with `None` in the cell, because an object array has somewhere to put one.
+`fetchnumpy()` is the same columns as numpy arrays, keyed by column name, for the code that takes arrays rather than frames. It needs numpy and nothing else: an integer, float, datetime or duration column is the engine's own buffer moved into numpy and named, so there is no pass over the values and no second copy of the result in memory. Over a million rows in two columns on this machine it takes 9.9 ms, against 11.9 ms for building the Arrow table and calling `to_numpy` on each of its columns, and 199 ms for the same rows as tuples. Dates are `datetime64[D]`, datetimes `datetime64[ns]`, and durations and times of day `timedelta64[ns]`, the last of those being nanoseconds since midnight, which is what a clock reading is on a number line and the closest thing numpy has to one. A column with a null in it comes back as a `numpy.ma.masked_array`, since numpy has no missing integer, and the mask is built from the validity bitmap the engine already filled rather than by walking the column again. Strings, byte strings, nodes, rels, paths, lists and records come back as object arrays with `None` in the cell, because an object array has somewhere to put one. A byte string is an object array rather than an `S` one because `S` pads every cell to the longest and drops trailing nulls, which is a different value from the one stored.
 
 ## Reading a result as it arrives
 

conformance/values.py

@@ -74,6 +74,11 @@
     "FLOAT32": True,
     "FLOAT64": True,
     "STRING": False,
+    # A byte string is written in quotes because its hexits are digits
+    # as often as not: a bare 0041 is a number with a leading zero in
+    # one reader and the string it looks like in another, and neither of
+    # them is the two octets the case meant.
+    "BYTES": True,
     "DATE": True,
     "LOCALTIME": True,
     "ZONEDTIME": True,
@@ -93,8 +98,10 @@
 
 #: The types the encoding reserves a name for and the engine has no
 #: runtime value for yet, kept apart from an outright typo so that the
-#: error says which of the two it is.
-_RESERVED = ("DECIMAL", "BYTES")
+#: error says which of the two it is. BYTES was here and is not any
+#: more: the engine gained the value in tamnd/zu#543 and this reader
+#: gained it with the pin that brought it in.
+_RESERVED = ("DECIMAL",)
 
 #: The range each integer width holds, so that a case writing a value
 #: its own type cannot carry is refused rather than stored wider than it
@@ -400,6 +407,8 @@ def _scalar(ty: str, text: str) -> object:
         return {"true": True, "false": False}.get(text, _NOT_ONE)
     if ty == "STRING":
         return text
+    if ty == "BYTES":
+        return _hexits(text)
     if ty in _RANGES:
         try:
             n = int(text)
@@ -440,6 +449,46 @@ def _scalar(ty: str, text: str) -> object:
     return _NOT_ONE
 
 
+#: What each hexit is worth, both cases of the six letters, because ISO
+#: writes the literal in upper case and a case is free to write either.
+#: A table rather than ``int(c, 16)``, which also takes the digits of
+#: every other script and would read an Arabic-Indic three as a three.
+_HEXITS = {c: n for n, c in enumerate("0123456789abcdef")} | {
+    c: n + 10 for n, c in enumerate("ABCDEF")
+}
+
+#: What the reference reader drops between hexits, which is Rust's
+#: `is_ascii_whitespace`: a vertical tab is not in it.
+_SPACE = " \t\n\r\f"
+
+
+def _hexits(text: str) -> object:
+    """The octets a run of hexits names, or ``_NOT_ONE`` for text that is
+    not a run of them or that names half a byte.
+
+    Space is allowed anywhere and dropped, which is what the standard's
+    production allows and what lets a long literal be written in groups.
+
+    Written out rather than handed to ``bytes.fromhex``, which is the
+    same function on two counts and not on a third: it drops a vertical
+    tab as well, and which whitespace it drops has changed between
+    Python versions this client supports. A second reader of the corpus
+    that accepts a shade more than the first is a reader that lets a
+    malformed case through on one client and not on another, which is
+    the failure this whole module exists to make impossible."""
+    nibbles: list[int] = []
+    for c in text:
+        if c in _SPACE:
+            continue
+        nibble = _HEXITS.get(c)
+        if nibble is None:
+            return _NOT_ONE
+        nibbles.append(nibble)
+    if len(nibbles) % 2:
+        return _NOT_ONE
+    return bytes((high << 4) | low for high, low in zip(nibbles[::2], nibbles[1::2], strict=True))
+
+
 def _parse(text: str, fn) -> object:
     try:
         return fn(text)
@@ -634,6 +683,11 @@ def show(value: object) -> str:
         return f'FLOAT64 "{_show_float(value)}"'
     if isinstance(value, str):
         return f"STRING {quote(value)}"
+    # Upper case because ISO writes the literal that way, and a report
+    # that is diffed against the reference one is comparing text: one
+    # case is one answer.
+    if isinstance(value, bytes):
+        return f'BYTES "{value.hex().upper()}"'
     if isinstance(value, Duration):
         return f'DURATION "{_show_duration(value)}"'
     if isinstance(value, TooFine):

python/zudb/types.py

@@ -20,13 +20,16 @@
 #: because a list holds values and one of them may be a list. A
 #: ``timedelta`` goes in and never comes out: zu stores it as a day-time
 #: duration and hands one back, since a ``timedelta`` cannot hold every
-#: duration zu can.
+#: duration zu can. ``bytes`` and not ``bytearray``, because a parameter
+#: is read after the call that takes it returns and a mutable buffer is a
+#: promise the caller can break.
 Value: TypeAlias = (
     None
     | bool
     | int
     | float
     | str
+    | bytes
     | datetime.date
     | datetime.time
     | datetime.datetime

src/numpy.rs

@@ -31,7 +31,7 @@
 
 use pyo3::exceptions::{PyTypeError, PyValueError};
 use pyo3::prelude::*;
-use pyo3::types::PyDict;
+use pyo3::types::{PyBytes, PyDict};
 use zudb::query::QueryResult;
 use zudb::query::column::{Column, ColumnData, ColumnType, Offsets, Validity};
 
@@ -102,6 +102,25 @@ fn column<'py>(
             }
             return objects(py, out);
         }
+        // The same two buffers a string column keeps, over bytes that
+        // are not text, so the walk is the string walk without the
+        // UTF-8 check. An object array for the same reason: numpy has
+        // no dtype for a run of octets whose length varies by row, and
+        // `S` pads every cell to the longest one and drops trailing
+        // nulls, which is a different value from the one stored.
+        ColumnData::Bytes(octets) => {
+            let mut out = Vec::with_capacity(len);
+            let bytes = &octets.bytes;
+            let mut spans = spans(&octets.offsets);
+            for at in 0..len {
+                let (from, upto) = spans.next().unwrap_or((0, 0));
+                out.push(match missing(&valid, at) {
+                    true => py.None(),
+                    false => PyBytes::new(py, &bytes[from..upto]).into_any().unbind(),
+                });
+            }
+            return objects(py, out);
+        }
         ColumnData::Complex(values) => {
             let mut out = Vec::with_capacity(values.len());
             for value in values {