diff --git a/CHANGELOG.md b/CHANGELOG.md index 83f74b256..b6e55a893 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -67,7 +67,7 @@ The Gradle wrapper and build scripts were removed and each project now has a standalone `pom.xml`. (https://github.com/ClickHouse/clickhouse-java/issues/2915) -## 0.10.0-rc1, +## 0.10.0, [Release Migration Guide](docs/releases/0_10_0.md) @@ -115,6 +115,12 @@ previously set in milliseconds but mistakenly retrieved and used in seconds in some places. Now it correctly uses milliseconds consistently. (https://github.com/ClickHouse/clickhouse-java/issues/2358) +- **[client-v2]** The public `ClickHouseBinaryFormatWriter` interface gained two methods, `setString(String, byte[])` + and `setString(int, byte[])`, for writing raw `String`/`FixedString` bytes. Code that only *uses* the interface is + unaffected, but any third party that *implements* `ClickHouseBinaryFormatWriter` directly is source- and + binary-incompatible until it adds these methods (recompiling against the new version is required; otherwise an + `AbstractMethodError` can occur at runtime). + - **[client-v2]** HTTP `503 Service Unavailable` responses are now surfaced as a connection-style failure ( `java.net.ConnectException`) and are retried by default. Previously a `503` was treated as a server error ( `ServerException`) and fell under the `ServerRetryable` fault cause. It has been moved to the `ConnectTimeout` fault @@ -186,12 +192,25 @@ to be returned in their native form regardless of the user-supplied map. Existing maps keyed only by JDBC `SQLType` names continue to work unchanged. (https://github.com/ClickHouse/clickhouse-java/pull/2865) - - **[jdbc-v2]** Added support of custom mapping for JDBC types. Mainly used in cases when big integers should be +- **[jdbc-v2]** Added support of custom mapping for JDBC types. Mainly used in cases when big integers should be presented as string. Use `DriverProperties.JDBC_TYPE_MAPPINGS` (`jdbc_type_mappings`) and set needed type mapping as `key=value[,]` list (For example, `Int32=Long,UInt64=String`). Deprecation notice: V1 property `typeMappings` is supported but will be removed. Please migrate to the new property. (https://github.com/ClickHouse/clickhouse-java/issues/2858) +- **[client-v2, jdbc-v2]** Added opt-in binary string support through the `binary_string_support` configuration property + (or `Client.Builder#binaryStringSupport(boolean)`), disabled by default. The setting is resolved per operation from + the merged client and query settings, so it can be overridden for a single request via the `binary_string_support` + operation option (e.g. `QuerySettings#setOption(ClientConfigProperties.BINARY_STRING_SUPPORT.getKey(), true)`) + independently of the client-level default. When enabled, top-level `String` and `FixedString` columns are read + into a `StringValue` that preserves the raw bytes instead of decoding them into a `String`, allowing non-UTF-8/binary + content to round-trip byte-for-byte. `StringValue` exposes the bytes via `toByteArray()`/`asByteBuffer()` and + lazily decodes a `String` via `asString()` (UTF-8 by default, or a caller-supplied `Charset`). Values nested inside + containers (`Array`, `Map`, `Tuple`, `Nested`, `Variant`) continue to be read as `String`, since those types are not + expected to carry large/binary strings. On the JDBC side, `ResultSet#getBinaryStream(int)` and + `ResultSet#getBinaryStream(String)` are now implemented (previously unsupported) and, together with `getBytes(...)`, + return the raw column bytes. + - **[client-v2]** Added `Client#cancelTransportRequest(String queryId)` to cancel an in-flight request that has not yet received a response from the server, identified by the query id supplied in the operation settings. This aborts the request on the client side (cancels the underlying IO operation) but does **not** issue a `KILL QUERY` on the server, @@ -214,7 +233,10 @@ like `ch_db_01`. This is mostly used in k8s environment. (https://github.com/Cli - **[repo]** Added a contribution guide. Please review and send us your feedback. (https://github.com/ClickHouse/clickhouse-java/pull/2859) -- **[client-v2]** Added endpoint failover support: when multiple endpoints are configured and a request fails with a retryable error (connect timeout, connection refused, HTTP 503, etc.), the client now automatically retries against the next available endpoint instead of always targeting the first one. Failed endpoints are quarantined for 30 seconds before being retried. (https://github.com/ClickHouse/clickhouse-java/issues/2855) +- **[client-v2]** Added endpoint failover support: when multiple endpoints are configured and a request fails with a + retryable error (connect timeout, connection refused, HTTP 503, etc.), the client now automatically retries against + the next available endpoint instead of always targeting the first one. Failed endpoints are quarantined for 30 seconds + before being retried. (https://github.com/ClickHouse/clickhouse-java/issues/2855) ### Bug Fixes @@ -241,6 +263,11 @@ of `NULL` was not set and read. (https://github.com/ClickHouse/clickhouse-java/i - **[jdbc-v2, client-v2]** Fixed writing nullable marker for nested `Tuple` and `Map values. (https://github.com/ClickHouse/clickhouse-java/issues/2721) +- **[jdbc-v2]** Fixed `ResultSet.getObject` leaking the internal `StringValue` holder for `String`/`FixedString` + columns when `binary_string_support` is enabled. `getObject(column, byte[].class)` now returns the exact raw bytes, + and `getObject(column, Object.class)` and the no-type `getObject(column)` overloads now return a decoded `String` + instead of the internal holder. + ## 0.9.8 ### Improvements diff --git a/client-v2/src/main/java/com/clickhouse/client/api/Client.java b/client-v2/src/main/java/com/clickhouse/client/api/Client.java index 796fc225f..c8fd9eee0 100644 --- a/client-v2/src/main/java/com/clickhouse/client/api/Client.java +++ b/client-v2/src/main/java/com/clickhouse/client/api/Client.java @@ -3,6 +3,7 @@ import com.clickhouse.client.api.command.CommandResponse; import com.clickhouse.client.api.command.CommandSettings; import com.clickhouse.client.api.data_formats.ClickHouseBinaryFormatReader; +import com.clickhouse.client.api.data_formats.ClickHouseFormatReader; import com.clickhouse.client.api.data_formats.NativeFormatReader; import com.clickhouse.client.api.data_formats.RowBinaryFormatReader; import com.clickhouse.client.api.data_formats.RowBinaryWithNamesAndTypesFormatReader; @@ -1178,6 +1179,20 @@ public Builder typeHintMapping(Map> typeHintMapping return this; } + /** + * Enables reading {@code String} and {@code FixedString} columns into an intermediate {@code byte[]} + * (a new array each time) instead of decoding them into a {@link String}. This improves working with + * large strings and allows {@link ClickHouseFormatReader#getByteArray} to be used more effectively. Can also be configured + * per operation. + * + * @param enable - if the feature is enabled + * @return this builder instance + */ + public Builder binaryStringSupport(boolean enable) { + this.configuration.put(ClientConfigProperties.BINARY_STRING_SUPPORT.getKey(), String.valueOf(enable)); + return this; + } + /** * SNI SSL parameter that will be set for each outbound SSL socket. diff --git a/client-v2/src/main/java/com/clickhouse/client/api/ClientConfigProperties.java b/client-v2/src/main/java/com/clickhouse/client/api/ClientConfigProperties.java index cf3408905..bbce7a37b 100644 --- a/client-v2/src/main/java/com/clickhouse/client/api/ClientConfigProperties.java +++ b/client-v2/src/main/java/com/clickhouse/client/api/ClientConfigProperties.java @@ -1,5 +1,6 @@ package com.clickhouse.client.api; +import com.clickhouse.client.api.data_formats.ClickHouseFormatReader; import com.clickhouse.client.api.data_formats.internal.AbstractBinaryFormatReader; import com.clickhouse.client.api.enums.SSLMode; import com.clickhouse.client.api.internal.ClickHouseLZ4OutputStream; @@ -185,6 +186,13 @@ public Object parseValue(String value) { */ TYPE_HINT_MAPPING("type_hint_mapping", Map.class), + /** + * When enabled, {@code String} and {@code FixedString} columns are read into an intermediate {@code byte[]} + * instead of decoding them into a {@link String}. Improves working with large strings and lets + * {@link ClickHouseFormatReader#getByteArray} be used more effectively. Can be configured per operation. + */ + BINARY_STRING_SUPPORT("binary_string_support", Boolean.class, "false"), + /** * SNI SSL parameter that will be set for each outbound SSL socket. */ diff --git a/client-v2/src/main/java/com/clickhouse/client/api/data_formats/ClickHouseBinaryFormatWriter.java b/client-v2/src/main/java/com/clickhouse/client/api/data_formats/ClickHouseBinaryFormatWriter.java index 8494c16c3..a9a212f6a 100644 --- a/client-v2/src/main/java/com/clickhouse/client/api/data_formats/ClickHouseBinaryFormatWriter.java +++ b/client-v2/src/main/java/com/clickhouse/client/api/data_formats/ClickHouseBinaryFormatWriter.java @@ -84,6 +84,10 @@ public interface ClickHouseBinaryFormatWriter { void setString(int colIndex, String value); + void setString(String column, byte[] value); + + void setString(int colIndex, byte[] value); + void setDate(String column, LocalDate value); void setDate(int colIndex, LocalDate value); diff --git a/client-v2/src/main/java/com/clickhouse/client/api/data_formats/RowBinaryFormatSerializer.java b/client-v2/src/main/java/com/clickhouse/client/api/data_formats/RowBinaryFormatSerializer.java index ad8ee680a..303a5e7f4 100644 --- a/client-v2/src/main/java/com/clickhouse/client/api/data_formats/RowBinaryFormatSerializer.java +++ b/client-v2/src/main/java/com/clickhouse/client/api/data_formats/RowBinaryFormatSerializer.java @@ -126,10 +126,18 @@ public void writeString(String value) throws IOException { BinaryStreamUtils.writeString(out, value); } + public void writeString(byte[] value) throws IOException { + BinaryStreamUtils.writeString(out, value); + } + public void writeFixedString(String value, int len) throws IOException { BinaryStreamUtils.writeFixedString(out, value, len); } + public void writeFixedString(byte[] value, int len) throws IOException { + SerializerUtils.writeFixedStringBytes(out, value, len); + } + public void writeDate(ZonedDateTime value) throws IOException { SerializerUtils.writeDate(out, value, value.getZone()); } diff --git a/client-v2/src/main/java/com/clickhouse/client/api/data_formats/RowBinaryFormatWriter.java b/client-v2/src/main/java/com/clickhouse/client/api/data_formats/RowBinaryFormatWriter.java index a487da1b9..2a2ecedd3 100644 --- a/client-v2/src/main/java/com/clickhouse/client/api/data_formats/RowBinaryFormatWriter.java +++ b/client-v2/src/main/java/com/clickhouse/client/api/data_formats/RowBinaryFormatWriter.java @@ -203,6 +203,16 @@ public void setString(int colIndex, String value) { setValue(colIndex, value); } + @Override + public void setString(String column, byte[] value) { + setValue(column, value); + } + + @Override + public void setString(int colIndex, byte[] value) { + setValue(colIndex, value); + } + @Override public void setDate(String column, LocalDate value) { setValue(column, value); diff --git a/client-v2/src/main/java/com/clickhouse/client/api/data_formats/RowBinaryWithNamesFormatReader.java b/client-v2/src/main/java/com/clickhouse/client/api/data_formats/RowBinaryWithNamesFormatReader.java index fcfaa7625..e572441ab 100644 --- a/client-v2/src/main/java/com/clickhouse/client/api/data_formats/RowBinaryWithNamesFormatReader.java +++ b/client-v2/src/main/java/com/clickhouse/client/api/data_formats/RowBinaryWithNamesFormatReader.java @@ -23,7 +23,7 @@ public RowBinaryWithNamesFormatReader(InputStream inputStream, TableSchema schema, BinaryStreamReader.ByteBufferAllocator byteBufferAllocator, Map> typeHintMapping) { - super(inputStream, querySettings, schema, byteBufferAllocator, typeHintMapping); + super(inputStream, querySettings, schema, byteBufferAllocator, typeHintMapping); int nCol = 0; try { nCol = BinaryStreamReader.readVarInt(input); diff --git a/client-v2/src/main/java/com/clickhouse/client/api/data_formats/internal/AbstractBinaryFormatReader.java b/client-v2/src/main/java/com/clickhouse/client/api/data_formats/internal/AbstractBinaryFormatReader.java index 8019577d3..6c175e2f4 100644 --- a/client-v2/src/main/java/com/clickhouse/client/api/data_formats/internal/AbstractBinaryFormatReader.java +++ b/client-v2/src/main/java/com/clickhouse/client/api/data_formats/internal/AbstractBinaryFormatReader.java @@ -76,7 +76,7 @@ public abstract class AbstractBinaryFormatReader implements ClickHouseBinaryForm private long row = -1; // before first row private long lastNextCallTs; // for exception to detect slow reader - protected AbstractBinaryFormatReader(InputStream inputStream, QuerySettings querySettings, TableSchema schema,BinaryStreamReader.ByteBufferAllocator byteBufferAllocator, Map> defaultTypeHintMap) { + protected AbstractBinaryFormatReader(InputStream inputStream, QuerySettings querySettings, TableSchema schema, BinaryStreamReader.ByteBufferAllocator byteBufferAllocator, Map> defaultTypeHintMap) { this.input = inputStream; Map settings = querySettings == null ? Collections.emptyMap() : querySettings.getAllSettings(); Boolean useServerTimeZone = (Boolean) settings.get(ClientConfigProperties.USE_SERVER_TIMEZONE.getKey()); @@ -88,8 +88,12 @@ protected AbstractBinaryFormatReader(InputStream inputStream, QuerySettings quer } boolean jsonAsString = MapUtils.getFlag(settings, ClientConfigProperties.serverSetting(ServerSettings.OUTPUT_FORMAT_BINARY_WRITE_JSON_AS_STRING), false); + // Binary string support is resolved from the (already merged client + operation) query settings so it can be + // toggled per operation, not just per client. + boolean binaryStringSupport = MapUtils.getFlag(settings, + ClientConfigProperties.BINARY_STRING_SUPPORT.getKey(), false); this.binaryStreamReader = new BinaryStreamReader(inputStream, timeZone, LOG, byteBufferAllocator, jsonAsString, - defaultTypeHintMap); + defaultTypeHintMap, binaryStringSupport); if (schema != null) { setSchema(schema); } @@ -533,8 +537,9 @@ private T getPrimitiveArray(int index, Class componentType) { } return (T)array; } else if (componentType == byte.class) { - if (value instanceof String) { - return (T) ((String) value).getBytes(StandardCharsets.UTF_8); + byte[] bytes = stringLikeToBytes(value); + if (bytes != null) { + return (T) bytes; } else if (value instanceof InetAddress) { return (T) ((InetAddress) value).getAddress(); } @@ -677,6 +682,24 @@ public Instant getInstant(int index) { throw new ClientException("Column of type " + column.getDataType() + " cannot be converted to Instant"); } + /** + * Converts a string-like value into its raw bytes. For a {@link StringValue} the original bytes are + * returned without re-encoding (so binary content is preserved). For a {@link String} the bytes are + * produced using UTF-8, matching the historical behaviour. Returns {@code null} when the value is not + * a string-like type so callers can fall back to other handling. + * + * @param value value to convert + * @return raw bytes or {@code null} if the value is not string-like + */ + public static byte[] stringLikeToBytes(Object value) { + if (value instanceof StringValue) { + return ((StringValue) value).toByteArray(); + } else if (value instanceof String) { + return ((String) value).getBytes(StandardCharsets.UTF_8); + } + return null; + } + static Instant objectToInstant(Object value) { if (value instanceof LocalDateTime) { LocalDateTime dateTime = (LocalDateTime) value; @@ -867,6 +890,10 @@ public String[] getStringArray(int index) { BinaryStreamReader.ArrayValue array = (BinaryStreamReader.ArrayValue) value; if (array.itemType == String.class) { return (String[]) array.getArray(); + } else if (array.itemType == StringValue.class) { + StringValue[] stringValues = (StringValue[]) array.getArray(); + return Arrays.stream(stringValues) + .map(sv -> sv == null ? null : sv.asString()).toArray(String[]::new); } else if (array.itemType == BinaryStreamReader.EnumValue.class) { BinaryStreamReader.EnumValue[] enumValues = (BinaryStreamReader.EnumValue[]) array.getArray(); return Arrays.stream(enumValues).map(BinaryStreamReader.EnumValue::getName).toArray(String[]::new); diff --git a/client-v2/src/main/java/com/clickhouse/client/api/data_formats/internal/BinaryStreamReader.java b/client-v2/src/main/java/com/clickhouse/client/api/data_formats/internal/BinaryStreamReader.java index 782b6d0d1..95e02dd4f 100644 --- a/client-v2/src/main/java/com/clickhouse/client/api/data_formats/internal/BinaryStreamReader.java +++ b/client-v2/src/main/java/com/clickhouse/client/api/data_formats/internal/BinaryStreamReader.java @@ -57,6 +57,8 @@ public class BinaryStreamReader { private final Class arrayDefaultTypeHint; + private final boolean binaryStringSupport; + private static final int SB_INIT_SIZE = 100; private ClickHouseColumn lastDataColumn = null; @@ -70,8 +72,10 @@ public class BinaryStreamReader { * @param bufferAllocator - byte buffer allocator * @param jsonAsString - use string to serialize/deserialize JSON columns * @param typeHintMapping - what type use as hint if hint is not set or may not be known. + * @param binaryStringSupport - when {@code true}, top-level {@code String}/{@code FixedString} columns are read + * as {@link StringValue} preserving raw bytes; nested string values stay {@link String}. */ - BinaryStreamReader(InputStream input, TimeZone timeZone, Logger log, ByteBufferAllocator bufferAllocator, boolean jsonAsString, Map> typeHintMapping) { + public BinaryStreamReader(InputStream input, TimeZone timeZone, Logger log, ByteBufferAllocator bufferAllocator, boolean jsonAsString, Map> typeHintMapping, boolean binaryStringSupport) { this.log = log == null ? NOPLogger.NOP_LOGGER : log; this.timeZone = timeZone; this.input = input; @@ -80,6 +84,7 @@ public class BinaryStreamReader { this.arrayDefaultTypeHint = typeHintMapping == null || typeHintMapping.isEmpty()? NO_TYPE_HINT : typeHintMapping.get(ClickHouseDataType.Array); + this.binaryStringSupport = binaryStringSupport; } /** @@ -103,8 +108,14 @@ public T readValue(ClickHouseColumn column) throws IOException { * @param - target type of the value * @throws IOException when IO error occurs */ - @SuppressWarnings("unchecked") public T readValue(ClickHouseColumn column, Class typeHint) throws IOException { + // Top-level reads honor the binary-string feature flag. Values nested inside containers always read + // strings as String (see readArray/readMap/readTuple/readNested/readVariant/readJsonData). + return readValue(column, typeHint, binaryStringSupport); + } + + @SuppressWarnings("unchecked") + private T readValue(ClickHouseColumn column, Class typeHint, boolean stringAsBytes) throws IOException { if (column.isNullable()) { int isNull = readByteOrEOF(input); if (isNull == 1) { // is Null? @@ -122,12 +133,18 @@ public T readValue(ClickHouseColumn column, Class typeHint) throws IOExce switch (dataType) { // Primitives case FixedString: { + if (stringAsBytes) { + return (T) new StringValue(readStringBytes(input, precision)); + } byte[] bytes = precision > STRING_BUFF.length ? new byte[precision] : STRING_BUFF; readNBytes(input, bytes, 0, precision); return (T) new String(bytes, 0, precision, StandardCharsets.UTF_8); } case String: { + if (stringAsBytes) { + return (T) readStringValue(); + } return (T) readString(); } case Int8: @@ -244,14 +261,14 @@ public T readValue(ClickHouseColumn column, Class typeHint) throws IOExce case Nothing: return null; case SimpleAggregateFunction: - return (T) readValue(column.getNestedColumns().get(0)); + return (T) readValue(column.getNestedColumns().get(0), typeHint, false); case AggregateFunction: return (T) readBitmap( actualColumn); case Variant: case Geometry: return (T) readVariant(actualColumn); case Dynamic: - return (T) readValue(actualColumn, typeHint); + return (T) readValue(actualColumn, typeHint, stringAsBytes); case Nested: return convertArray(readNested(actualColumn), typeHint); default: @@ -650,10 +667,10 @@ public ArrayValue readArrayItem(ClickHouseColumn itemTypeColumn, int len) throws || itemTypeColumn.getDataType() == ClickHouseDataType.Geometry) { array = new ArrayValue(Object.class, len); for (int i = 0; i < len; i++) { - array.set(i, readValue(itemTypeColumn)); + array.set(i, readNestedValue(itemTypeColumn)); } } else { - Object firstValue = readValue(itemTypeColumn); + Object firstValue = readNestedValue(itemTypeColumn); Class itemClass = firstValue.getClass(); if (firstValue instanceof Byte) { itemClass = byte.class; @@ -680,7 +697,7 @@ public ArrayValue readArrayItem(ClickHouseColumn itemTypeColumn, int len) throws array = new ArrayValue(itemClass, len); array.set(0, firstValue); for (int i = 1; i < len; i++) { - array.set(i, readValue(itemTypeColumn)); + array.set(i, readNestedValue(itemTypeColumn)); } } return array; @@ -691,7 +708,7 @@ public ArrayValue readArrayItem(ClickHouseColumn itemTypeColumn, int len) throws * so that it can be used as the component type of an array. * *

For unsigned integer types, {@code readValue} widens the value (e.g. UInt8 → Short, - * UInt32 → Long), so we use {@link ClickHouseDataType#getWiderObjectClass()} or + * UInt32 → Long), so we use {@link ClickHouseDataType#getWiderObjectClass()} or * {@link ClickHouseDataType#getWiderPrimitiveClass()} which mirrors that widening. * For Enum types, {@code readValue} returns {@link EnumValue} rather than the * declared {@code String.class}. All other types use {@link ClickHouseDataType#getObjectClass()} @@ -744,6 +761,15 @@ private static Class resolveArrayItemClass(ClickHouseColumn itemTypeColumn) { } } + /** + * Reads a value nested inside a container (Array, Map, Tuple, Nested, Variant, JSON). Strings are always + * decoded into {@link String} here, regardless of the binary-string feature flag, because nested types + * are not expected to carry large/binary strings. + */ + private Object readNestedValue(ClickHouseColumn column) throws IOException { + return readValue(column, null, false); + } + public void skipValue(ClickHouseColumn column) throws IOException { readValue(column, null); } @@ -916,8 +942,8 @@ public String toString() { ClickHouseColumn valueType = column.getValueInfo(); LinkedHashMap map = new LinkedHashMap<>(len); for (int i = 0; i < len; i++) { - Object key = readValue(keyType); - Object value = readValue(valueType); + Object key = readNestedValue(keyType); + Object value = readNestedValue(valueType); map.put(key, value); } return map; @@ -933,7 +959,7 @@ public Object[] readTuple(ClickHouseColumn column) throws IOException { int len = column.getNestedColumns().size(); Object[] tuple = new Object[len]; for (int i = 0; i < len; i++) { - tuple[i] = readValue(column.getNestedColumns().get(i)); + tuple[i] = readNestedValue(column.getNestedColumns().get(i)); } return tuple; @@ -957,7 +983,7 @@ public ArrayValue readNested(ClickHouseColumn column) throws IOException { int tupleLen = column.getNestedColumns().size(); Object[] tuple = new Object[tupleLen]; for (int j = 0; j < tupleLen; j++) { - tuple[j] = readValue(column.getNestedColumns().get(j)); + tuple[j] = readNestedValue(column.getNestedColumns().get(j)); } array.set(i, tuple); @@ -971,7 +997,7 @@ public Object readVariant(ClickHouseColumn column) throws IOException { if (ordNum == 0xFF) { return null; } - return readValue(column.getNestedColumns().get(ordNum)); + return readNestedValue(column.getNestedColumns().get(ordNum)); } /** @@ -1238,17 +1264,41 @@ public String readString() throws IOException { } /** - * Reads a decimal value from input stream. + * Reads a string from the internal input stream preserving the raw bytes as a {@link StringValue}. + * Unlike {@link #readString()} this does not decode bytes into a {@link String} and never reuses the + * shared buffer, so the value is safe to keep after the next read. + * + * @return string value holding the raw bytes + * @throws IOException when IO error occurs + */ + public StringValue readStringValue() throws IOException { + return new StringValue(readStringBytes(input, readVarInt(input))); + } + + /** + * Reads the raw bytes of a string from the input stream given its length. + * * @param input - source of bytes - * @return String + * @param len - number of bytes to read + * @return byte[] containing the raw string bytes * @throws IOException when IO error occurs */ - public static String readString(InputStream input) throws IOException { - int len = readVarInt(input); + public static byte[] readStringBytes(InputStream input, int len) throws IOException { if (len == 0) { - return ""; + return new byte[0]; } - return new String(readNBytes(input, len), StandardCharsets.UTF_8); + return readNBytes(input, len); + } + + /** + * Reads a string value from input stream. + * @param input - source of bytes + * @return String + * @throws IOException when IO error occurs + */ + public static String readString(InputStream input) throws IOException { + byte[] bytes = readStringBytes(input, readVarInt(input)); + return bytes.length == 0 ? "" : new String(bytes, StandardCharsets.UTF_8); } public static int readByteOrEOF(InputStream input) throws IOException { @@ -1508,7 +1558,7 @@ private Map readJsonData(InputStream input, ClickHouseColumn col String path = readString(input); ClickHouseColumn dataColumn = predefinedColumns == null? JSON_PLACEHOLDER_COL : predefinedColumns.getOrDefault(path, JSON_PLACEHOLDER_COL); - Object value = readValue(dataColumn); + Object value = readNestedValue(dataColumn); if (value == null && (lastDataColumn != null && lastDataColumn.getDataType() == ClickHouseDataType.Nothing) ) { continue; } diff --git a/client-v2/src/main/java/com/clickhouse/client/api/data_formats/internal/MapBackedRecord.java b/client-v2/src/main/java/com/clickhouse/client/api/data_formats/internal/MapBackedRecord.java index 75f7ea314..0d24063f4 100644 --- a/client-v2/src/main/java/com/clickhouse/client/api/data_formats/internal/MapBackedRecord.java +++ b/client-v2/src/main/java/com/clickhouse/client/api/data_formats/internal/MapBackedRecord.java @@ -3,7 +3,6 @@ import com.clickhouse.client.api.ClientException; import com.clickhouse.client.api.DataTypeUtils; import com.clickhouse.client.api.internal.DataTypeConverter; -import com.clickhouse.client.api.metadata.NoSuchColumnException; import com.clickhouse.client.api.metadata.TableSchema; import com.clickhouse.client.api.query.GenericRecord; import com.clickhouse.client.api.query.NullValueException; @@ -13,7 +12,6 @@ import com.clickhouse.data.value.ClickHouseGeoPointValue; import com.clickhouse.data.value.ClickHouseGeoPolygonValue; import com.clickhouse.data.value.ClickHouseGeoRingValue; -import com.google.common.collect.ImmutableList; import java.math.BigDecimal; import java.math.BigInteger; @@ -276,6 +274,14 @@ private T getPrimitiveArray(String colName) { @Override public byte[] getByteArray(String colName) { + Object value = readValue(colName); + if (value == null) { + return null; + } + byte[] bytes = AbstractBinaryFormatReader.stringLikeToBytes(value); + if (bytes != null) { + return bytes; + } return getPrimitiveArray(colName); } @@ -319,6 +325,10 @@ public String[] getStringArray(String colName) { BinaryStreamReader.ArrayValue array = (BinaryStreamReader.ArrayValue) value; if (array.itemType == String.class) { return (String[]) array.getArray(); + } else if (array.itemType == StringValue.class) { + StringValue[] stringValues = (StringValue[]) array.getArray(); + return Arrays.stream(stringValues) + .map(sv -> sv == null ? null : sv.asString()).toArray(String[]::new); } else if (array.itemType == BinaryStreamReader.EnumValue.class) { BinaryStreamReader.EnumValue[] enumValues = (BinaryStreamReader.EnumValue[]) array.getArray(); return Arrays.stream(enumValues).map(BinaryStreamReader.EnumValue::getName).toArray(String[]::new); @@ -465,7 +475,7 @@ public List getList(int index) { @Override public byte[] getByteArray(int index) { - return getPrimitiveArray(schema.columnIndexToName(index)); + return getByteArray(schema.columnIndexToName(index)); } @Override diff --git a/client-v2/src/main/java/com/clickhouse/client/api/data_formats/internal/SerializerUtils.java b/client-v2/src/main/java/com/clickhouse/client/api/data_formats/internal/SerializerUtils.java index e00438f12..db476bc08 100644 --- a/client-v2/src/main/java/com/clickhouse/client/api/data_formats/internal/SerializerUtils.java +++ b/client-v2/src/main/java/com/clickhouse/client/api/data_formats/internal/SerializerUtils.java @@ -26,6 +26,7 @@ import java.math.BigInteger; import java.net.Inet4Address; import java.net.Inet6Address; +import java.nio.charset.StandardCharsets; import java.sql.Timestamp; import java.time.Duration; import java.time.Instant; @@ -574,10 +575,22 @@ private static void serializePrimitiveData(OutputStream stream, Object value, Cl BinaryStreamUtils.writeBoolean(stream, (Boolean) value); break; case String: - BinaryStreamUtils.writeString(stream, convertToString(value)); + if (value instanceof byte[]) { + BinaryStreamUtils.writeString(stream, (byte[]) value); + } else if (value instanceof StringValue) { + BinaryStreamUtils.writeString(stream, ((StringValue) value).toByteArray()); + } else { + BinaryStreamUtils.writeString(stream, convertToString(value)); + } break; case FixedString: - BinaryStreamUtils.writeFixedString(stream, convertToString(value), column.getPrecision()); + if (value instanceof byte[]) { + writeFixedStringBytes(stream, (byte[]) value, column.getPrecision()); + } else if (value instanceof StringValue) { + writeFixedStringBytes(stream, ((StringValue) value).toByteArray(), column.getPrecision()); + } else { + BinaryStreamUtils.writeFixedString(stream, convertToString(value), column.getPrecision()); + } break; case Date: writeDate(stream, value, ZoneId.of("UTC")); // TODO: check @@ -938,6 +951,63 @@ public static String convertToString(Object value) { return java.lang.String.valueOf(value); } + /** + * Used from the bytecode generated by {@link #compilePOJOSetter(Method, ClickHouseColumn)}. + * Does all needed conversions so have to accept Object instead of specific value type. + * + * @param value value returned by {@code BinaryStreamReader.readValue} + * @return the decoded string, or {@code null} + */ + public static String stringValueToString(Object value) { + if (value == null) { + return null; + } + if (value instanceof StringValue) { + return ((StringValue) value).asString(); + } + return (String) value; + } + + /** + * Used from the bytecode generated by {@link #compilePOJOSetter(Method, ClickHouseColumn)}. + * Does all needed conversions so have to accept Object instead of specific value type. + * + * @param value value returned by {@code BinaryStreamReader.readValue} + * @return the raw bytes, or {@code null} + */ + public static byte[] stringValueToByteArray(Object value) { + if (value == null) { + return null; + } + if (value instanceof StringValue) { + return ((StringValue) value).toByteArray(); + } + if (value instanceof String) { + return ((String) value).getBytes(StandardCharsets.UTF_8); + } + return (byte[]) value; + } + + /** + * Writes raw bytes as a ClickHouse {@code FixedString(length)} value. The bytes are written as-is and + * right-padded with zero bytes when shorter than {@code length}. + * + * @param stream output stream + * @param value raw bytes + * @param length fixed string length + * @throws IOException when failed to write to the stream + */ + public static void writeFixedStringBytes(OutputStream stream, byte[] value, int length) throws IOException { + if (value.length > length) { + throw new IllegalArgumentException("Value of length " + value.length + + " is longer than FixedString(" + length + ")"); + } + stream.write(value); + for (int i = value.length; i < length; i++) { + stream.write(0); + } + } + public static > Set parseEnumList(String value, Class enumType) { Set values = new HashSet<>(); for (StringTokenizer causes = new StringTokenizer(value, Client.VALUES_LIST_DELIMITER); causes.hasMoreTokens(); ) { @@ -978,6 +1048,32 @@ public static List convertArrayValueToList(Object value) { } } + /** + * + * @param setterMethod + * @param column + * @return + * @see ClickHouseDataType#toPrimitiveType(Class) + * @see BinaryStreamReader#isReadToPrimitive(ClickHouseDataType) + * @see SerializerUtils#stringValueToString(Object) + * @see SerializerUtils#stringValueToByteArray(Object) + * @see SerializerUtils#binaryReaderMethodForType(MethodVisitor, Class, ClickHouseDataType) + * @see SerializerUtils#intToOpcode(Class) + * @see SerializerUtils#longToOpcode(Class) + * @see SerializerUtils#floatToOpcode(Class) + * @see SerializerUtils#doubleToOpcode(Class) + * @see BinaryStreamReader#readValue(ClickHouseColumn, Class) + * @see BinaryStreamReader#readByte() + * @see BinaryStreamReader#readUnsignedByte() + * @see BinaryStreamReader#readShortLE() + * @see BinaryStreamReader#readUnsignedShortLE() + * @see BinaryStreamReader#readIntLE() + * @see BinaryStreamReader#readUnsignedIntLE() + * @see BinaryStreamReader#readLongLE() + * @see BinaryStreamReader#readFloatLE() + * @see BinaryStreamReader#readDoubleLE() + * @see java.util.Arrays#asList(Object[]) + */ public static POJOFieldDeserializer compilePOJOSetter(Method setterMethod, ClickHouseColumn column) { Class dtoClass = setterMethod.getDeclaringClass(); @@ -1047,7 +1143,21 @@ public static POJOFieldDeserializer compilePOJOSetter(Method setterMethod, Click Type.getType(Class.class)), false); - if (List.class.isAssignableFrom(targetType) && column.getDataType() == ClickHouseDataType.Tuple) { + if (targetType == String.class) { + // call converter function + mv.visitMethodInsn(INVOKESTATIC, + Type.getInternalName(SerializerUtils.class), + "stringValueToString", + Type.getMethodDescriptor(Type.getType(String.class), Type.getType(Object.class)), + false); + } else if (targetType == byte[].class) { + // call converter function + mv.visitMethodInsn(INVOKESTATIC, + Type.getInternalName(SerializerUtils.class), + "stringValueToByteArray", + Type.getMethodDescriptor(Type.getType(byte[].class), Type.getType(Object.class)), + false); + } else if (List.class.isAssignableFrom(targetType) && column.getDataType() == ClickHouseDataType.Tuple) { mv.visitTypeInsn(CHECKCAST, Type.getInternalName(Object[].class)); mv.visitMethodInsn(INVOKESTATIC, Type.getInternalName(Arrays.class), diff --git a/client-v2/src/main/java/com/clickhouse/client/api/data_formats/internal/StringValue.java b/client-v2/src/main/java/com/clickhouse/client/api/data_formats/internal/StringValue.java new file mode 100644 index 000000000..5584c6564 --- /dev/null +++ b/client-v2/src/main/java/com/clickhouse/client/api/data_formats/internal/StringValue.java @@ -0,0 +1,165 @@ +package com.clickhouse.client.api.data_formats.internal; + +import java.nio.ByteBuffer; +import java.nio.charset.Charset; +import java.nio.charset.StandardCharsets; +import java.util.Arrays; +import java.util.Objects; + +/** + * Read-time holder for ClickHouse {@code String} or {@code FixedString} values that preserves raw bytes + * to avoid lossy decoding and unnecessary allocations. + *

+ * This is an internal value holder, not a general-purpose type for user code. It is produced only + * by the read path when the binary-string feature is enabled (for example {@code GenericRecord.getObject} + * or {@code BinaryStreamReader.readValue} without a type hint), so that callers that need exact bytes can + * obtain them via {@link #toByteArray()} and callers that need text can decode via {@link #asString()}. + * Instances cannot be created by user code: the constructors are package-private and only the binary + * reader builds them. It is not a supported field type for POJO binding: declare POJO fields for + * String/FixedString columns as {@link String} or {@code byte[]} instead. Normal application code should + * generally consume {@link String} or {@code byte[]} rather than holding onto a {@code StringValue}. + *

+ * This is a mutable structure and must be used with care. To avoid copying, it does not + * duplicate the bytes it is given: the constructor wraps the supplied array instead of copying it, and + * {@link #toByteArray()} returns a direct reference to the backing array when the value spans the whole + * array. Consequently, mutating the source array, the array returned by {@link #toByteArray()}, or reading + * the same value concurrently while it is being modified will change the observed value. Callers that need + * an independent snapshot must copy the bytes themselves. + *

+ * Backed by a {@link ByteBuffer} for a richer API and future off-heap memory support. The decoded + * {@link String} produced by {@link #asString()} is cached. + */ +public class StringValue { + + /** Charset used by {@link #asString()} and {@link #toString()} when no charset is provided. */ + public static final Charset DEFAULT_CHARSET = StandardCharsets.UTF_8; + + private final ByteBuffer buffer; + + private final Charset defaultCharset; + + private volatile String cached; + + /** + * Creates a value backed by the given bytes. The array is wrapped, not copied, so it must not be + * modified after being passed in. + * + * @param bytes raw value bytes (not null) + */ + StringValue(byte[] bytes) { + this(bytes, DEFAULT_CHARSET); + } + + /** + * Creates a value backed by the given bytes using the provided default charset. The array is wrapped, + * not copied, so it must not be modified after being passed in. + * + * @param bytes raw value bytes (not null) + * @param defaultCharset charset used by {@link #asString()} and {@link #toString()} (not null) + */ + StringValue(byte[] bytes, Charset defaultCharset) { + Objects.requireNonNull(bytes, "bytes cannot be null"); + Objects.requireNonNull(defaultCharset, "charset is required to convert bytes to String"); + + this.buffer = ByteBuffer.wrap(bytes); + this.defaultCharset = defaultCharset; + } + + /** + * Returns a read-only view over the raw bytes of this value. The returned buffer is independent + * (its own position/limit) and shares no mutable state with this value. + * + * @return read-only buffer positioned at the first byte of the value + */ + public ByteBuffer asByteBuffer() { + return buffer.asReadOnlyBuffer(); + } + + /** + * Returns the raw bytes of this value, honoring the backing buffer's offset and position. + *

+ * As a zero-copy shortcut, when the value spans the entire backing array the live backing storage is + * returned directly (mutating it mutates this value); otherwise an exact-size copy of the value's bytes + * is returned. Callers that need a guaranteed independent snapshot should copy the result themselves. + * + * @return the value bytes (the live backing array when it spans the whole value, otherwise a copy) + */ + public byte[] toByteArray() { + byte[] array = buffer.array(); + int offset = buffer.arrayOffset() + buffer.position(); + int length = buffer.remaining(); + if (offset == 0 && length == array.length) { + return array; + } + return Arrays.copyOfRange(array, offset, offset + length); + } + + /** + * @return number of bytes in this value + */ + public int size() { + return buffer.remaining(); + } + + /** + * @return {@code true} if the value has no bytes + */ + public boolean isEmpty() { + return buffer.remaining() == 0; + } + + /** + * Decodes the value using the default charset (UTF-8 unless another was provided at construction). + * The result is cached so repeated calls do not allocate a new string. + * + * @return decoded string + */ + public String asString() { + String s = cached; + if (s == null) { + s = decode(defaultCharset); + cached = s; + } + return s; + } + + /** + * Decodes the value using the given charset. The result is cached only when the charset matches the + * default charset of this value. + * + * @param charset charset to decode with (not null) + * @return decoded string + */ + public String asString(Charset charset) { + Objects.requireNonNull(charset, "charset cannot be null"); + if (charset.equals(defaultCharset)) { + return asString(); + } + return decode(charset); + } + + private String decode(Charset charset) { + return new String(buffer.array(), buffer.arrayOffset() + buffer.position(), buffer.remaining(), charset); + } + + @Override + public String toString() { + return asString(); + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (!(o instanceof StringValue)) { + return false; + } + return buffer.equals(((StringValue) o).buffer); + } + + @Override + public int hashCode() { + return buffer.hashCode(); + } +} diff --git a/client-v2/src/main/java/com/clickhouse/client/api/internal/DataTypeConverter.java b/client-v2/src/main/java/com/clickhouse/client/api/internal/DataTypeConverter.java index d7f4a0f34..acd1372b6 100644 --- a/client-v2/src/main/java/com/clickhouse/client/api/internal/DataTypeConverter.java +++ b/client-v2/src/main/java/com/clickhouse/client/api/internal/DataTypeConverter.java @@ -2,6 +2,7 @@ import com.clickhouse.client.api.ClickHouseException; import com.clickhouse.client.api.DataTypeUtils; +import com.clickhouse.client.api.data_formats.internal.StringValue; import com.clickhouse.client.api.data_formats.internal.BinaryStreamReader; import com.clickhouse.data.ClickHouseColumn; import com.clickhouse.data.ClickHouseDataType; @@ -90,7 +91,9 @@ public String stringToString(Object bytesOrString, ClickHouseColumn column) { if (column.isArray()) { sb.append(QUOTE); } - if (bytesOrString instanceof CharSequence) { + if (bytesOrString instanceof StringValue) { + sb.append(((StringValue) bytesOrString).asString()); + } else if (bytesOrString instanceof CharSequence) { sb.append(((CharSequence) bytesOrString)); } else if (bytesOrString instanceof byte[]) { sb.append(new String((byte[]) bytesOrString)); diff --git a/client-v2/src/test/java/com/clickhouse/client/ClientTests.java b/client-v2/src/test/java/com/clickhouse/client/ClientTests.java index 0b7b622d6..b50d3700b 100644 --- a/client-v2/src/test/java/com/clickhouse/client/ClientTests.java +++ b/client-v2/src/test/java/com/clickhouse/client/ClientTests.java @@ -333,7 +333,7 @@ public void testDefaultSettings() { Assert.assertEquals(config.get(p.getKey()), p.getDefaultValue(), "Default value doesn't match"); } } - Assert.assertEquals(config.size(), 36); // to check everything is set. Increment when new added. + Assert.assertEquals(config.size(), 37); // to check everything is set. Increment when new added. } try (Client client = new Client.Builder() @@ -364,9 +364,10 @@ public void testDefaultSettings() { .setSocketTimeout(20, SECONDS) .setSocketRcvbuf(100000) .setSocketSndbuf(100000) + .binaryStringSupport(true) .build()) { Map config = client.getConfiguration(); - Assert.assertEquals(config.size(), 37); // to check everything is set. Increment when new added. + Assert.assertEquals(config.size(), 38); // to check everything is set. Increment when new added. Assert.assertEquals(config.get(ClientConfigProperties.DATABASE.getKey()), "mydb"); Assert.assertEquals(config.get(ClientConfigProperties.MAX_EXECUTION_TIME.getKey()), "10"); Assert.assertEquals(config.get(ClientConfigProperties.COMPRESSION_LZ4_UNCOMPRESSED_BUF_SIZE.getKey()), "300000"); @@ -391,6 +392,8 @@ public void testDefaultSettings() { Assert.assertEquals(config.get(ClientConfigProperties.SOCKET_RCVBUF_OPT.getKey()), "100000"); Assert.assertEquals(config.get(ClientConfigProperties.SOCKET_SNDBUF_OPT.getKey()), "100000"); Assert.assertEquals(config.get(ClientConfigProperties.SSL_MODE.getKey()), "STRICT"); + Assert.assertEquals(config.get(ClientConfigProperties.BINARY_STRING_SUPPORT.getKey()), "true"); + } } @@ -434,7 +437,7 @@ public void testWithOldDefaults() { Assert.assertEquals(config.get(p.getKey()), p.getDefaultValue(), "Default value doesn't match"); } } - Assert.assertEquals(config.size(), 36); // to check everything is set. Increment when new added. + Assert.assertEquals(config.size(), 37); // to check everything is set. Increment when new added. } } diff --git a/client-v2/src/test/java/com/clickhouse/client/api/data_formats/internal/BaseReaderTests.java b/client-v2/src/test/java/com/clickhouse/client/api/data_formats/internal/BaseReaderTests.java index b3e9f0676..2cfea137d 100644 --- a/client-v2/src/test/java/com/clickhouse/client/api/data_formats/internal/BaseReaderTests.java +++ b/client-v2/src/test/java/com/clickhouse/client/api/data_formats/internal/BaseReaderTests.java @@ -5,11 +5,13 @@ import com.clickhouse.client.ClickHouseProtocol; import com.clickhouse.client.ClickHouseServerForTest; import com.clickhouse.client.api.Client; +import com.clickhouse.client.api.ClientConfigProperties; import com.clickhouse.client.api.command.CommandSettings; import com.clickhouse.client.api.data_formats.ClickHouseBinaryFormatReader; import com.clickhouse.client.api.enums.Protocol; import com.clickhouse.client.api.query.GenericRecord; import com.clickhouse.client.api.query.QueryResponse; +import com.clickhouse.client.api.query.QuerySettings; import com.clickhouse.data.ClickHouseVersion; import com.clickhouse.data.ClickHouseDataType; import org.testng.Assert; @@ -28,6 +30,7 @@ import java.util.Arrays; import java.util.Collections; import java.util.List; +import java.util.Map; @Test(groups = {"integration"}) public class BaseReaderTests extends BaseIntegrationTest { @@ -572,4 +575,197 @@ private Client.Builder newClient() { .setPassword(ClickHouseServerForTest.getPassword()); } + @Test(groups = {"integration"}) + public void testReadingStringValue() throws Exception { + final String table = "test_reading_stringvalue"; + + client.execute("DROP TABLE IF EXISTS " + table).get(); + client.execute("CREATE TABLE " + table + " (id Int32, s String, fs FixedString(5), e FixedString(1)) ENGINE = MergeTree ORDER BY id").get(); + client.execute("INSERT INTO " + table + " VALUES (1, 'hello', 'world', 'a'), (2, 'ClickHouse', 'Rocks', 'b')").get(); + + Client customClient = newClient() + .binaryStringSupport(true) + .build(); + + try { + try (QueryResponse response = customClient.query("SELECT * FROM " + table + " ORDER BY id").get()) { + ClickHouseBinaryFormatReader reader = customClient.newBinaryFormatReader(response); + + // Test reading multiple strings in a row and check that their content differs + Assert.assertNotNull(reader.next()); + Assert.assertEquals(reader.getInteger("id"), 1); + StringValue s1 = (StringValue) reader.readValue("s"); + StringValue fs1 = (StringValue) reader.readValue("fs"); + StringValue e1 = (StringValue) reader.readValue("e"); + + Assert.assertEquals(s1.asString(), "hello"); + Assert.assertEquals(fs1.asString(), "world"); + Assert.assertEquals(e1.asString(), "a"); + + // Test getting read value multiple times + Assert.assertSame(s1, reader.readValue("s"), "Consecutive reads for the same row should return the same instance or equal value"); + Assert.assertEquals(reader.getString("s"), "hello"); + // Test reading byte[] from String columns + Assert.assertEquals(reader.getByteArray("s"), "hello".getBytes()); + Assert.assertEquals(reader.getByteArray("fs"), "world".getBytes()); + Assert.assertEquals(reader.getByteArray("e"), "a".getBytes()); + + Assert.assertNotNull(reader.next()); + Assert.assertEquals(reader.getInteger("id"), 2); + StringValue s2 = (StringValue) reader.readValue("s"); + StringValue fs2 = (StringValue) reader.readValue("fs"); + StringValue e2 = (StringValue) reader.readValue("e"); + + Assert.assertEquals(s2.asString(), "ClickHouse"); + Assert.assertEquals(fs2.asString(), "Rocks"); + Assert.assertEquals(e2.asString(), "b"); + + Assert.assertNotEquals(s1.asString(), s2.asString()); + Assert.assertNotEquals(fs1.asString(), fs2.asString()); + } + + // test queryAll with string value + List records = customClient.queryAll("SELECT * FROM " + table + " ORDER BY id"); + Assert.assertEquals(records.size(), 2); + + Assert.assertEquals(records.get(0).getInteger("id"), 1); + Assert.assertEquals(records.get(0).getString("s"), "hello"); + Assert.assertEquals(records.get(0).getString("fs"), "world"); + Assert.assertEquals(records.get(0).getByteArray("s"), "hello".getBytes()); + Assert.assertEquals(records.get(0).getByteArray("fs"), "world".getBytes()); + Assert.assertEquals(records.get(0).getByteArray("e"), "a".getBytes()); + + Assert.assertEquals(records.get(1).getInteger("id"), 2); + Assert.assertEquals(records.get(1).getString("s"), "ClickHouse"); + Assert.assertEquals(records.get(1).getString("fs"), "Rocks"); + Assert.assertEquals(records.get(1).getByteArray("s"), "ClickHouse".getBytes()); + Assert.assertEquals(records.get(1).getByteArray("fs"), "Rocks".getBytes()); + Assert.assertEquals(records.get(1).getByteArray("e"), "b".getBytes()); + } finally { + customClient.close(); + } + } + + /** + * Regression test for https://github.com/ClickHouse/clickhouse-java/issues/1397: a String value that holds + * arbitrary binary content (here a SHA-512 hash, which is almost never valid UTF-8) must be read back byte + * for byte instead of being mangled by lossy UTF-8 decoding. + */ + @Test(groups = {"integration"}) + public void testReadingBinaryStringFromHash() throws Exception { + final String message = "abc"; + final byte[] expectedHash = java.security.MessageDigest.getInstance("SHA-512") + .digest(message.getBytes(java.nio.charset.StandardCharsets.UTF_8)); + Assert.assertEquals(expectedHash.length, 64); + + Client customClient = newClient() + .binaryStringSupport(true) + .build(); + + final String query = "SELECT SHA512('" + message + "') AS hash"; + try { + try (QueryResponse response = customClient.query(query).get()) { + ClickHouseBinaryFormatReader reader = customClient.newBinaryFormatReader(response); + Assert.assertNotNull(reader.next()); + + StringValue hash = (StringValue) reader.readValue("hash"); + Assert.assertEquals(hash.size(), expectedHash.length); + Assert.assertEquals(hash.toByteArray(), expectedHash, + "Binary hash bytes must be preserved exactly"); + // getByteArray must agree with the raw StringValue bytes + Assert.assertEquals(reader.getByteArray("hash"), expectedHash); + } + + List records = customClient.queryAll(query); + Assert.assertEquals(records.size(), 1); + Assert.assertEquals(records.get(0).getByteArray("hash"), expectedHash, + "Binary hash read via queryAll must match the locally computed digest"); + } finally { + customClient.close(); + } + } + + /** + * String values nested inside a JSON column must be read as plain {@link String} even when + * {@code binary_string_support} is enabled, consistent with all other container types. Only top-level + * String/FixedString columns are promoted to {@link StringValue}. + */ + @Test(groups = {"integration"}) + public void testJsonStringPathsStayStringWithBinaryStringSupport() throws Exception { + if (isCloud()) { + return; // TODO: add support on cloud + } + if (isVersionMatch("(,24.8]")) { + return; + } + + final String table = "test_json_binary_string_support"; + CommandSettings commandSettings = new CommandSettings(); + commandSettings.serverSetting("allow_experimental_json_type", "1"); + client.execute("DROP TABLE IF EXISTS " + table, commandSettings).get(); + client.execute("CREATE TABLE " + table + " (id Int32, json JSON) ENGINE = MergeTree ORDER BY id", commandSettings).get(); + client.execute("INSERT INTO " + table + " VALUES (1, '{\"name\" : \"hello\"}')", commandSettings).get(); + + Client customClient = newClient() + .binaryStringSupport(true) + .build(); + + try (QueryResponse response = customClient.query("SELECT json FROM " + table + " ORDER BY id").get()) { + ClickHouseBinaryFormatReader reader = customClient.newBinaryFormatReader(response); + Assert.assertNotNull(reader.next()); + + Map json = reader.readValue("json"); + Object name = json.get("name"); + Assert.assertTrue(name instanceof String, + "String values nested in JSON must stay plain String, but got " + + (name == null ? "null" : name.getClass().getName())); + Assert.assertEquals(name, "hello"); + } finally { + customClient.close(); + } + } + + /** + * Binary string support is resolved per operation from the merged client and query settings, so two reads of the + * same table issued from the same client must honor each operation's {@code binary_string_support} override + * independently. The client default is left disabled here; one query opts in while the other relies on the + * default. + */ + @Test(groups = {"integration"}) + public void testBinaryStringSupportIsPerOperation() throws Exception { + final String table = "test_binary_string_support_per_operation"; + + client.execute("DROP TABLE IF EXISTS " + table).get(); + client.execute("CREATE TABLE " + table + " (id Int32, s String) ENGINE = Memory").get(); + client.execute("INSERT INTO " + table + " VALUES (1, 'hello')").get(); + + // The shared client keeps binary string support disabled (the default). + final String query = "SELECT s FROM " + table + " ORDER BY id"; + + // First operation: opt-in via per-operation QuerySettings -> reads a StringValue. + QuerySettings enabled = new QuerySettings() + .setOption(ClientConfigProperties.BINARY_STRING_SUPPORT.getKey(), true); + try (QueryResponse response = client.query(query, enabled).get()) { + ClickHouseBinaryFormatReader reader = client.newBinaryFormatReader(response); + Assert.assertNotNull(reader.next()); + Object value = reader.readValue("s"); + Assert.assertTrue(value instanceof StringValue, + "With binary_string_support enabled for the operation, top-level String must be a StringValue, but got " + + (value == null ? "null" : value.getClass().getName())); + Assert.assertEquals(((StringValue) value).asString(), "hello"); + } + + // Second operation on the same table/client without the override -> reads a plain String. + QuerySettings disabled = new QuerySettings() + .setOption(ClientConfigProperties.BINARY_STRING_SUPPORT.getKey(), false); + try (QueryResponse response = client.query(query, disabled).get()) { + ClickHouseBinaryFormatReader reader = client.newBinaryFormatReader(response); + Assert.assertNotNull(reader.next()); + Object value = reader.readValue("s"); + Assert.assertTrue(value instanceof String, + "With binary_string_support disabled for the operation, top-level String must stay a plain String, but got " + + (value == null ? "null" : value.getClass().getName())); + Assert.assertEquals(value, "hello"); + } + } } \ No newline at end of file diff --git a/client-v2/src/test/java/com/clickhouse/client/api/data_formats/internal/BinaryStreamReaderTests.java b/client-v2/src/test/java/com/clickhouse/client/api/data_formats/internal/BinaryStreamReaderTests.java index 4b089a8ea..dca014999 100644 --- a/client-v2/src/test/java/com/clickhouse/client/api/data_formats/internal/BinaryStreamReaderTests.java +++ b/client-v2/src/test/java/com/clickhouse/client/api/data_formats/internal/BinaryStreamReaderTests.java @@ -201,7 +201,8 @@ public void testReadNullVariantReturnsNull() throws Exception { null, new BinaryStreamReader.CachingByteBufferAllocator(), false, - null); + null, + false); Assert.assertNull(reader.readValue(column)); } @@ -221,7 +222,8 @@ public void testNullableArrayValueUsesBoxedComponentType() throws Exception { null, new BinaryStreamReader.CachingByteBufferAllocator(), false, - null); + null, + false); BinaryStreamReader.ArrayValue array = (BinaryStreamReader.ArrayValue) reader.readValue( ClickHouseColumn.of("v", "Array(Nullable(Float64))")); @@ -244,7 +246,8 @@ public void testNullableUnsignedArrayUsesWidenedType() throws Exception { null, new BinaryStreamReader.CachingByteBufferAllocator(), false, - null); + null, + false); BinaryStreamReader.ArrayValue array = (BinaryStreamReader.ArrayValue) reader.readValue( ClickHouseColumn.of("v", "Array(Nullable(UInt8))")); @@ -267,7 +270,8 @@ public void testNullableEnumArrayUsesEnumValueType() throws Exception { null, new BinaryStreamReader.CachingByteBufferAllocator(), false, - null); + null, + false); BinaryStreamReader.ArrayValue array = (BinaryStreamReader.ArrayValue) reader.readValue( ClickHouseColumn.of("v", "Array(Nullable(Enum8('a'=1,'b'=2)))")); @@ -298,7 +302,8 @@ private void assertEmptyArrayComponentType(String columnType, Class expectedC null, new BinaryStreamReader.CachingByteBufferAllocator(), false, - null); + null, + false); BinaryStreamReader.ArrayValue array = (BinaryStreamReader.ArrayValue) reader.readValue( ClickHouseColumn.of("v", columnType)); diff --git a/client-v2/src/test/java/com/clickhouse/client/api/data_formats/internal/SerializerUtilsTest.java b/client-v2/src/test/java/com/clickhouse/client/api/data_formats/internal/SerializerUtilsTest.java index 9574bed72..41601c2d6 100644 --- a/client-v2/src/test/java/com/clickhouse/client/api/data_formats/internal/SerializerUtilsTest.java +++ b/client-v2/src/test/java/com/clickhouse/client/api/data_formats/internal/SerializerUtilsTest.java @@ -12,6 +12,7 @@ import java.math.BigDecimal; import java.math.BigInteger; import java.net.InetAddress; +import java.nio.charset.StandardCharsets; import java.time.LocalDate; import java.util.ArrayList; import java.util.Arrays; @@ -25,7 +26,7 @@ public class SerializerUtilsTest { private BinaryStreamReader newReader(byte[] data) { return new BinaryStreamReader(new ByteArrayInputStream(data), TimeZone.getTimeZone("UTC"), null, - new BinaryStreamReader.DefaultByteBufferAllocator(), false, null); + new BinaryStreamReader.DefaultByteBufferAllocator(), false, null, false); } @Test @@ -290,6 +291,129 @@ private static Map newMap(Object... kv) { return map; } + @Test + public void testReadNestedReadsArrayOfTuples() throws Exception { + ClickHouseColumn nested = ClickHouseColumn.of("n", "Nested(a String, b Int32)"); + List fields = nested.getNestedColumns(); + + ByteArrayOutputStream out = new ByteArrayOutputStream(); + SerializerUtils.writeVarInt(out, 2); + SerializerUtils.serializeData(out, "x", fields.get(0)); + SerializerUtils.serializeData(out, 1, fields.get(1)); + SerializerUtils.serializeData(out, "y", fields.get(0)); + SerializerUtils.serializeData(out, 2, fields.get(1)); + + BinaryStreamReader.ArrayValue array = newReader(out.toByteArray()).readNested(nested); + Assert.assertEquals(array.length(), 2); + Assert.assertEquals((Object[]) array.get(0), new Object[]{"x", 1}); + Assert.assertEquals((Object[]) array.get(1), new Object[]{"y", 2}); + } + + @Test + public void testReadNestedEmpty() throws Exception { + ClickHouseColumn nested = ClickHouseColumn.of("n", "Nested(a String, b Int32)"); + + ByteArrayOutputStream out = new ByteArrayOutputStream(); + SerializerUtils.writeVarInt(out, 0); + + BinaryStreamReader.ArrayValue array = newReader(out.toByteArray()).readNested(nested); + Assert.assertEquals(array.length(), 0); + } + + @Test + public void testReadValueOnNestedColumnReturnsArrayOfTuples() throws Exception { + ClickHouseColumn nested = ClickHouseColumn.of("n", "Nested(a String, b Int32)"); + List fields = nested.getNestedColumns(); + + ByteArrayOutputStream out = new ByteArrayOutputStream(); + SerializerUtils.writeVarInt(out, 1); + SerializerUtils.serializeData(out, "only", fields.get(0)); + SerializerUtils.serializeData(out, 42, fields.get(1)); + + Object value = newReader(out.toByteArray()).readValue(nested); + Assert.assertTrue(value instanceof BinaryStreamReader.ArrayValue, + "Nested column must read back as an ArrayValue"); + BinaryStreamReader.ArrayValue array = (BinaryStreamReader.ArrayValue) value; + Assert.assertEquals(array.length(), 1); + Assert.assertEquals((Object[]) array.get(0), new Object[]{"only", 42}); + } + + @Test + public void testWriteFixedStringBytesPadsShorterValue() throws Exception { + ByteArrayOutputStream out = new ByteArrayOutputStream(); + SerializerUtils.writeFixedStringBytes(out, new byte[]{1, 2}, 5); + Assert.assertEquals(out.toByteArray(), new byte[]{1, 2, 0, 0, 0}); + } + + @Test + public void testWriteFixedStringBytesWritesExactLength() throws Exception { + ByteArrayOutputStream out = new ByteArrayOutputStream(); + SerializerUtils.writeFixedStringBytes(out, new byte[]{1, 2, 3}, 3); + Assert.assertEquals(out.toByteArray(), new byte[]{1, 2, 3}); + } + + @Test + public void testWriteFixedStringBytesEmptyValueIsAllPadding() throws Exception { + ByteArrayOutputStream out = new ByteArrayOutputStream(); + SerializerUtils.writeFixedStringBytes(out, new byte[0], 3); + Assert.assertEquals(out.toByteArray(), new byte[]{0, 0, 0}); + } + + @Test + public void testWriteFixedStringBytesRejectsValueLongerThanLength() { + Assert.assertThrows(IllegalArgumentException.class, + () -> SerializerUtils.writeFixedStringBytes(new ByteArrayOutputStream(), + new byte[]{1, 2, 3, 4}, 3)); + } + + // stringValueToString / stringValueToByteArray are invoked from the bytecode generated for POJO setters. + // They are exercised end-to-end in StringValueTests, but these unit tests pin every input branch directly + // so the behaviour is locked in even if the set of column types that reach them is extended later. + + @Test + public void testStringValueToStringPassesThroughNull() { + Assert.assertNull(SerializerUtils.stringValueToString(null)); + } + + @Test + public void testStringValueToStringDecodesStringValue() { + StringValue value = new StringValue("héllo".getBytes(StandardCharsets.UTF_8)); + Assert.assertEquals(SerializerUtils.stringValueToString(value), "héllo"); + } + + @Test + public void testStringValueToStringReturnsPlainStringAsIs() { + String value = "plain"; + Assert.assertSame(SerializerUtils.stringValueToString(value), value); + } + + @Test + public void testStringValueToByteArrayPassesThroughNull() { + Assert.assertNull(SerializerUtils.stringValueToByteArray(null)); + } + + @Test + public void testStringValueToByteArrayPreservesStringValueBytes() { + // Non-UTF-8 bytes must survive without re-encoding. + byte[] binary = {(byte) 0xDE, (byte) 0xAD, (byte) 0x00, (byte) 0xBE, (byte) 0xEF}; + StringValue value = new StringValue(binary); + Assert.assertEquals(SerializerUtils.stringValueToByteArray(value), binary); + } + + @Test + public void testStringValueToByteArrayEncodesStringAsUtf8() { + Assert.assertEquals(SerializerUtils.stringValueToByteArray("héllo"), + "héllo".getBytes(StandardCharsets.UTF_8)); + } + + @Test + public void testStringValueToByteArrayPassesThroughByteArray() { + // This is the branch that lets future string-backed columns (e.g. Array(UInt8)) reuse the helper: + // a value that is already a byte[] must be returned unchanged, not re-wrapped or copied. + byte[] bytes = {1, 2, 3}; + Assert.assertSame(SerializerUtils.stringValueToByteArray(bytes), bytes); + } + private void assertCustomGeoTypeTag(String typeName) throws Exception { ByteArrayOutputStream out = new ByteArrayOutputStream(); SerializerUtils.writeDynamicTypeTag(out, ClickHouseColumn.of("v", typeName)); diff --git a/client-v2/src/test/java/com/clickhouse/client/api/data_formats/internal/StringValueTests.java b/client-v2/src/test/java/com/clickhouse/client/api/data_formats/internal/StringValueTests.java new file mode 100644 index 000000000..03ebfd0da --- /dev/null +++ b/client-v2/src/test/java/com/clickhouse/client/api/data_formats/internal/StringValueTests.java @@ -0,0 +1,736 @@ +package com.clickhouse.client.api.data_formats.internal; + +import com.clickhouse.client.api.ClientConfigProperties; +import com.clickhouse.client.api.data_formats.RowBinaryWithNamesAndTypesFormatReader; +import com.clickhouse.client.api.metadata.TableSchema; +import com.clickhouse.client.api.query.QuerySettings; +import com.clickhouse.client.api.serde.POJOFieldDeserializer; +import com.clickhouse.data.ClickHouseColumn; +import com.clickhouse.data.format.BinaryStreamUtils; +import org.testng.Assert; +import org.testng.annotations.DataProvider; +import org.testng.annotations.Test; + +import java.io.ByteArrayInputStream; +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.lang.reflect.Method; +import java.nio.ByteBuffer; +import java.nio.charset.Charset; +import java.nio.charset.StandardCharsets; +import java.util.Arrays; +import java.util.Collections; +import java.util.HashMap; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.TimeZone; + +public class StringValueTests { + + private static BinaryStreamReader reader(byte[] input, boolean binaryStringSupport) { + return new BinaryStreamReader(new ByteArrayInputStream(input), TimeZone.getTimeZone("UTC"), null, + new BinaryStreamReader.DefaultByteBufferAllocator(), false, null, binaryStringSupport); + } + + // ---- StringValue API ---- + + @Test + public void testStringValueApiBasics() { + byte[] bytes = "hello world".getBytes(StandardCharsets.UTF_8); + StringValue sv = new StringValue(bytes); + + Assert.assertEquals(sv.size(), bytes.length); + Assert.assertFalse(sv.isEmpty()); + Assert.assertEquals(sv.asString(), "hello world"); + Assert.assertEquals(sv.toString(), "hello world"); + Assert.assertEquals(sv.toByteArray(), bytes); + } + + @Test + public void testToByteArrayReturnsBackingArrayReference() { + byte[] bytes = {1, 2, 3, 4}; + StringValue sv = new StringValue(bytes); + byte[] backing = sv.toByteArray(); + // No copy is made: the returned array is the live backing storage and mutating it mutates the value. + Assert.assertSame(backing, bytes, "toByteArray() must return the backing array without copying"); + backing[0] = 42; + Assert.assertEquals(sv.toByteArray()[0], 42, "Mutating the returned array mutates the value (no copy)"); + } + + @Test + public void testAsByteBufferIsReadOnly() { + StringValue sv = new StringValue(new byte[]{1, 2, 3}); + ByteBuffer buffer = sv.asByteBuffer(); + Assert.assertTrue(buffer.isReadOnly()); + Assert.assertEquals(buffer.remaining(), 3); + } + + @Test + public void testAsStringIsCached() { + StringValue sv = new StringValue("cached".getBytes(StandardCharsets.UTF_8)); + String first = sv.asString(); + String second = sv.asString(); + Assert.assertSame(first, second, "asString() should cache and return the same instance"); + } + + @Test + public void testAsStringWithCharset() { + String original = "Привет, мир"; + StringValue sv = new StringValue(original.getBytes(StandardCharsets.UTF_16)); + Assert.assertEquals(sv.asString(StandardCharsets.UTF_16), original); + } + + @Test + public void testEqualsAndHashCode() { + StringValue a = new StringValue("abc".getBytes(StandardCharsets.UTF_8)); + StringValue b = new StringValue("abc".getBytes(StandardCharsets.UTF_8)); + StringValue c = new StringValue("abd".getBytes(StandardCharsets.UTF_8)); + + // Reflexive + Assert.assertEquals(a, a); + // Equal content -> equal value and equal hash code + Assert.assertEquals(a, b); + Assert.assertEquals(b, a, "equals must be symmetric"); + Assert.assertEquals(a.hashCode(), b.hashCode()); + // Different content -> not equal + Assert.assertNotEquals(a, c); + } + + @Test + public void testEqualsRejectsNullAndOtherTypes() { + StringValue a = new StringValue("abc".getBytes(StandardCharsets.UTF_8)); + Assert.assertFalse(a.equals(null), "A value must never equal null"); + Assert.assertFalse(a.equals("abc"), "A value must not equal a raw String of the same text"); + Assert.assertNotEquals(a, new Object()); + } + + @Test + public void testEqualsIgnoresDefaultCharset() { + // equals/hashCode are defined on the raw bytes, so the default charset must not affect them. + byte[] bytes = "abc".getBytes(StandardCharsets.UTF_8); + StringValue utf8 = new StringValue(bytes, StandardCharsets.UTF_8); + StringValue latin1 = new StringValue("abc".getBytes(StandardCharsets.UTF_8), StandardCharsets.ISO_8859_1); + Assert.assertEquals(utf8, latin1, "Values with identical bytes must be equal regardless of default charset"); + Assert.assertEquals(utf8.hashCode(), latin1.hashCode()); + } + + @Test + public void testEqualsDistinguishesByContentAndLength() { + StringValue ab = new StringValue(new byte[]{1, 2}); + StringValue abc = new StringValue(new byte[]{1, 2, 3}); + StringValue empty = new StringValue(new byte[0]); + + // Same prefix but different length must not be equal. + Assert.assertNotEquals(ab, abc); + Assert.assertNotEquals(abc, ab); + // Empty values are only equal to other empty values. + Assert.assertEquals(empty, new StringValue(new byte[0])); + Assert.assertNotEquals(empty, ab); + } + + @Test + public void testEqualsIsConsistentWithBinaryReads() throws IOException { + // Two independently read StringValues over the same bytes must compare equal. + byte[] binary = new byte[]{(byte) 0x00, (byte) 0xFF, (byte) 0x80, (byte) 0x7F}; + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + BinaryStreamUtils.writeString(baos, binary); + byte[] wire = baos.toByteArray(); + + ClickHouseColumn column = ClickHouseColumn.of("s", "String"); + StringValue first = reader(wire, true).readValue(column); + StringValue second = reader(wire, true).readValue(column); + + Assert.assertEquals(first, second); + Assert.assertEquals(first.hashCode(), second.hashCode()); + Assert.assertEquals(first, new StringValue(binary)); + } + + @Test + public void testEmptyValue() { + StringValue sv = new StringValue(new byte[0]); + Assert.assertTrue(sv.isEmpty()); + Assert.assertEquals(sv.size(), 0); + Assert.assertEquals(sv.asString(), ""); + Assert.assertEquals(sv.toByteArray().length, 0); + } + + // ---- Reading String columns as StringValue ---- + + @DataProvider(name = "charsetStrings") + private Object[][] charsetStrings() { + return new Object[][]{ + {"plain ascii", StandardCharsets.UTF_8}, + {"unicode: Привет 你好 🚀", StandardCharsets.UTF_8}, + {"latin1 café", StandardCharsets.ISO_8859_1}, + {"utf16 текст", StandardCharsets.UTF_16}, + {" leading and trailing ", StandardCharsets.UTF_8}, + {"", StandardCharsets.UTF_8}, + }; + } + + @Test(dataProvider = "charsetStrings") + public void testReadStringAsStringValuePreservesBytes(String value, Charset charset) throws IOException { + byte[] encoded = value.getBytes(charset); + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + BinaryStreamUtils.writeString(baos, encoded); // binary string write (raw bytes) + + ClickHouseColumn column = ClickHouseColumn.of("s", "String"); + Object read = reader(baos.toByteArray(), true).readValue(column); + + Assert.assertTrue(read instanceof StringValue, "Expected StringValue but got " + read.getClass()); + StringValue sv = (StringValue) read; + Assert.assertEquals(sv.toByteArray(), encoded, "Raw bytes must be preserved"); + Assert.assertEquals(sv.asString(charset), value, "Decoding with the source charset must round-trip"); + } + + @Test + public void testReadBinaryNonUtf8IsPreserved() throws IOException { + // Bytes that are not valid UTF-8 (e.g. a binary hash). Decoding as UTF-8 would be lossy. + byte[] binary = new byte[]{(byte) 0xDE, (byte) 0xAD, (byte) 0xBE, (byte) 0xEF, + (byte) 0xFF, (byte) 0x00, (byte) 0x80, (byte) 0xC0, (byte) 0xFE}; + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + BinaryStreamUtils.writeString(baos, binary); + + ClickHouseColumn column = ClickHouseColumn.of("s", "String"); + StringValue sv = reader(baos.toByteArray(), true).readValue(column); + + Assert.assertEquals(sv.toByteArray(), binary, "Binary content must be preserved exactly"); + Assert.assertEquals(AbstractBinaryFormatReader.stringLikeToBytes(sv), binary, + "Shared string->bytes conversion must preserve binary content"); + } + + @Test + public void testFixedStringAsStringValue() throws IOException { + byte[] binary = new byte[]{(byte) 0x01, (byte) 0xFF, (byte) 0x00, (byte) 0x10, (byte) 0x80}; + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + baos.write(binary); // FixedString(5) is written as exactly 5 raw bytes + + ClickHouseColumn column = ClickHouseColumn.of("s", "FixedString(5)"); + Object read = reader(baos.toByteArray(), true).readValue(column); + + Assert.assertTrue(read instanceof StringValue); + Assert.assertEquals(((StringValue) read).toByteArray(), binary); + } + + @Test + public void testReadStringArrayKeepsStringWhenBinarySupportEnabled() throws IOException { + // Even with the binary-string feature enabled, nested Array(String) elements are read as String: + // nested types are not expected to carry large/binary strings. + String[] elements = {"plain", "Привет", "中文", ""}; + + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + BinaryStreamUtils.writeVarInt(baos, elements.length); + for (String element : elements) { + BinaryStreamUtils.writeString(baos, element.getBytes(StandardCharsets.UTF_8)); + } + + ClickHouseColumn column = ClickHouseColumn.of("a", "Array(String)"); + Object read = reader(baos.toByteArray(), true).readValue(column); + + Assert.assertTrue(read instanceof BinaryStreamReader.ArrayValue, + "Expected ArrayValue but got " + read.getClass()); + BinaryStreamReader.ArrayValue array = (BinaryStreamReader.ArrayValue) read; + Assert.assertEquals(array.length(), elements.length); + + Object raw = array.getArray(); + Assert.assertTrue(raw instanceof String[], "Nested array items must be String, got " + raw.getClass()); + String[] values = (String[]) raw; + for (int i = 0; i < elements.length; i++) { + Assert.assertEquals(values[i], elements[i], "Element " + i + " must round-trip as String"); + } + } + + @Test + public void testReadStringMapKeepsStringWhenBinarySupportEnabled() throws IOException { + // Even with the binary-string feature enabled, nested Map(String, String) keys and values are read as String. + String[] keys = {"k1", "ключ"}; + String[] vals = {"v1", "значение"}; + + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + BinaryStreamUtils.writeVarInt(baos, keys.length); + for (int i = 0; i < keys.length; i++) { + BinaryStreamUtils.writeString(baos, keys[i].getBytes(StandardCharsets.UTF_8)); + BinaryStreamUtils.writeString(baos, vals[i].getBytes(StandardCharsets.UTF_8)); + } + + ClickHouseColumn column = ClickHouseColumn.of("m", "Map(String, String)"); + Object read = reader(baos.toByteArray(), true).readValue(column); + + Assert.assertTrue(read instanceof Map, "Expected Map but got " + read.getClass()); + Map map = (Map) read; + Assert.assertEquals(map.size(), keys.length); + + int i = 0; + for (Map.Entry entry : map.entrySet()) { + Assert.assertTrue(entry.getKey() instanceof String, "Nested map key must be a String"); + Assert.assertTrue(entry.getValue() instanceof String, "Nested map value must be a String"); + Assert.assertEquals(entry.getKey(), keys[i], "Key " + i); + Assert.assertEquals(entry.getValue(), vals[i], "Value " + i); + i++; + } + + Assert.assertEquals(map.get(keys[0]), vals[0]); + } + + @Test + public void testDefaultBehaviorReturnsString() throws IOException { + byte[] encoded = "still a string".getBytes(StandardCharsets.UTF_8); + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + BinaryStreamUtils.writeString(baos, encoded); + + ClickHouseColumn column = ClickHouseColumn.of("s", "String"); + Object read = reader(baos.toByteArray(), false).readValue(column); + + Assert.assertTrue(read instanceof String, "Without a type hint Strings must still be returned as String"); + Assert.assertEquals(read, "still a string"); + } + + // ---- POJO binding (queryAll/readToPOJO) over String columns with the feature enabled ---- + + /** + * Minimal POJO with the only two field representations supported for top-level String/FixedString + * columns: {@link String} and {@code byte[]}. {@link StringValue} is a read-time holder and is not a + * supported POJO field type. + */ + public static class StringPojo { + private String asString; + private byte[] asBytes; + + public void setAsString(String asString) { this.asString = asString; } + public void setAsBytes(byte[] asBytes) { this.asBytes = asBytes; } + } + + private static byte[] stringWire(byte[] value) throws IOException { + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + BinaryStreamUtils.writeString(baos, value); + return baos.toByteArray(); + } + + private static POJOFieldDeserializer setterFor(String name, ClickHouseColumn column) throws Exception { + Method setter = StringPojo.class.getMethod(name, + name.equals("setAsString") ? String.class : byte[].class); + return SerializerUtils.compilePOJOSetter(setter, column); + } + + @Test + public void testPojoSetterStringFieldDecodesWhenFeatureEnabled() throws Exception { + // Regression: with the feature enabled the reader produces StringValue, but a String setter must + // still receive a decoded String (the compiled setter casts the readValue result to String). + ClickHouseColumn column = ClickHouseColumn.of("s", "String"); + byte[] wire = stringWire("hello".getBytes(StandardCharsets.UTF_8)); + + StringPojo pojo = new StringPojo(); + setterFor("setAsString", column).setValue(pojo, reader(wire, true), column); + Assert.assertEquals(pojo.asString, "hello"); + } + + @Test + public void testPojoSetterByteArrayFieldReceivesRawBytesWhenFeatureEnabled() throws Exception { + // A byte[] setter must receive the raw bytes (preserving non-UTF-8 content) instead of a StringValue. + ClickHouseColumn column = ClickHouseColumn.of("s", "String"); + byte[] binary = {(byte) 0xDE, (byte) 0xAD, (byte) 0x00, (byte) 0xBE, (byte) 0xEF}; + byte[] wire = stringWire(binary); + + StringPojo pojo = new StringPojo(); + setterFor("setAsBytes", column).setValue(pojo, reader(wire, true), column); + Assert.assertEquals(pojo.asBytes, binary); + } + + @Test + public void testPojoSetterFixedStringByteArrayFieldWhenFeatureEnabled() throws Exception { + // A byte[] setter over a FixedString column must receive the raw bytes, preserving binary content. + ClickHouseColumn column = ClickHouseColumn.of("s", "FixedString(3)"); + byte[] binary = {(byte) 0xAA, (byte) 0xBB, (byte) 0xCC}; + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + baos.write(binary); // FixedString(3) is exactly 3 raw bytes on the wire + + StringPojo pojo = new StringPojo(); + setterFor("setAsBytes", column).setValue(pojo, reader(baos.toByteArray(), true), column); + Assert.assertEquals(pojo.asBytes, binary); + } + + @Test + public void testPojoSetterFixedStringStringFieldWhenFeatureEnabled() throws Exception { + // A String setter over a FixedString column must receive the decoded String. + ClickHouseColumn column = ClickHouseColumn.of("s", "FixedString(3)"); + byte[] bytes = "abc".getBytes(StandardCharsets.UTF_8); + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + baos.write(bytes); + + StringPojo pojo = new StringPojo(); + setterFor("setAsString", column).setValue(pojo, reader(baos.toByteArray(), true), column); + Assert.assertEquals(pojo.asString, "abc"); + } + + @Test + public void testPojoSetterStringFieldOverSimpleAggregateFunctionWhenFeatureEnabled() throws Exception { + // Regression: a String column wrapped in SimpleAggregateFunction(String) also reads as a StringValue + // when the feature is on. The compiled setter must convert it to String based on the target field type + // (the column data type is SimpleAggregateFunction, not String), otherwise it would ClassCastException. + ClickHouseColumn column = ClickHouseColumn.of("s", "SimpleAggregateFunction(anyLast, String)"); + // On the wire SimpleAggregateFunction(String) is serialized exactly like its nested String. + byte[] wire = stringWire("hello".getBytes(StandardCharsets.UTF_8)); + + StringPojo pojo = new StringPojo(); + setterFor("setAsString", column).setValue(pojo, reader(wire, true), column); + Assert.assertEquals(pojo.asString, "hello"); + } + + @Test + public void testPojoSetterSimpleAggregateFunctionWhenFeatureDisabled() throws Exception { + // With the feature off the reader returns a plain String for SimpleAggregateFunction(String); the + // String field receives it directly and the byte[] field gets the UTF-8 encoding. + ClickHouseColumn column = ClickHouseColumn.of("s", "SimpleAggregateFunction(anyLast, String)"); + byte[] wire = stringWire("world".getBytes(StandardCharsets.UTF_8)); + + StringPojo asString = new StringPojo(); + setterFor("setAsString", column).setValue(asString, reader(wire, false), column); + Assert.assertEquals(asString.asString, "world"); + + StringPojo asBytes = new StringPojo(); + setterFor("setAsBytes", column).setValue(asBytes, reader(wire, false), column); + Assert.assertEquals(asBytes.asBytes, "world".getBytes(StandardCharsets.UTF_8)); + } + + // ---- Writing binary String values ---- + + @Test + public void testWriteByteArrayToStringRoundTrip() throws IOException { + byte[] binary = new byte[]{(byte) 0x00, (byte) 0xFF, (byte) 0xAB, (byte) 0xCD, (byte) 0x7F}; + ClickHouseColumn column = ClickHouseColumn.of("s", "String"); + + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + SerializerUtils.serializeData(baos, binary, column); + StringValue read = reader(baos.toByteArray(), true).readValue(column); + Assert.assertEquals(read.toByteArray(), binary); + } + + @Test + public void testWriteStringValueToStringRoundTrip() throws IOException { + byte[] binary = new byte[]{(byte) 0x10, (byte) 0x20, (byte) 0xFE, (byte) 0xFF, (byte) 0x00}; + StringValue value = new StringValue(binary); + ClickHouseColumn column = ClickHouseColumn.of("s", "String"); + + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + SerializerUtils.serializeData(baos, value, column); + StringValue read = reader(baos.toByteArray(), true).readValue(column); + Assert.assertEquals(read.toByteArray(), binary); + } + + @Test + public void testWriteByteArrayToFixedStringRoundTrip() throws IOException { + byte[] binary = new byte[]{(byte) 0xAA, (byte) 0xBB, (byte) 0xCC}; + ClickHouseColumn column = ClickHouseColumn.of("s", "FixedString(3)"); + + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + SerializerUtils.serializeData(baos, binary, column); + StringValue read = reader(baos.toByteArray(), true).readValue(column); + Assert.assertEquals(read.toByteArray(), binary); + } + + @Test + public void testWriteStringValueToFixedStringRoundTrip() throws IOException { + // FixedString serialization must accept a StringValue and write its raw bytes (padded to the + // column width), mirroring the byte[] branch. + byte[] binary = new byte[]{(byte) 0x10, (byte) 0x20, (byte) 0x00}; + StringValue value = new StringValue(binary); + ClickHouseColumn column = ClickHouseColumn.of("s", "FixedString(3)"); + + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + SerializerUtils.serializeData(baos, value, column); + StringValue read = reader(baos.toByteArray(), true).readValue(column); + Assert.assertEquals(read.toByteArray(), binary); + } + + @Test + public void testWriteStringValueToFixedStringPadsShorterValue() throws IOException { + // A StringValue shorter than the column width is right-padded with zero bytes. + byte[] binary = new byte[]{(byte) 0xAB, (byte) 0xCD}; + ClickHouseColumn column = ClickHouseColumn.of("s", "FixedString(5)"); + + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + SerializerUtils.serializeData(baos, new StringValue(binary), column); + StringValue read = reader(baos.toByteArray(), true).readValue(column); + Assert.assertEquals(read.toByteArray(), new byte[]{(byte) 0xAB, (byte) 0xCD, 0, 0, 0}); + } + + // ---- MapBackedRecord (queryAll materialization) ---- + + /** + * Materializes a single-row {@link MapBackedRecord} the same way {@code Client.queryAll(...)} does: + * a {@code RowBinaryWithNamesAndTypes} stream is read into a map and wrapped together with the + * reader's converters and schema. {@code binaryStringSupport} mirrors the client property. + */ + private static MapBackedRecord materializeRow(String[] names, String[] types, Object[] values, + boolean binaryStringSupport) throws IOException { + ByteArrayOutputStream out = new ByteArrayOutputStream(); + BinaryStreamUtils.writeVarInt(out, names.length); + for (String name : names) { + BinaryStreamUtils.writeString(out, name); + } + for (String type : types) { + BinaryStreamUtils.writeString(out, type); + } + for (int i = 0; i < names.length; i++) { + SerializerUtils.serializeData(out, values[i], ClickHouseColumn.of(names[i], types[i])); + } + + RowBinaryWithNamesAndTypesFormatReader reader = new RowBinaryWithNamesAndTypesFormatReader( + new ByteArrayInputStream(out.toByteArray()), + new QuerySettings().setUseTimeZone(TimeZone.getTimeZone("UTC").toZoneId().getId()) + .setOption(ClientConfigProperties.BINARY_STRING_SUPPORT.getKey(), binaryStringSupport), + new BinaryStreamReader.CachingByteBufferAllocator(), null); + + Map record = new LinkedHashMap<>(); + Assert.assertTrue(reader.readRecord(record), "Expected a row to be read"); + return new MapBackedRecord(record, reader.getConvertions(), reader.getSchema()); + } + + @Test + public void testMapBackedRecordReadsStringAsStringValueWhenEnabled() throws IOException { + byte[] binary = new byte[]{(byte) 0xDE, (byte) 0xAD, (byte) 0x00, (byte) 0xBE, (byte) 0xEF}; + MapBackedRecord record = materializeRow(new String[]{"s"}, new String[]{"String"}, + new Object[]{binary}, true); + + // getObject exposes the raw holder + Object value = record.getObject("s"); + Assert.assertTrue(value instanceof StringValue, "Expected StringValue but got " + value.getClass()); + Assert.assertEquals(((StringValue) value).toByteArray(), binary); + + // getByteArray must preserve the raw bytes through both the label and index overloads + Assert.assertEquals(record.getByteArray("s"), binary, "getByteArray(name) must preserve raw bytes"); + Assert.assertEquals(record.getByteArray(1), binary, "getByteArray(index) must preserve raw bytes"); + } + + @Test + public void testMapBackedRecordGetByteArrayIndexAndLabelParity() throws IOException { + // The index and label overloads of getByteArray must behave identically for a String column. + byte[] binary = new byte[]{(byte) 0x01, (byte) 0x80, (byte) 0xFF, (byte) 0x00}; + MapBackedRecord record = materializeRow(new String[]{"s"}, new String[]{"String"}, + new Object[]{binary}, true); + Assert.assertEquals(record.getByteArray(1), record.getByteArray("s")); + Assert.assertEquals(record.getByteArray(1), binary); + } + + @Test + public void testMapBackedRecordFixedStringByteArray() throws IOException { + byte[] binary = new byte[]{(byte) 0xAA, (byte) 0xBB, (byte) 0xCC}; + MapBackedRecord record = materializeRow(new String[]{"s"}, new String[]{"FixedString(3)"}, + new Object[]{binary}, true); + + Assert.assertTrue(record.getObject("s") instanceof StringValue); + Assert.assertEquals(record.getByteArray("s"), binary); + Assert.assertEquals(record.getByteArray(1), binary); + } + + @Test + public void testMapBackedRecordGetStringDecodesStringValue() throws IOException { + String text = "Привет, мир"; + MapBackedRecord record = materializeRow(new String[]{"s"}, new String[]{"String"}, + new Object[]{text}, true); + Assert.assertEquals(record.getString("s"), text); + Assert.assertEquals(record.getString(1), text); + } + + @Test + public void testMapBackedRecordNestedArrayStaysString() throws IOException { + // Even with the feature enabled, nested Array(String) elements stay String; getStringArray must + // work through both overloads and getList must expose String elements. + String[] elements = {"plain", "Привет", ""}; + MapBackedRecord record = materializeRow(new String[]{"a"}, new String[]{"Array(String)"}, + new Object[]{elements}, true); + + Assert.assertEquals(record.getStringArray("a"), elements); + Assert.assertEquals(record.getStringArray(1), elements); + + List list = record.getList("a"); + Assert.assertEquals(list.size(), elements.length); + for (Object element : list) { + Assert.assertTrue(element instanceof String, "Nested array element must be a String"); + } + } + + @Test + public void testMapBackedRecordDefaultReturnsString() throws IOException { + // Default behavior (feature off): top-level Strings are plain String and byte access re-encodes as UTF-8. + String text = "ascii text"; + MapBackedRecord record = materializeRow(new String[]{"s"}, new String[]{"String"}, + new Object[]{text}, false); + + Assert.assertTrue(record.getObject("s") instanceof String); + Assert.assertEquals(record.getString("s"), text); + Assert.assertEquals(record.getByteArray("s"), text.getBytes(StandardCharsets.UTF_8)); + Assert.assertEquals(record.getByteArray(1), text.getBytes(StandardCharsets.UTF_8)); + } + + // ---- getStringArray over arrays whose items are StringValue ---- + + private static BinaryStreamReader.ArrayValue stringValueArray() { + // Build an array whose item type is StringValue (the branch that is not reachable through the + // wire, where nested array elements are always decoded as String). A null element exercises the + // null-guard inside the StringValue mapping. + BinaryStreamReader.ArrayValue array = new BinaryStreamReader.ArrayValue(StringValue.class, 3); + array.set(0, new StringValue("plain".getBytes(StandardCharsets.UTF_8))); + array.set(1, new StringValue("Привет".getBytes(StandardCharsets.UTF_8))); + array.set(2, null); + return array; + } + + @Test + public void testAbstractReaderGetStringArrayFromStringValueItems() { + TableSchema schema = new TableSchema( + Collections.singletonList(ClickHouseColumn.of("a", "Array(String)"))); + InjectableReader reader = new InjectableReader(schema); + reader.setCurrentRecord(new Object[]{stringValueArray()}); + + String[] expected = {"plain", "Привет", null}; + Assert.assertEquals(reader.getStringArray(1), expected); + Assert.assertEquals(reader.getStringArray("a"), expected); + } + + @Test + public void testMapBackedRecordGetStringArrayFromStringValueItems() { + MapBackedRecord record = singleColumnRecord("a", "Array(String)", stringValueArray()); + + String[] expected = {"plain", "Привет", null}; + Assert.assertEquals(record.getStringArray("a"), expected); + Assert.assertEquals(record.getStringArray(1), expected); + } + + // ---- MapBackedRecord.getByteArray edge cases ---- + + @Test + public void testMapBackedRecordGetByteArrayReturnsNullForNullValue() { + // A null value must short-circuit before any string-like or primitive-array handling. + MapBackedRecord record = singleColumnRecord("s", "String", null); + Assert.assertNull(record.getByteArray("s")); + Assert.assertNull(record.getByteArray(1)); + } + + @Test + public void testMapBackedRecordGetByteArrayFromPrimitiveArray() { + // A non-string, primitive Array(Int8) value is not string-like, so getByteArray falls through to + // getPrimitiveArray and returns the backing byte[] (the branch the string-based tests never reach). + BinaryStreamReader.ArrayValue array = new BinaryStreamReader.ArrayValue(byte.class, 3); + array.set(0, (byte) 1); + array.set(1, (byte) -2); + array.set(2, (byte) 127); + MapBackedRecord record = singleColumnRecord("b", "Array(Int8)", array); + + byte[] expected = {(byte) 1, (byte) -2, (byte) 127}; + Assert.assertEquals(record.getByteArray("b"), expected); + Assert.assertEquals(record.getByteArray(1), expected); + } + + // ---- Nested structures round-tripped through the real read path ---- + + @Test + public void testAbstractReaderArrayOfArrayOfString() throws IOException { + List> value = Arrays.asList( + Arrays.asList("a", "b"), + Collections.singletonList("c"), + Collections.emptyList()); + RowBinaryWithNamesAndTypesFormatReader reader = readerForRow( + new String[]{"a"}, new String[]{"Array(Array(String))"}, new Object[]{value}, true); + + Assert.assertEquals(reader.getList("a"), value); + + Object[] outer = reader.getObjectArray("a"); + Assert.assertEquals(outer.length, 3); + Assert.assertEquals((Object[]) outer[0], new Object[]{"a", "b"}); + Assert.assertEquals((Object[]) outer[1], new Object[]{"c"}); + Assert.assertEquals((Object[]) outer[2], new Object[0]); + } + + @Test + public void testMapBackedRecordArrayOfArrayOfString() throws IOException { + List> value = Arrays.asList( + Arrays.asList("a", "b"), + Collections.singletonList("c"), + Collections.emptyList()); + MapBackedRecord record = materializeRow( + new String[]{"a"}, new String[]{"Array(Array(String))"}, new Object[]{value}, true); + + Assert.assertEquals(record.getList("a"), value); + + Object[] outer = record.getObjectArray("a"); + Assert.assertEquals(outer.length, 3); + Assert.assertEquals((Object[]) outer[0], new Object[]{"a", "b"}); + Assert.assertEquals((Object[]) outer[1], new Object[]{"c"}); + Assert.assertEquals((Object[]) outer[2], new Object[0]); + } + + @Test + public void testAbstractReaderTupleStringIntString() throws IOException { + List value = Arrays.asList("first", 7, "third"); + RowBinaryWithNamesAndTypesFormatReader reader = readerForRow( + new String[]{"t"}, new String[]{"Tuple(String, Int32, String)"}, new Object[]{value}, true); + + Assert.assertEquals(reader.getTuple("t"), new Object[]{"first", 7, "third"}); + Assert.assertEquals(reader.getTuple(1), new Object[]{"first", 7, "third"}); + } + + @Test + public void testMapBackedRecordTupleStringIntString() throws IOException { + List value = Arrays.asList("first", 7, "third"); + MapBackedRecord record = materializeRow( + new String[]{"t"}, new String[]{"Tuple(String, Int32, String)"}, new Object[]{value}, true); + + Assert.assertEquals(record.getTuple("t"), new Object[]{"first", 7, "third"}); + Assert.assertEquals(record.getTuple(1), new Object[]{"first", 7, "third"}); + } + + /** + * Serializes a single row and returns a positioned {@link RowBinaryWithNamesAndTypesFormatReader} + * (which is an {@link AbstractBinaryFormatReader}) so its accessors can be exercised over real data. + */ + private static RowBinaryWithNamesAndTypesFormatReader readerForRow(String[] names, String[] types, + Object[] values, boolean binaryStringSupport) + throws IOException { + ByteArrayOutputStream out = new ByteArrayOutputStream(); + BinaryStreamUtils.writeVarInt(out, names.length); + for (String name : names) { + BinaryStreamUtils.writeString(out, name); + } + for (String type : types) { + BinaryStreamUtils.writeString(out, type); + } + for (int i = 0; i < names.length; i++) { + SerializerUtils.serializeData(out, values[i], ClickHouseColumn.of(names[i], types[i])); + } + + RowBinaryWithNamesAndTypesFormatReader reader = new RowBinaryWithNamesAndTypesFormatReader( + new ByteArrayInputStream(out.toByteArray()), + new QuerySettings().setUseTimeZone(TimeZone.getTimeZone("UTC").toZoneId().getId()) + .setOption(ClientConfigProperties.BINARY_STRING_SUPPORT.getKey(), binaryStringSupport), + new BinaryStreamReader.CachingByteBufferAllocator(), null); + Assert.assertNotNull(reader.next(), "Expected a row to be read"); + return reader; + } + + private static MapBackedRecord singleColumnRecord(String name, String type, Object value) { + TableSchema schema = new TableSchema(Collections.singletonList(ClickHouseColumn.of(name, type))); + Map record = new HashMap<>(); + record.put(name, value); + return new MapBackedRecord(record, new Map[]{null}, schema); + } + + /** + * Minimal concrete {@link AbstractBinaryFormatReader} that lets a test inject a materialized record + * directly, so accessor branches can be exercised without a live binary stream. + */ + private static final class InjectableReader extends AbstractBinaryFormatReader { + InjectableReader(TableSchema schema) { + super(new ByteArrayInputStream(new byte[0]), + new QuerySettings().setUseTimeZone("UTC"), + schema, + new BinaryStreamReader.DefaultByteBufferAllocator(), + NO_TYPE_HINT_MAPPING); + } + + void setCurrentRecord(Object[] record) { + this.currentRecord = record; + } + } +} diff --git a/client-v2/src/test/java/com/clickhouse/client/datatypes/DataTypeTests.java b/client-v2/src/test/java/com/clickhouse/client/datatypes/DataTypeTests.java index b24883ca9..3fc39ac46 100644 --- a/client-v2/src/test/java/com/clickhouse/client/datatypes/DataTypeTests.java +++ b/client-v2/src/test/java/com/clickhouse/client/datatypes/DataTypeTests.java @@ -332,6 +332,84 @@ public void testBFloat16InDynamicColumn() throws Exception { Assert.assertEquals(rows.get(0).getObject("v"), Float.valueOf(1.5f)); } + @Data + @AllArgsConstructor + @NoArgsConstructor + public static class DTOForBinaryStringTests { + private int rowId; + + // Mapped to a ClickHouse String column but holds raw (non-UTF-8) bytes. + private byte[] binaryString; + + // Mapped to a ClickHouse FixedString(N) column, also holding raw bytes. + private byte[] fixedBinary; + + public static String tblCreateSQL(String table, int fixedLength) { + return tableDefinition(table, "rowId Int32", "binaryString String", + "fixedBinary FixedString(" + fixedLength + ")"); + } + } + + /** + * Verifies that raw, non-UTF-8 binary content survives a round-trip through {@code String} and + * {@code FixedString} columns when binary string support is enabled. The check covers both the + * binary format reader ({@link ClickHouseBinaryFormatReader#getByteArray}) and the POJO mapping + * path (a {@code byte[]} field bound to a string-backed column). + */ + @Test(groups = {"integration"}) + public void testBinaryStringRoundTrip() throws Exception { + final String table = "test_binary_string_round_trip"; + final int fixedLength = 16; + + // A blob with every possible byte value guarantees invalid UTF-8 sequences (e.g. lone 0x80). + final byte[] binaryBlob = new byte[256]; + for (int i = 0; i < binaryBlob.length; i++) { + binaryBlob[i] = (byte) i; + } + final byte[] fixedBlob = new byte[fixedLength]; + for (int i = 0; i < fixedLength; i++) { + fixedBlob[i] = (byte) (0xFF - i); + } + + final DTOForBinaryStringTests sample = new DTOForBinaryStringTests(1, binaryBlob, fixedBlob); + + try (Client binClient = newClient().binaryStringSupport(true).build()) { + binClient.execute("DROP TABLE IF EXISTS " + table).get(); + binClient.execute(DTOForBinaryStringTests.tblCreateSQL(table, fixedLength)).get(); + + final TableSchema tableSchema = binClient.getTableSchema(table); + binClient.register(DTOForBinaryStringTests.class, tableSchema); + binClient.insert(table, Collections.singletonList(sample)).get().close(); + + // Reader path: getByteArray must return the original raw bytes. + try (QueryResponse response = binClient.query("SELECT * FROM " + table).get()) { + ClickHouseBinaryFormatReader reader = binClient.newBinaryFormatReader(response); + Assert.assertNotNull(reader.next()); + Assert.assertEquals(reader.getByteArray("binaryString"), binaryBlob); + Assert.assertEquals(reader.getByteArray("fixedBinary"), fixedBlob); + } + + // POJO path: a byte[] field bound to a string-backed column must receive the raw bytes. + List pojos = + binClient.queryAll("SELECT * FROM " + table + " ORDER BY rowId", + DTOForBinaryStringTests.class, tableSchema); + Assert.assertEquals(pojos.size(), 1); + Assert.assertEquals(pojos.get(0).getBinaryString(), binaryBlob); + Assert.assertEquals(pojos.get(0).getFixedBinary(), fixedBlob); + } + + // Negative control: without binary string support the String column is decoded as UTF-8, + // which is lossy for this blob, so the round-tripped bytes must NOT match the original. + try (QueryResponse response = client.query("SELECT * FROM " + table).get()) { + ClickHouseBinaryFormatReader reader = client.newBinaryFormatReader(response); + Assert.assertNotNull(reader.next()); + Assert.assertFalse(Arrays.equals(reader.getByteArray("binaryString"), binaryBlob), + "Expected lossy UTF-8 decoding without binary string support"); + } + + client.execute("DROP TABLE IF EXISTS " + table).get(); + } + @Test(groups = {"integration"}) public void testVariantWithSimpleDataTypes() throws Exception { if (isVersionMatch("(,24.8]")) { diff --git a/client-v2/src/test/java/com/clickhouse/client/datatypes/RowBinaryFormatWriterTest.java b/client-v2/src/test/java/com/clickhouse/client/datatypes/RowBinaryFormatWriterTest.java index 333feaa5f..d7f59d796 100644 --- a/client-v2/src/test/java/com/clickhouse/client/datatypes/RowBinaryFormatWriterTest.java +++ b/client-v2/src/test/java/com/clickhouse/client/datatypes/RowBinaryFormatWriterTest.java @@ -6,14 +6,17 @@ import com.clickhouse.client.ClickHouseServerForTest; import com.clickhouse.client.api.Client; import com.clickhouse.client.api.command.CommandSettings; +import com.clickhouse.client.api.data_formats.RowBinaryFormatSerializer; import com.clickhouse.client.api.data_formats.RowBinaryFormatWriter; import com.clickhouse.client.api.data_formats.internal.BinaryStreamReader; +import com.clickhouse.client.api.data_formats.internal.StringValue; import com.clickhouse.client.api.enums.Protocol; import com.clickhouse.client.api.insert.InsertResponse; import com.clickhouse.client.api.insert.InsertSettings; import com.clickhouse.client.api.internal.ServerSettings; import com.clickhouse.client.api.metadata.TableSchema; import com.clickhouse.client.api.query.GenericRecord; +import com.clickhouse.data.ClickHouseColumn; import com.clickhouse.data.ClickHouseFormat; import com.clickhouse.data.ClickHouseVersion; import org.apache.commons.lang3.RandomStringUtils; @@ -22,19 +25,24 @@ import org.testng.annotations.DataProvider; import org.testng.annotations.Test; +import java.io.ByteArrayInputStream; +import java.io.ByteArrayOutputStream; import java.io.IOException; import java.math.BigDecimal; import java.math.BigInteger; import java.net.Inet4Address; import java.net.Inet6Address; +import java.nio.charset.StandardCharsets; import java.time.LocalDate; import java.time.ZonedDateTime; import java.util.ArrayList; import java.util.Arrays; +import java.util.Collections; import java.util.HashMap; import java.util.List; import java.util.Map; import java.util.Random; +import java.util.TimeZone; import java.util.UUID; import java.util.concurrent.TimeUnit; @@ -160,6 +168,11 @@ private static void assertEqualsKinda(Object actual, Object expected) { expected = ((BigDecimal) expected).stripTrailingZeros(); } + if (actual instanceof byte[] && expected instanceof byte[]) { + org.testng.Assert.assertEquals((byte[]) actual, (byte[]) expected); + return; + } + assertEquals(String.valueOf(actual), String.valueOf(expected)); } @@ -458,6 +471,139 @@ public void writeStringsTest() throws Exception { writeTest(tableName, tableCreate, rows); } + @Test (groups = { "integration" }) + public void writeBinaryStringsTest() throws Exception { + String tableName = "rowBinaryFormatWriterTest_writeBinaryStringsTests_" + UUID.randomUUID().toString().replace('-', '_'); + String tableCreate = "CREATE TABLE \"" + tableName + "\" " + + " (id Int32, " + + " string String, " + + " fixed_string FixedString(5), " + + " fixed_string_one FixedString(1) " + + " ) Engine = MergeTree ORDER BY id"; + + // Row 1 is written via setValue(byte[]), row 2 via setString(byte[]); use distinct + // payloads per row so the rows are not identical (identical rows would be collapsed + // by insert deduplication on Cloud's replicated/shared engines). + byte[] binaryData1 = new byte[]{(byte) 0xDE, (byte) 0xAD, (byte) 0xBE, (byte) 0xEF, (byte) 0x00, (byte) 0xFF, (byte) 0x80}; + byte[] fixedStringData1 = new byte[]{(byte) 0xAA, (byte) 0xBB, (byte) 0xCC, (byte) 0xDD, (byte) 0xEE}; + byte[] fixedStringOneData1 = new byte[]{(byte) 0x7F}; + + byte[] binaryData2 = new byte[]{(byte) 0x01, (byte) 0x02, (byte) 0x03, (byte) 0xFE, (byte) 0xFD, (byte) 0x00, (byte) 0x7F}; + byte[] fixedStringData2 = new byte[]{(byte) 0x11, (byte) 0x22, (byte) 0x33, (byte) 0x44, (byte) 0x55}; + byte[] fixedStringOneData2 = new byte[]{(byte) 0x01}; + + // Instead of writeTest which reads back using default string decoding, we write manually + // and query back using typeHintMapping to preserve raw bytes + initTable(tableName, tableCreate, new CommandSettings()); + TableSchema schema = client.getTableSchema(tableName); + + // Write both rows in a single insert: row 1 exercises setValue(byte[]) and row 2 setString(byte[]). + ClickHouseFormat format = ClickHouseFormat.RowBinaryWithDefaults; + try (InsertResponse response = client.insert(tableName, out -> { + RowBinaryFormatWriter w = new RowBinaryFormatWriter(out, schema, format); + w.setValue(schema.nameToColumnIndex("id"), 1); + w.setValue(schema.nameToColumnIndex("string"), binaryData1); + w.setValue(schema.nameToColumnIndex("fixed_string"), fixedStringData1); + w.setValue(schema.nameToColumnIndex("fixed_string_one"), fixedStringOneData1); + w.commitRow(); + + w.setValue(schema.nameToColumnIndex("id"), 2); + w.setString("string", binaryData2); + w.setString("fixed_string", fixedStringData2); + w.setString("fixed_string_one", fixedStringOneData2); + w.commitRow(); + }, format, settings).get()) { + System.out.println("Rows written: " + response.getWrittenRows()); + } + + Client customClient = newClient() + .binaryStringSupport(true) + .build(); + + List records = customClient.queryAll("SELECT * FROM \"" + tableName + "\" ORDER BY id" ); + assertEquals(records.size(), 2); + + GenericRecord row1 = records.get(0); + org.testng.Assert.assertEquals(row1.getByteArray("string"), binaryData1); + org.testng.Assert.assertEquals(row1.getByteArray("fixed_string"), fixedStringData1); + org.testng.Assert.assertEquals(row1.getByteArray("fixed_string_one"), fixedStringOneData1); + + GenericRecord row2 = records.get(1); + org.testng.Assert.assertEquals(row2.getByteArray("string"), binaryData2); + org.testng.Assert.assertEquals(row2.getByteArray("fixed_string"), fixedStringData2); + org.testng.Assert.assertEquals(row2.getByteArray("fixed_string_one"), fixedStringOneData2); + + customClient.close(); + } + + @Test (groups = { "integration" }) + public void writeAndReadImageTest() throws Exception { + // Demonstrates that large binary blobs (here a ~10KB PNG) survive a full write/read round-trip + // through a String column without being corrupted by lossy UTF-8 decoding. + byte[] imageData = readResource("clickhouse-logo.png"); + org.testng.Assert.assertTrue(imageData.length > 1024, "Expected a non-trivial binary payload"); + + String tableName = "rowBinaryFormatWriterTest_writeAndReadImageTest_" + UUID.randomUUID().toString().replace('-', '_'); + String tableCreate = "CREATE TABLE \"" + tableName + "\" " + + " (id Int32, image String) Engine = MergeTree ORDER BY id"; + + initTable(tableName, tableCreate, new CommandSettings()); + TableSchema schema = client.getTableSchema(tableName); + + ClickHouseFormat format = ClickHouseFormat.RowBinaryWithDefaults; + try (InsertResponse response = client.insert(tableName, out -> { + RowBinaryFormatWriter w = new RowBinaryFormatWriter(out, schema, format); + w.setValue(schema.nameToColumnIndex("id"), 1); + w.setValue(schema.nameToColumnIndex("image"), imageData); + w.commitRow(); + }, format, settings).get()) { + System.out.println("Image bytes written: " + imageData.length + ", rows: " + response.getWrittenRows()); + } + + try (Client customClient = newClient().binaryStringSupport(true).build()) { + // Idiomatic path: stream rows and read the binary payload via the index-based getByteArray(int). + try (com.clickhouse.client.api.query.QueryResponse response = + customClient.query("SELECT * FROM \"" + tableName + "\" ORDER BY id").get()) { + com.clickhouse.client.api.data_formats.ClickHouseBinaryFormatReader reader = + customClient.newBinaryFormatReader(response); + org.testng.Assert.assertNotNull(reader.next()); + + int imageIndex = reader.getSchema().nameToColumnIndex("image"); + byte[] streamed = reader.getByteArray(imageIndex); + org.testng.Assert.assertEquals(streamed, imageData, + "Image bytes read via getByteArray(int) must match the source exactly"); + // The name-based overload must agree with the index-based one. + org.testng.Assert.assertEquals(reader.getByteArray("image"), streamed); + } + + List records = customClient.queryAll("SELECT * FROM \"" + tableName + "\" ORDER BY id"); + assertEquals(records.size(), 1); + + GenericRecord record = records.get(0); + // Raw bytes must be preserved exactly, regardless of how they are accessed. + org.testng.Assert.assertEquals(record.getByteArray("image"), imageData, + "Image bytes read back via getByteArray must match the source exactly"); + + com.clickhouse.client.api.data_formats.internal.StringValue value = + (com.clickhouse.client.api.data_formats.internal.StringValue) record.getObject("image"); + org.testng.Assert.assertEquals(value.size(), imageData.length); + org.testng.Assert.assertEquals(value.toByteArray(), imageData, + "StringValue must preserve the full binary payload"); + } + } + + private byte[] readResource(String name) throws IOException { + try (java.io.InputStream is = getClass().getClassLoader().getResourceAsStream(name)) { + org.testng.Assert.assertNotNull(is, "Test resource not found on classpath: " + name); + java.io.ByteArrayOutputStream buffer = new java.io.ByteArrayOutputStream(); + byte[] chunk = new byte[8192]; + int read; + while ((read = is.read(chunk)) != -1) { + buffer.write(chunk, 0, read); + } + return buffer.toByteArray(); + } + } @Test (groups = { "integration" }) public void writeDatetimeTests() throws Exception { @@ -791,4 +937,90 @@ public void writeVariantTests() throws Exception { writeTest(tableName, tableCreate, rows); } + + // ----------------------------------------------------------------------------------------------------------------- + // Unit coverage for the writeString helpers on RowBinaryFormatSerializer and the setString(byte[]) overloads on + // RowBinaryFormatWriter, which the integration tests above do not exercise in isolation. These run without a server. + // ----------------------------------------------------------------------------------------------------------------- + + private static final ClickHouseColumn STRING_COLUMN = ClickHouseColumn.of("s", "String"); + + private enum StringWriteVia { + SERIALIZER, + WRITER_BY_NAME, + WRITER_BY_INDEX + } + + private static BinaryStreamReader stringReader(byte[] data) { + return new BinaryStreamReader(new ByteArrayInputStream(data), TimeZone.getTimeZone("UTC"), null, + new BinaryStreamReader.DefaultByteBufferAllocator(), false, null, true); + } + + private static byte[] writeStringBytes(StringWriteVia via, byte[] payload) throws IOException { + ByteArrayOutputStream out = new ByteArrayOutputStream(); + switch (via) { + case SERIALIZER: + new RowBinaryFormatSerializer(out).writeString(payload); + break; + case WRITER_BY_NAME: { + TableSchema schema = new TableSchema(Collections.singletonList(STRING_COLUMN)); + RowBinaryFormatWriter writer = new RowBinaryFormatWriter(out, schema, ClickHouseFormat.RowBinary); + writer.setString("s", payload); + writer.commitRow(); + break; + } + case WRITER_BY_INDEX: { + TableSchema schema = new TableSchema(Collections.singletonList(STRING_COLUMN)); + RowBinaryFormatWriter writer = new RowBinaryFormatWriter(out, schema, ClickHouseFormat.RowBinary); + writer.setString(1, payload); + writer.commitRow(); + break; + } + } + return out.toByteArray(); + } + + @DataProvider(name = "stringValues") + public static Object[][] stringValues() { + return new Object[][] { + {"ascii", "hello world"}, + {"unicode", "Привет 你好 🚀"}, + }; + } + + @Test(dataProvider = "stringValues") + public void testSerializerWriteString(String description, String value) throws IOException { + ByteArrayOutputStream out = new ByteArrayOutputStream(); + new RowBinaryFormatSerializer(out).writeString(value); + + StringValue read = stringReader(out.toByteArray()).readValue(STRING_COLUMN); + assertEquals(read.asString(), value, description); + assertEquals(read.toByteArray(), value.getBytes(StandardCharsets.UTF_8), description); + } + + @DataProvider(name = "binaryStringWrites") + public static Object[][] binaryStringWrites() { + byte[] arbitraryBinary = new byte[]{(byte) 0x00, (byte) 0xFF, (byte) 0x80, (byte) 0x7F, (byte) 0xAB}; + byte[] utf8Content = "ascii and Юникод".getBytes(StandardCharsets.UTF_8); + + Object[][] cases = new Object[StringWriteVia.values().length * 2][]; + int i = 0; + for (StringWriteVia via : StringWriteVia.values()) { + cases[i++] = new Object[]{via, "arbitrary-binary", arbitraryBinary, null}; + cases[i++] = new Object[]{via, "utf8-content", utf8Content, "ascii and Юникод"}; + } + return cases; + } + + @Test(dataProvider = "binaryStringWrites") + public void testWriteStringFromBytes(StringWriteVia via, String description, byte[] payload, + String expectedString) throws IOException { + byte[] encoded = writeStringBytes(via, payload); + + StringValue read = stringReader(encoded).readValue(STRING_COLUMN); + assertEquals(read.toByteArray(), payload, via + "/" + description); + if (expectedString != null) { + assertEquals(read.asString(), expectedString, via + "/" + description); + } + } } diff --git a/client-v2/src/test/java/com/clickhouse/client/metrics/MetricsTest.java b/client-v2/src/test/java/com/clickhouse/client/metrics/MetricsTest.java index 3356417f1..401917404 100644 --- a/client-v2/src/test/java/com/clickhouse/client/metrics/MetricsTest.java +++ b/client-v2/src/test/java/com/clickhouse/client/metrics/MetricsTest.java @@ -67,8 +67,8 @@ public void testRegisterMetrics() throws Exception { Assert.assertEquals((int) available.value(), 1); Assert.assertEquals((int) leased.value(), 0); + final long maxDelay = isCloud() ? 300 : 15; Runnable task = () -> { - final long maxDelay = isCloud() ? 130 : 15; long t1 = System.currentTimeMillis(); try (QueryResponse response = client.query("SELECT 1").get()) { long t = System.currentTimeMillis() - t1; diff --git a/client-v2/src/test/resources/clickhouse-logo.png b/client-v2/src/test/resources/clickhouse-logo.png new file mode 100644 index 000000000..d68e65e11 Binary files /dev/null and b/client-v2/src/test/resources/clickhouse-logo.png differ diff --git a/docs/features.md b/docs/features.md index 9cb74479a..b720d3931 100644 --- a/docs/features.md +++ b/docs/features.md @@ -18,6 +18,7 @@ This document lists stable, user-visible behavior in `client-v2` and `jdbc-v2` t - Parameterized SQL: Accepts named query parameters and can send them through supported HTTP request encodings. - Result materialization helpers: Provides streaming `Records`, generic row access, and convenience APIs that materialize all rows into generic records or typed POJOs. - Binary format readers: Reads ClickHouse binary result formats including `Native`, `RowBinary`, `RowBinaryWithNames`, and `RowBinaryWithNamesAndTypes`. +- Binary string support: Opt-in through the `binary_string_support` property (or `Client.Builder#binaryStringSupport(boolean)`), disabled by default. The setting is resolved per operation from the merged client and query settings, so it can be overridden for a single request by setting the `binary_string_support` option on the operation's settings (e.g. `QuerySettings#setOption(ClientConfigProperties.BINARY_STRING_SUPPORT.getKey(), true)`) regardless of the client-level default. When enabled, untyped reads (e.g. `GenericRecord.getObject(...)`/`BinaryStreamReader.readValue(...)`) of top-level `String` and `FixedString` columns return a `StringValue`, which preserves the raw bytes (`toByteArray()`/`asByteBuffer()`) and lazily decodes a `String` (`asString()`), instead of eagerly decoding to a `String`. `StringValue` is a read-time holder, not a supported POJO field type: typed `queryAll(...)`/`readToPOJO` binding still maps these columns to `String` (decoded) or `byte[]` (raw bytes) according to the POJO field type. Strings nested inside containers (`Array`, `Map`, `Tuple`, `Nested`, `Variant`, `JSON`) are still read as `String`. - JSONEachRow text reader: Can stream `JSONEachRow` responses through a caller-supplied `JsonParser`, with Jackson and Gson parser factory implementations available as optional classpath dependencies, and infers a best-effort schema from the first row. - Data type conversion: Maps ClickHouse types to Java values for binary reads, POJO binding, and SQL parameter formatting, including date/time handling. - BFloat16 type support: For ClickHouse `24.11+`, reads and writes the `BFloat16` type through generic records, binary readers, POJO binding, and `Nullable`/`Dynamic`/`Variant` wrappers. `BFloat16` maps to the Java `float` type: a read widens the stored 16-bit value losslessly, while a write keeps the high 16 bits of the `float`. In `jdbc-v2`, `BFloat16` maps to `java.sql.Types.FLOAT` / `java.lang.Float` and is read and written through the standard `getFloat`/`setFloat` and `getObject` accessors. @@ -53,6 +54,7 @@ Compatibility-sensitive traits: - Certificate-as-content support is compatibility-sensitive: any certificate or key value containing a PEM begin marker (`-----BEGIN`) is treated as inline PEM content, otherwise it is treated as a file path (also searched in the home directory and on the classpath). - JSONEachRow reading depends on the selected parser factory and request format settings: parser materialization determines Java value types, the reader infers minimal schema from the first row, and JSON number server settings are applied only when `QuerySettings` resolves to `ClickHouseFormat.JSONEachRow` and `json_disable_number_quoting` is enabled. - JSONEachRow schema inference is intentionally best-effort: scalar values use Java-to-ClickHouse type mappings, while JSON arrays and objects are identified structurally as `Array` and `Map`. For arrays, maps, and some nested or ambiguous values, the inferred type may not include the most specific element, key, value, or nested ClickHouse type. +- Binary string support is scope-sensitive and off by default: it only changes how top-level `String`/`FixedString` columns are read (to `StringValue` instead of `String`); nested string values are always read as `String`. `StringValue` is produced only by the read path and has no public constructor (it cannot be instantiated by user code). It is intentionally a mutable, zero-copy holder — it wraps (does not copy) the bytes it is given and `toByteArray()` returns the live backing array when the value spans the whole array (otherwise an exact-size copy), so callers needing an independent snapshot must copy the bytes themselves. ## `jdbc-v2` @@ -73,6 +75,7 @@ Compatibility-sensitive traits: - SQL parsing and classification: Classifies SQL to distinguish queries, updates, inserts, `USE`, and role-changing statements, with selectable parser backends. - JDBC escape processing: Translates supported JDBC escape syntax for dates, timestamps, and functions before execution. - Result set streaming: Streams result sets from ClickHouse binary formats and `FORMAT JSONEachRow`, enforces max-row limits, and manages result-set lifecycle correctly. +- Binary string reads: `ResultSet#getBytes(int|String)` and `ResultSet#getBinaryStream(int|String)` return the raw bytes of a `String`/`FixedString` column. Combined with the `binary_string_support` connection property, non-UTF-8/binary content stored in `String` columns round-trips byte-for-byte; `NULL` values report `null` with `wasNull()` set. `ResultSet#getObject(...)` never exposes the internal `StringValue` holder for these columns: `getObject(column, byte[].class)` returns the raw bytes, while `getObject(column, Object.class)` and the no-type `getObject(column)` overloads return a decoded `String`. - Result-set metadata: Exposes JDBC `ResultSetMetaData` backed by ClickHouse column schema. - Database metadata: Implements JDBC `DatabaseMetaData` for ClickHouse catalogs, schemas, tables, columns, and related capability reporting. - Parameter metadata: Reports prepared-statement parameter counts. @@ -91,6 +94,7 @@ Compatibility-sensitive traits: - String parameters are escaped with backslash-based escaping: backslashes are doubled and single quotes are backslash-escaped before values are wrapped in single quotes. - `?` placeholder detection is SQL-aware and should not treat question marks inside quoted strings, quoted identifiers, comments, casts, or similar syntax as bind parameters. - String-like ClickHouse values have stable JDBC expectations: `String`, `FixedString`, and `Enum` values are returned as strings, while `UUID` is available both as `getString()` and `getObject(..., UUID.class)`. +- Binary access to `String`/`FixedString` columns is compatibility-sensitive: `getBytes(...)` and `getBinaryStream(...)` expose the raw column bytes (not a re-encoded text literal), and a `NULL` column returns `null` with `wasNull()` reporting `true`. The `binary_string_support` connection property is passed through to the underlying `client-v2` transport. - `Geometry` has a stable JDBC mapping: metadata reports SQL type `ARRAY` with type name `Geometry`, read paths return nested Java arrays rather than custom wrappers, and write paths depend on the caller preserving the intended point/array nesting shape. - JDBC `Geometry` writes share the same ambiguity as the client serializer: variant selection is inferred from nesting depth, so `Ring` versus `LineString` and `Polygon` versus `MultiLineString` are not currently distinguishable when writing through the generic `Geometry` path. - JDBC `FORMAT JSONEachRow` support is opt-in through the `jdbc_json_parser_factory` driver property, whose value must be a fully-qualified `JsonParserFactory` class name with a public no-argument constructor; JSONEachRow numeric and structured value behavior follows the selected parser and configured server output settings. Inferred JSON arrays are returned from `ResultSet.getObject(...)` as parser-native `List` values rather than JDBC `Array` values because JSONEachRow does not include element metadata. JDBC temporal typed accessors such as `getTimestamp(...)` are not guaranteed for JSONEachRow result sets; callers that need stable JDBC temporal conversions should use the binary default format or perform application-level conversion from string/object values. diff --git a/jdbc-v2/src/main/java/com/clickhouse/jdbc/ResultSetImpl.java b/jdbc-v2/src/main/java/com/clickhouse/jdbc/ResultSetImpl.java index 62259be99..cd4406cb6 100644 --- a/jdbc-v2/src/main/java/com/clickhouse/jdbc/ResultSetImpl.java +++ b/jdbc-v2/src/main/java/com/clickhouse/jdbc/ResultSetImpl.java @@ -287,7 +287,24 @@ public InputStream getUnicodeStream(int columnIndex) throws SQLException { @Override public InputStream getBinaryStream(int columnIndex) throws SQLException { - return getBinaryStream(columnIndexToName(columnIndex)); + checkClosed(); + try { + if (reader.hasValue(columnIndex)) { + byte[] bytes = reader.getByteArray(columnIndex); + if (bytes == null) { + wasNull = true; + return null; + } + wasNull = false; + return new ByteArrayInputStream(bytes); + } else { + wasNull = true; + return null; + } + } catch (Exception e) { + throw ExceptionUtils.toSqlState(String.format("Method: getBinaryStream(\"%d\") encountered an exception.", columnIndex), + String.format("SQL: [%s]", parentStatement.getLastStatementSql()), e); + } } @Override @@ -470,10 +487,7 @@ public InputStream getUnicodeStream(String columnLabel) throws SQLException { @Override public InputStream getBinaryStream(String columnLabel) throws SQLException { - checkClosed(); - featureManager.unsupportedFeatureThrow("getBinaryStream"); - - return null; + return getBinaryStream(getSchema().nameToColumnIndex(columnLabel)); } @Override diff --git a/jdbc-v2/src/main/java/com/clickhouse/jdbc/internal/JdbcUtils.java b/jdbc-v2/src/main/java/com/clickhouse/jdbc/internal/JdbcUtils.java index bf7292428..a10d2f9e0 100644 --- a/jdbc-v2/src/main/java/com/clickhouse/jdbc/internal/JdbcUtils.java +++ b/jdbc-v2/src/main/java/com/clickhouse/jdbc/internal/JdbcUtils.java @@ -3,6 +3,7 @@ import com.clickhouse.client.api.DataTypeUtils; import com.clickhouse.client.api.data_formats.internal.BinaryStreamReader; import com.clickhouse.client.api.data_formats.internal.InetAddressConverter; +import com.clickhouse.client.api.data_formats.internal.StringValue; import com.clickhouse.data.ClickHouseColumn; import com.clickhouse.data.ClickHouseDataType; import com.clickhouse.data.Tuple; @@ -12,6 +13,7 @@ import java.net.Inet4Address; import java.net.Inet6Address; import java.net.InetAddress; +import java.nio.charset.Charset; import java.nio.charset.StandardCharsets; import java.sql.Date; import java.sql.JDBCType; @@ -291,6 +293,11 @@ public static Object convert(Object value, Class type, ClickHouseColumn colum return value; } + // Special case + if (value instanceof StringValue && type == Object.class) { + return ((StringValue)value).asString(); + } + type = unwrapPrimitiveType(type); if (type.isInstance(value)) { return value; @@ -344,6 +351,12 @@ static Object convertObject(Object value, Class type, ClickHouseColumn column try { if (type == String.class) { return value.toString(); + } else if (type == byte[].class && value instanceof StringValue) { + // Raw bytes of a String/FixedString column read as a StringValue (binary_string_support). + // String and numeric targets already work through the value.toString() branches below. + return ((StringValue) value).toByteArray(); + } else if (type == byte[].class && value instanceof String) { + return ((String)value).getBytes(StandardCharsets.UTF_8); } else if (type == Boolean.class) { String str = value.toString(); return !("false".equalsIgnoreCase(str) || "0".equalsIgnoreCase(str)); diff --git a/jdbc-v2/src/test/java/com/clickhouse/jdbc/JdbcDataTypeTests.java b/jdbc-v2/src/test/java/com/clickhouse/jdbc/JdbcDataTypeTests.java index 423acb8fd..2569c254e 100644 --- a/jdbc-v2/src/test/java/com/clickhouse/jdbc/JdbcDataTypeTests.java +++ b/jdbc-v2/src/test/java/com/clickhouse/jdbc/JdbcDataTypeTests.java @@ -19,6 +19,7 @@ import org.testng.annotations.Test; import java.io.IOException; +import java.io.InputStream; import java.math.BigDecimal; import java.math.BigInteger; import java.net.Inet4Address; @@ -1778,12 +1779,146 @@ public void testStringsUsedAsBytes() throws Exception { for (String[] expected : testData) { assertTrue(rs.next()); assertEquals(new String(rs.getBytes("str"), "UTF-8"), expected[0]); + assertEquals(new String(rs.getObject("str", byte[].class), "UTF-8"), expected[0]); assertEquals(new String(rs.getBytes("fixed"), "UTF-8").replace("\0", ""), expected[1]); } assertFalse(rs.next()); } } + private static byte[] readClickHouseLogo() { + try (InputStream is = JdbcDataTypeTests.class.getResourceAsStream("/ch_logo.png")) { + Assert.assertNotNull(is, "ch_logo.png not found in test resources"); + return is.readAllBytes(); + } catch (Exception e) { + throw new RuntimeException("Failed to read test resource", e); + } + } + + private static void assertEqualsToClickHouseLogo(byte[] actual) { + Assert.assertNotNull(actual, "Read bytes must not be null"); + assertEquals(actual.length, CH_LOGO_PNG.length, "Read byte count must match ch_logo.png size"); + assertEquals(actual, CH_LOGO_PNG, "Read bytes must match ch_logo.png content"); + } + + private static final byte[] CH_LOGO_PNG = readClickHouseLogo(); + + @Test(groups = { "integration" }) + public void testBinaryStringSupportGetBytes() throws Exception { + // ch_logo.png is real binary content that is not valid UTF-8, so it must survive a + // round-trip through a String column byte-for-byte when binary_string_support is enabled. + + runQuery("CREATE TABLE test_binary_string_get_bytes (id Int8, str String) ENGINE = MergeTree ORDER BY ()"); + + try (Connection conn = getJdbcConnection(); + PreparedStatement insert = conn.prepareStatement("INSERT INTO test_binary_string_get_bytes VALUES (?, ?)")) { + insert.setInt(1, 1); + insert.setBytes(2, CH_LOGO_PNG); + insert.executeUpdate(); + } + + Properties props = new Properties(); + props.put(ClientConfigProperties.BINARY_STRING_SUPPORT.getKey(), "true"); + + try (Connection conn = getJdbcConnection(props); + Statement stmt = conn.createStatement(); + ResultSet rs = stmt.executeQuery("SELECT * FROM test_binary_string_get_bytes ORDER BY id")) { + assertTrue(rs.next()); + assertEqualsToClickHouseLogo(rs.getBytes("str")); + assertEqualsToClickHouseLogo(rs.getBytes(2)); + assertFalse(rs.wasNull()); + assertFalse(rs.next()); + } + } + + @Test(groups = { "integration" }) + public void testBinaryStringSupportGetBinaryStream() throws Exception { + runQuery("CREATE TABLE test_binary_string_stream (id Int8, str String, nullable_str Nullable(String)) ENGINE = MergeTree ORDER BY ()"); + + try (Connection conn = getJdbcConnection(); + PreparedStatement insert = conn.prepareStatement("INSERT INTO test_binary_string_stream VALUES (?, ?, ?)")) { + insert.setInt(1, 1); + insert.setBytes(2, CH_LOGO_PNG); + insert.setNull(3, Types.VARCHAR); + insert.executeUpdate(); + } + + Properties props = new Properties(); + props.put(ClientConfigProperties.BINARY_STRING_SUPPORT.getKey(), "true"); + + try (Connection conn = getJdbcConnection(props); + Statement stmt = conn.createStatement(); + ResultSet rs = stmt.executeQuery("SELECT * FROM test_binary_string_stream ORDER BY id")) { + assertTrue(rs.next()); + + // by column label (delegates to the index-based implementation) + try (InputStream stream = rs.getBinaryStream("str")) { + Assert.assertNotNull(stream); + assertEqualsToClickHouseLogo(stream.readAllBytes()); + } + assertFalse(rs.wasNull()); + + // by column index + try (InputStream stream = rs.getBinaryStream(2)) { + Assert.assertNotNull(stream); + assertEqualsToClickHouseLogo(stream.readAllBytes()); + } + assertFalse(rs.wasNull()); + + // null value on a nullable column + assertNull(rs.getBinaryStream("nullable_str")); + assertTrue(rs.wasNull()); + assertNull(rs.getBytes("nullable_str")); + assertTrue(rs.wasNull()); + + assertFalse(rs.next()); + } + } + + @Test(groups = { "integration" }) + public void testBinaryStringSupportGetObject() throws Exception { + // With binary_string_support enabled the read path returns an internal StringValue holder for + // String/FixedString columns. getObject must never leak that holder: it should return a decoded + // String for Object.class and the no-type overload, and exact raw bytes for byte[].class. + runQuery("CREATE TABLE test_binary_string_get_object (id Int8, str String, txt String) ENGINE = MergeTree ORDER BY ()"); + + String text = "Hello, ClickHouse!"; + try (Connection conn = getJdbcConnection(); + PreparedStatement insert = conn.prepareStatement("INSERT INTO test_binary_string_get_object VALUES (?, ?, ?)")) { + insert.setInt(1, 1); + insert.setBytes(2, CH_LOGO_PNG); + insert.setString(3, text); + insert.executeUpdate(); + } + + Properties props = new Properties(); + props.put(ClientConfigProperties.BINARY_STRING_SUPPORT.getKey(), "true"); + + try (Connection conn = getJdbcConnection(props); + Statement stmt = conn.createStatement(); + ResultSet rs = stmt.executeQuery("SELECT * FROM test_binary_string_get_object ORDER BY id")) { + assertTrue(rs.next()); + + // byte[].class must return the exact bytes without lossy decoding + Object bytesObj = rs.getObject("str", byte[].class); + assertTrue(bytesObj instanceof byte[], "getObject(byte[].class) should return byte[], got: " + bytesObj.getClass().getName()); + assertEqualsToClickHouseLogo((byte[]) bytesObj); + assertFalse(rs.wasNull()); + + // Object.class must return a decoded String, not the internal StringValue holder + Object textObj = rs.getObject("txt", Object.class); + assertTrue(textObj instanceof String, "getObject(Object.class) should return String, got: " + textObj.getClass().getName()); + assertEquals(textObj, text); + + // The no-type overload must also return a String + Object defaultObj = rs.getObject("txt"); + assertTrue(defaultObj instanceof String, "getObject() should return String, got: " + defaultObj.getClass().getName()); + assertEquals(defaultObj, text); + + assertFalse(rs.next()); + } + } + @Test(groups = { "integration" }) public void testNestedArrays() throws Exception { try (Connection conn = getJdbcConnection()) { diff --git a/jdbc-v2/src/test/java/com/clickhouse/jdbc/metadata/DatabaseMetaDataTest.java b/jdbc-v2/src/test/java/com/clickhouse/jdbc/metadata/DatabaseMetaDataTest.java index 6b58011c7..7ea68db19 100644 --- a/jdbc-v2/src/test/java/com/clickhouse/jdbc/metadata/DatabaseMetaDataTest.java +++ b/jdbc-v2/src/test/java/com/clickhouse/jdbc/metadata/DatabaseMetaDataTest.java @@ -1,6 +1,7 @@ package com.clickhouse.jdbc.metadata; import com.clickhouse.client.ClickHouseServerForTest; +import com.clickhouse.client.api.ClientConfigProperties; import com.clickhouse.data.ClickHouseDataType; import com.clickhouse.data.ClickHouseVersion; import com.clickhouse.jdbc.ClientInfoProperties; @@ -147,6 +148,67 @@ public void testGetColumns() throws Exception { } } + /** + * Database metadata is materialized internally by querying ClickHouse system tables, whose results contain + * top-level {@code String} columns (e.g. {@code TABLE_NAME}, {@code COLUMN_NAME}, {@code TYPE_NAME}). When the + * connection enables {@code binary_string_support}, those reads must still surface proper {@link String} values + * so the JDBC metadata API keeps working unchanged. + */ + @Test(groups = {"integration"}) + public void testGetColumnsWithBinaryStringSupport() throws Exception { + Properties props = new Properties(); + props.put(ClientConfigProperties.BINARY_STRING_SUPPORT.getKey(), "true"); + + try (Connection conn = getJdbcConnection(props)) { + final String tableName = "get_columns_binary_string_support_test"; + try (Statement stmt = conn.createStatement()) { + stmt.executeUpdate("CREATE TABLE " + tableName + + " (id Int32, name String NOT NULL, v1 Nullable(Int8), v2 Array(Int8)) " + + "ENGINE MergeTree ORDER BY tuple()"); + } + + DatabaseMetaData dbmd = conn.getMetaData(); + + try (ResultSet rs = dbmd.getColumns(null, getDatabase(), tableName, null)) { + assertTrue(rs.next()); + assertEquals(rs.getString("TABLE_SCHEM"), getDatabase()); + assertEquals(rs.getString("TABLE_NAME"), tableName); + assertEquals(rs.getString("COLUMN_NAME"), "id"); + assertEquals(rs.getInt("DATA_TYPE"), Types.INTEGER); + assertEquals(rs.getString("TYPE_NAME"), "Int32"); + assertFalse(rs.getBoolean("NULLABLE")); + + assertTrue(rs.next()); + assertEquals(rs.getString("TABLE_NAME"), tableName); + assertEquals(rs.getString("COLUMN_NAME"), "name"); + assertEquals(rs.getInt("DATA_TYPE"), Types.VARCHAR); + assertEquals(rs.getString("TYPE_NAME"), "String"); + assertFalse(rs.getBoolean("NULLABLE")); + + assertTrue(rs.next()); + assertEquals(rs.getString("COLUMN_NAME"), "v1"); + assertEquals(rs.getString("TYPE_NAME"), "Nullable(Int8)"); + assertTrue(rs.getBoolean("NULLABLE")); + + assertTrue(rs.next()); + assertEquals(rs.getString("COLUMN_NAME"), "v2"); + assertEquals(rs.getInt("DATA_TYPE"), Types.ARRAY); + assertEquals(rs.getString("TYPE_NAME"), "Array(Int8)"); + } + + // getTables exercises a different system-table query whose String columns must also stay String. + try (ResultSet rs = dbmd.getTables(null, getDatabase(), tableName, null)) { + assertTrue(rs.next()); + assertEquals(rs.getString("TABLE_SCHEM"), getDatabase()); + assertEquals(rs.getString("TABLE_NAME"), tableName); + Object tableNameObj = rs.getObject("TABLE_NAME"); + assertTrue(tableNameObj instanceof String, + "Metadata String columns must be plain String even with binary_string_support enabled, but got " + + (tableNameObj == null ? "null" : tableNameObj.getClass().getName())); + } + } + } + @Test(groups = { "integration" }) public void testGetColumnsBFloat16() throws Exception { if (isVersionMatch("(,24.10]")) {