diff --git a/docs/source/contributor-guide/spark_expressions_support.md b/docs/source/contributor-guide/spark_expressions_support.md index 8f82847bb6..774d3b2b05 100644 --- a/docs/source/contributor-guide/spark_expressions_support.md +++ b/docs/source/contributor-guide/spark_expressions_support.md @@ -163,17 +163,61 @@ ### bitwise_funcs - [x] `&` + - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8. + - Spark 3.5.8 (audited 2026-05-27): baseline. `BitwiseAnd(left, right) extends BinaryArithmetic` over `IntegralType`. Comet routes via `CometBitwiseAnd` to the proto's `bitwise_and` binary expression. + - Spark 4.0.1 (audited 2026-05-27): semantics unchanged; `NullIntolerant` trait replaced by `nullIntolerant: Boolean`. + - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1. - [x] `<<` + - Spark 3.4.3 (audited 2026-05-27): only the function form `shiftleft(...)` exists; the `<<` operator alias is added in Spark 4.0. + - Spark 3.5.8 (audited 2026-05-27): identical to 3.4.3. + - Spark 4.0.1 (audited 2026-05-27): `<<` parses to `ShiftLeft` (the same `case class` as `shiftleft`) via a parser alias. `ShiftLeft` now extends a shared `BitShiftOperation` trait; eval semantics unchanged. Comet `CometShiftLeft` casts the `right` operand to `LongType` when `left.dataType == LongType` because DataFusion's bitwise shift requires matching operand types. + - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1. - [x] `>>` + - Spark 3.4.3 (audited 2026-05-27): same as `<<` history — only the function form exists pre-4.0. + - Spark 3.5.8 (audited 2026-05-27): identical to 3.4.3. + - Spark 4.0.1 (audited 2026-05-27): `>>` parses to `ShiftRight`. Comet `CometShiftRight` mirrors the same operand-cast logic as `CometShiftLeft`. + - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1. - [ ] `>>>` - [x] `^` + - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8. + - Spark 3.5.8 (audited 2026-05-27): baseline. `BitwiseXor(left, right) extends BinaryArithmetic` over `IntegralType`. Comet routes via `CometBitwiseXor` to the proto's `bitwise_xor` binary expression. + - Spark 4.0.1 (audited 2026-05-27): semantics unchanged; `nullIntolerant` field refactor. + - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1. - [x] bit_count + - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8. + - Spark 3.5.8 (audited 2026-05-27): baseline. `BitwiseCount(child) extends UnaryExpression`; accepts `IntegralType` or `BooleanType` and returns the population count as `IntegerType`. Wired as `CometScalarFunction("bit_count")`. + - Spark 4.0.1 (audited 2026-05-27): semantics unchanged. + - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1. - [x] bit_get + - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8. + - Spark 3.5.8 (audited 2026-05-27): baseline. `BitwiseGet(left, right) extends BinaryExpression`; `inputTypes = (IntegralType, IntegerType) -> ByteType`. The position is bounds-checked at runtime via `BitwiseGetUtil.checkPosition`. Comet routes via `CometBitwiseGet` to the native `bit_get` scalar. + - Spark 4.0.1 (audited 2026-05-27): semantics unchanged. + - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1. - [x] getbit + - Spark 3.4.3 (audited 2026-05-27): registry alias of `BitwiseGet`. Same support as `bit_get`. + - Spark 3.5.8 (audited 2026-05-27): identical to 3.4.3. + - Spark 4.0.1 (audited 2026-05-27): identical to 3.4.3. + - Spark 4.1.1 (audited 2026-05-27): identical to 3.4.3. - [x] shiftright + - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8. + - Spark 3.5.8 (audited 2026-05-27): baseline. Function name for `ShiftRight`. Same support as the `>>` operator alias added in 4.0. + - Spark 4.0.1 (audited 2026-05-27): identical to 3.5.8 (the `>>` operator is the only addition). + - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1. - [x] shiftrightunsigned + - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8. + - Spark 3.5.8 (audited 2026-05-27): baseline. `ShiftRightUnsigned(left, right) extends BinaryExpression`; uses Java's `>>>` (zero-fill) semantics. Comet wires as `CometScalarFunction("shiftrightunsigned")` (added in #4375). + - Spark 4.0.1 (audited 2026-05-27): now extends the shared `BitShiftOperation` trait; eval semantics unchanged. The `>>>` operator alias is added at the parser level. + - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1. - [x] `|` + - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8. + - Spark 3.5.8 (audited 2026-05-27): baseline. `BitwiseOr(left, right) extends BinaryArithmetic` over `IntegralType`. Comet routes via `CometBitwiseOr` to the proto's `bitwise_or` binary expression. + - Spark 4.0.1 (audited 2026-05-27): semantics unchanged; `nullIntolerant` field refactor. + - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1. - [x] `~` + - Spark 3.4.3 (audited 2026-05-27): identical to 3.5.8. + - Spark 3.5.8 (audited 2026-05-27): baseline. `BitwiseNot(child) extends UnaryExpression`; accepts `IntegralType` and returns the bitwise complement of the same type. Comet routes via `CometBitwiseNot` to the native `bitwise_not` scalar. + - Spark 4.0.1 (audited 2026-05-27): semantics unchanged; `nullIntolerant` field refactor. + - Spark 4.1.1 (audited 2026-05-27): identical to 4.0.1. ### collection_funcs