Native panic casting to negative-scale decimal when spark.sql.legacy.allowNegativeScaleOfDecimal=true

[andygrove]

Describe the bug

With spark.sql.legacy.allowNegativeScaleOfDecimal=true, casting any integer column to a negative-scale decimal crashes the native engine:

Comet native panic: panicked at library/core/src/num/mod.rs:475:5:
attempt to multiply with overflow

spark.sql.legacy.allowNegativeScaleOfDecimal is one of the configs listed under #4180. CometCast does consult it (CometCast.scala:279-289), but only for the decimal → string direction: a negative-scale target is reported Compatible() when the flag is on. Casting into a negative-scale decimal, and arithmetic on negative-scale decimals, are not guarded at all.

Note that the native workspace sets overflow-checks = false for the release profile (native/Cargo.toml:67), so a release build will not panic here. The same multiplication wraps instead, which would turn this into silently wrong values rather than a crash. I verified the panic on a debug build and the profile setting by inspection; I did not run a release build to confirm what the wrapped result looks like.

Steps to reproduce

Negative scale cannot be written in SQL (the parser rejects decimal(20,-5) outright), so this needs the DataFrame API:

import org.apache.spark.sql.functions.col
import org.apache.spark.sql.types.DecimalType

spark.conf.set("spark.sql.legacy.allowNegativeScaleOfDecimal", "true")

val path = "/tmp/negscale"
spark.sql("SELECT cast(n as bigint) as n FROM VALUES (1),(2),(3) AS v(n)")
  .write.mode("overwrite").parquet(path)
val df = spark.read.parquet(path)

spark.conf.set("spark.comet.enabled", "false")
df.select(col("n").cast(DecimalType(10, -1))).collect()
// [0E+1], [0E+1], [0E+1]   (Spark rounds to the negative scale)

spark.conf.set("spark.comet.enabled", "true")
df.select(col("n").cast(DecimalType(10, -1))).collect()
// org.apache.comet.CometNativeException: native panic: attempt to multiply with overflow

Also reproduced with:

• DecimalType(20, -5) — same panic;
• arithmetic on the result, n + n / n * 2 — attempt to subtract with overflow;
• casting the negative-scale decimal back to string — same panic, i.e. the one direction CometCast does guard is unreachable because producing the value crashes first.

With spark.sql.legacy.allowNegativeScaleOfDecimal=false both engines raise NEGATIVE_SCALE_DISALLOWED identically, so only the true case is affected.

Expected behavior

Comet should either handle negative-scale decimals correctly, or report Unsupported/Incompatible for any cast or arithmetic involving a negative-scale decimal so the expression falls back to Spark. It should not panic, and it should not wrap silently in release builds.

Additional context

• Reproduced on apache/main @ 0761e549a, default Maven profile (Spark 4.1.2), macOS aarch64, debug build.
• Found while auditing the config inventory in Audit Spark SQL configs that affect query semantics and ensure Comet honors them (or falls back) #4180.

[0lai0]

take

[comphead]

this should be closed by #4799