[datafusion-spark] Support 2-argument ceil(value, scale)

[diegoQuinas]

Which issue does this PR close?

Part of #21560

Rationale for this change

The Spark ceil function supports an optional scale parameter that controls
the decimal position to round up to. This was not yet implemented in
datafusion-spark.

What changes are included in this PR?

• Updated Signature to accept 1 or 2 arguments, following the same pattern as SparkRound
• Updated return_type: floats preserve their type when a scale is provided (instead of returning Int64); scale=0 preserves the original behavior
• Added get_scale() helper to extract the optional scale argument, returning None for NULL scale (which produces a NULL result)
• Added ceil_float() helper for ceiling floats at arbitrary decimal positions
• Updated spark_ceil_scalar and spark_ceil_array to apply the scale

Are these changes tested?

Yes, unit tests covering:

• Float64/Float32 scalar with positive scale, negative scale, zero scale, and NULL scale
• Float64 array with scale
• Existing 1-argument tests continue to pass

Are there any user-facing changes?

Yes — ceil(expr, scale) is now supported in addition to the existing ceil(expr).

[kosiew]

@diegoQuinas
Thanks for the work here.
There are a few inconsistencies between the declared return types and the actual execution paths that could lead to planner/runtime mismatches. I’ve called those out below. Once those are addressed and we have a couple of type-contract tests in place, this should be in a much safer spot.

[kosiew]

@diegoQuinas
Nice follow-up overall. This addresses all the original comments and the refactor looks clean. The type handling is much more consistent now and the SLT coverage is solid. I left a few small suggestions, mostly around the decimal 2-arg behavior and some minor readability and edge-case clarifications.

[Jefffrey]

If we aren't supporting this yet we should keep the original issue open instead of marking this PR as closing it

[diegoQuinas]

I'm working on this. Maybe this night I add it.

[github-actions]

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
    Building datafusion-spark v53.1.0 (current)
       Built [  57.513s] (current)
     Parsing datafusion-spark v53.1.0 (current)
      Parsed [   0.060s] (current)
    Building datafusion-spark v53.1.0 (baseline)
       Built [  56.585s] (baseline)
     Parsing datafusion-spark v53.1.0 (baseline)
      Parsed [   0.062s] (baseline)
    Checking datafusion-spark v53.1.0 -> v53.1.0 (no change; assume patch)
     Checked [   0.362s] 222 checks: 218 pass, 4 fail, 0 warn, 30 skip

--- failure function_missing: pub fn removed or renamed ---

Description:
A publicly-visible function cannot be imported by its prior path. A `pub use` may have been removed, or the function itself may have been renamed or removed entirely.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.47.0/src/lints/function_missing.ron

Failed in:
  function datafusion_spark::function::math::pow, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/6cd247a557d226d9f3b72e3f2b0f8df2be95b610/datafusion/spark/src/function/math/mod.rs:46
  function datafusion_spark::function::math::expr_fn::pow, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/6cd247a557d226d9f3b72e3f2b0f8df2be95b610/datafusion/spark/src/function/math/mod.rs:71
  function datafusion_spark::expr_fn::pow, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/6cd247a557d226d9f3b72e3f2b0f8df2be95b610/datafusion/spark/src/function/math/mod.rs:71

--- failure function_parameter_count_changed: pub fn parameter count changed ---

Description:
A publicly-visible function now takes a different number of parameters.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#fn-change-arity
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.47.0/src/lints/function_parameter_count_changed.ron

Failed in:
  datafusion_spark::function::math::expr_fn::ceil now takes 2 parameters instead of 1, in /home/runner/work/datafusion/datafusion/datafusion/spark/src/function/math/mod.rs:59
  datafusion_spark::expr_fn::ceil now takes 2 parameters instead of 1, in /home/runner/work/datafusion/datafusion/datafusion/spark/src/function/math/mod.rs:59

--- failure module_missing: pub module removed or renamed ---

Description:
A publicly-visible module cannot be imported by its prior path. A `pub use` may have been removed, or the module may have been renamed, removed, or made non-public.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.47.0/src/lints/module_missing.ron

Failed in:
  mod datafusion_spark::function::math::pow, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/6cd247a557d226d9f3b72e3f2b0f8df2be95b610/datafusion/spark/src/function/math/pow.rs:18

--- failure struct_missing: pub struct removed or renamed ---

Description:
A publicly-visible struct cannot be imported by its prior path. A `pub use` may have been removed, or the struct itself may have been renamed or removed entirely.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.47.0/src/lints/struct_missing.ron

Failed in:
  struct datafusion_spark::function::math::pow::SparkPow, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/6cd247a557d226d9f3b72e3f2b0f8df2be95b610/datafusion/spark/src/function/math/pow.rs:42

     Summary semver requires new major version: 4 major and 0 minor checks failed
    Finished [ 116.160s] datafusion-spark
    Building datafusion-sqllogictest v53.1.0 (current)
       Built [ 169.240s] (current)
     Parsing datafusion-sqllogictest v53.1.0 (current)
      Parsed [   0.023s] (current)
    Building datafusion-sqllogictest v53.1.0 (baseline)
       Built [ 168.895s] (baseline)
     Parsing datafusion-sqllogictest v53.1.0 (baseline)
      Parsed [   0.024s] (baseline)
    Checking datafusion-sqllogictest v53.1.0 -> v53.1.0 (no change; assume patch)
     Checked [   0.091s] 222 checks: 222 pass, 30 skip
     Summary no semver update required
    Finished [ 341.391s] datafusion-sqllogictest

[diegoQuinas]

@Jefffrey , I will merge this one but not closing the issue since @kosiew already approved it. I will open another PR for the 2-argument for decimal inputs.

[Jefffrey]

I was checking some outputs against Spark 4.1.2, and it seems they actually operate only on decimal types for the 2-arg variant. See:

>>> spark.sql("select ceil(-25::int)").printSchema()
root
 |-- CEIL(CAST(-25 AS INT)): long (nullable = true)

>>> spark.sql("select ceil(-25::int, -1)").printSchema()
root
 |-- ceil(CAST(-25 AS INT), -1): decimal(11,0) (nullable = true)

And see reference code:

case class RoundCeil(child: Expression, scale: Expression)
  extends RoundBase(child, scale, BigDecimal.RoundingMode.CEILING, "ROUND_CEILING") {

  override def inputTypes: Seq[AbstractDataType] = Seq(DecimalType, IntegerType)

  override def nodeName: String = "ceil"

  override protected def withNewChildrenInternal(
      newLeft: Expression, newRight: Expression): RoundCeil =
    copy(child = newLeft, scale = newRight)
}

• https://github.com/apache/spark/blob/d6036f470d750a8296016f4a8a88c17679295505/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala#L332-L342
• Input types seem to force it to decimal

So I think this PR won't align with Spark since we need to operate only on decimals (and subsequently return decimal types)?