Skip to content

[SPARK-58044][SQL] Support the TIME data type in the aggregate function avg#57338

Open
marcuslin123 wants to merge 2 commits into
apache:masterfrom
marcuslin123:SPARK-58044-time-avg
Open

[SPARK-58044][SQL] Support the TIME data type in the aggregate function avg#57338
marcuslin123 wants to merge 2 commits into
apache:masterfrom
marcuslin123:SPARK-58044-time-avg

Conversation

@marcuslin123

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Add TimeType support to the avg/mean aggregate function. Previously AVG() only accepted numeric and ANSI interval types and rejected TIME. Since TIME is internally a Long number of nanoseconds since midnight, it can be averaged like the interval types.

SELECT avg(t) FROM VALUES (TIME'10:00:00'), (TIME'14:00:00') AS tab(t);
-- 12:00:00

The changes:

  • Average.scala: accept AnyTimeType in inputTypes; map a TIME input to a TIME output of the same precision in resultType; accumulate the sum as LongType; and add a TIME branch to evaluateExpression that divides the summed nanoseconds by the count and re-tags the result as TimeType(precision).
  • To accumulate the sum, the child value is converted to the sum's type via PreciseTimestampConversion rather than a regular cast. A regular CAST(time AS BIGINT) is lossy for this purpose — it truncates to whole seconds — so it would corrupt the average. PreciseTimestampConversion reinterprets the underlying Long nanoseconds without changing the value.
  • New TimeDivide expression (in intervalExpressions.scala, modeled on DivideDTInterval) that divides a Long-nanosecond sum by the count with HALF_UP rounding and returns a TimeType(precision). DivideDTInterval could not be reused because its output type is hardcoded to DayTimeIntervalType, and there is no CAST(BIGINT AS TIME) to re-wrap the result.
  • New TypeUtils.checkForAnsiIntervalOrNumericOrTimeType helper. Sum's existing checkForAnsiIntervalOrNumericType is intentionally left unchanged, since Sum does not (yet) support TIME.

The output preserves the input precision (e.g. avg over a TIME(3) column yields TIME(3)). This mirrors how min/max/percentile already support TIME.

Why are the changes needed?

TIME is orderable and quantifiable, and min, max, percentile, approx_percentile, etc. already support it. avg was an inconsistent gap: to average a TIME column, users had to cast to a numeric type, average, and cast back. Native support lets them write avg(time_col) directly.

Does this PR introduce any user-facing change?

Yes. avg/mean now accepts TIME columns and returns a TIME average (same precision). Previously it raised a DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE error.

How was this patch tested?

Added a test in DataFrameAggregateSuite covering: basic average, sub-second rounding, NULL handling, all-null/empty input (returns NULL), precision preservation (TIME(3)), grouped average, and the SQL avg surface over TIME literals. Updated the expected error message in ExpressionTypeCheckingSuite (avg's required-type list now includes TIME). Full DataFrameAggregateSuite (168 tests) passes with no regressions, including the existing interval-avg test.

Was this patch authored or co-authored using generative AI tooling?

Generative AI tooling (Claude Code) was used as an assistive tool for implementation guidance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant