fix: time ± interval returns a wrapped time instead of an interval#23279
fix: time ± interval returns a wrapped time instead of an interval#23279vismaytiwari wants to merge 7 commits into
time ± interval returns a wrapped time instead of an interval#23279Conversation
`time + interval`, `interval + time`, and `time - interval` previously widened the time operand into an interval, so `time '23:30' + interval '2 hours'` returned a `25 hours 30 mins` interval rather than wrapping within the 24-hour clock. Following the existing `Date - Date` special case, coercion now keeps the result as `time`, and a new `apply_time_interval` kernel adds/subtracts the interval's sub-day component modulo 24 hours to match PostgreSQL and DuckDB. Whole months and days are ignored, and all four time units are supported. Closes apache#22265
0a33c81 to
5b6be8b
Compare
|
Fixed the rustfmt failure from the first run — fmt, the unit tests, and the datetime sqllogictests all pass locally. The workflows just need approval to run on the updated commit. |
Jefffrey
left a comment
There was a problem hiding this comment.
it makes sense to make this change to me, given two other systems behave similarly and having time + interval = interval seems non-obvious (our current behaviour)
we'll need a note in the upgrade guide since this is a user facing change
| // Normalize to a single representation: time as Time64(ns), interval as MonthDayNano. | ||
| let time_ns_arr = cast(time_array, &DataType::Time64(TimeUnit::Nanosecond))?; |
There was a problem hiding this comment.
similarly here i feel we can avoid this cast
There was a problem hiding this comment.
Agreed. I moved the normalization up into the coercion layer, so the time operand now arrives as Time64(Nanosecond) and there's no cast here anymore. That does make the result Time64(Nanosecond) regardless of the input time unit — I called that out in the upgrade guide.
- Coerce the operands of `time ± interval` to Time64(Nanosecond) and Interval(MonthDayNano) in the type coercion layer, so the physical layer no longer casts either operand. The result type is Time64(Nanosecond). - Rewrite apply_time_interval to operate directly on ColumnarValue (like apply_date_subtraction) instead of materializing arrays, and use i64 arithmetic with rem_euclid rather than i128. - Note the user-facing behavior change in the 55.0.0 upgrade guide and cover array/null inputs in the sqllogictest.
|
my main concern now is that we always cast the time to im not sure looking to other systems can help us here since i dont think they have different granularities for their time types like we do here |
|
@Jefffrey yeah, I don't think other systems help here — but we've actually got the precedent in-tree. That gives us a consistent option: preserve the time unit and apply the interval at that resolution, so I lean toward preserving the unit — it matches timestamp/date arithmetic and keeps a |
|
thanks for double checking; if we do that for timestamps then it makes sense to do for time as well 👍 |
Rather than widening the result to `Time64(Nanosecond)`, keep the time operand's own unit and apply the interval at that resolution -- mirroring how `timestamp/date + interval` already behave via arrow's kernels (which preserve the unit and truncate finer interval precision). So `time(s) + interval '1 nanosecond'` is a no-op, exactly like `timestamp(s) + interval '1 nanosecond'`. The coercion arm now keeps the time type as the result type instead of forcing `Time64(Nanosecond)`, and `apply_time_interval` dispatches per unit through a generic `wrap_time_interval<T>`. Updates the sqllogictests (per-unit arrow_typeof assertions + finer-than-unit truncation cases) and the 55.0.0 upgrade note.
|
Done — pushed. The result now keeps the input time's unit instead of widening to Added per-unit |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #23279 +/- ##
=======================================
Coverage ? 80.65%
=======================================
Files ? 1086
Lines ? 366212
Branches ? 366212
=======================================
Hits ? 295359
Misses ? 53256
Partials ? 17597 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
| /// by the coercion layer) is applied at that resolution, mirroring `timestamp + interval`. | ||
| /// Only the sub-day portion of the interval affects a time-of-day -- whole months and | ||
| /// days are ignored, matching PostgreSQL -- and any interval precision finer than the | ||
| /// time's unit is truncated (so `time(s) + interval '1 nanosecond'` is a no-op). |
There was a problem hiding this comment.
something to consider is how subtraction affects this. see (checked out on this branch):
> select arrow_cast('2026-07-17T10:00:00', 'Timestamp(s)') - interval '1 nanosecond';
+----------------------------------------------------------------------------------------------------------------------------------------------------+
| arrow_cast(Utf8("2026-07-17T10:00:00"),Utf8("Timestamp(s)")) - IntervalMonthDayNano("IntervalMonthDayNano { months: 0, days: 0, nanoseconds: 1 }") |
+----------------------------------------------------------------------------------------------------------------------------------------------------+
| 2026-07-17T09:59:59 |
+----------------------------------------------------------------------------------------------------------------------------------------------------+
1 row(s) fetched.
Elapsed 0.005 seconds.
> select arrow_cast('10:00:00', 'Time32(s)') - interval '1 nanosecond';
+--------------------------------------------------------------------------------------------------------------------------------------+
| arrow_cast(Utf8("10:00:00"),Utf8("Time32(s)")) - IntervalMonthDayNano("IntervalMonthDayNano { months: 0, days: 0, nanoseconds: 1 }") |
+--------------------------------------------------------------------------------------------------------------------------------------+
| 10:00:00 |
+--------------------------------------------------------------------------------------------------------------------------------------+
1 row(s) fetched.
Elapsed 0.007 seconds.There was a problem hiding this comment.
my truncation-toward-zero disagreed with timestamp here. Switched to flooring the interval at ns precision (div_euclid, with the sign applied before the floor), so time(s) - interval '1 nanosecond' now rolls back a full second to 09:59:59, matching your timestamp(s) example. Added an slt case for it. a968da4
Apply the interval at nanosecond precision and floor to the time's unit via div_euclid (with the sign applied before the floor), so `time(s) - interval '1 nanosecond'` rolls back a full second exactly as `timestamp(s) - interval '1 nanosecond'` does, instead of truncating toward zero. Adds an slt case for it. Also take the suggestion to bind the time type and the result type in a single match arm in the coercion.
Jefffrey
left a comment
There was a problem hiding this comment.
should be good once conflicts are resolved and CI is green
|
Thanks @Jefffrey! Rebased on main and resolved the |
Which issue does this PR close?
time + intervalshould wrap within the 24-hour time domain #22265time - intervalshould wrap within the 24-hour time domain #22255Rationale for this change
time + interval(andinterval + time/time - interval) returned anIntervalinstead of atime, so the value was never wrapped within the 24-hour clock:PostgreSQL (and DuckDB) return a
timethat wraps around midnight:The root cause is in type coercion:
time <op> intervalwas coerced by widening thetimeoperand into anInterval, so the addition happened between two intervals and the result kept theIntervaltype.What changes are included in this PR?
Following the existing
Date - Datespecial case (in the same two files),time ± intervalis now handled explicitly:expr-common/src/type_coercion/binary.rs):time + interval,interval + time, andtime - intervalnow coerce to(time, interval)— the interval normalized toMonthDayNano, and the time operand kept at its own unit, which is also the result type.interval - time, which is not meaningful, is left unchanged.physical-expr/src/expressions/binary.rs): a newapply_time_intervaladds/subtracts the interval's sub-day component and wraps the result modulo 24 hours, for all four time units (Time32(Second|Millisecond),Time64(Microsecond|Nanosecond)) and both operand orders. The result keeps the input time's unit, mirroringtimestamp/date + interval(which preserve their unit and apply the interval at that resolution); interval precision finer than the time's unit is truncated, sotime(s) + interval '1 nanosecond'is a no-op, exactly liketimestamp(s) + interval '1 nanosecond'. Only the interval's sub-day portion affects a time-of-day; whole months and days are ignored, matching PostgreSQL (e.g.time '10:00' + interval '1 day 2 hours'=12:00:00).time - time(→Interval) is unchanged.Are these changes tested?
Yes.
datafusion/sqllogictest/test_files/datetime/arith_time_interval.sltwas previously a characterization test that documented the incorrectIntervaloutput; it now asserts the correct wrappedtimeresults — including wrapping past midnight in both directions (22:00 + 3h → 01:00:00,02:00 - 3h → 23:00:00), ignoring whole days, preserving each input time unit (arrow_typeofper unit), and the finer-than-unit interval truncation.Are there any user-facing changes?
Yes —
time ± intervalnow returns atimevalue (wrapped within 24 hours) instead of anInterval, aligning DataFusion with PostgreSQL and DuckDB. The result keeps the input time's unit rather than widening it (see the 55.0.0 upgrade guide).