[SPARK-59284][SQL] Read INT64 TIMESTAMP(MICROS) columns as nanosecond timestamps - #58550
[SPARK-59284][SQL] Read INT64 TIMESTAMP(MICROS) columns as nanosecond timestamps#58550stevomitric wants to merge 3 commits into
Conversation
… timestamps Enables reading a Parquet INT64 TIMESTAMP(MICROS) column under a nanosecond timestamp type (TimestampLTZNanosType / TimestampNTZNanosType) -- the read side of widening a microsecond-precision timestamp column (TIMESTAMP(6)) to nanosecond precision without rewriting existing files. Each micros value is promoted to the internal (epochMicros, nanosWithinMicro = 0) representation. Because that representation is microsecond-based, the promotion is exact and range-complete: there is no *1000 encode to epoch-nanoseconds and hence no int64 overflow cliff (unlike the on-disk nanos encoding, which is bounded to ~1677..2262). - Row-based read: TimestampNanosParquetOps overrides the extended newConverter (which supplies the datetime rebase spec) to accept a TIMESTAMP(MICROS) column. LTZ applies the Julian->Gregorian rebase like the microsecond TimestampType read path; NTZ needs none. - Vectorized read: ParquetVectorUpdaterFactory dispatches a nanos-type-over-micros column to new MicrosAsTimestampNanos(Rebase)Updater updaters, mirroring LongWithRebaseUpdater. - Filter pushdown: ParquetFilters is built from the file schema, so it only pushes down a micros/millis timestamp bound that lies exactly on the microsecond grid; a sub-microsecond bound (which can only arise from a nanos read) falls back to a full scan so row-group pruning never drops matching rows. Scope: INT64 TIMESTAMP(MICROS) sources only. INT96 and TIMESTAMP_MILLIS sources remain unsupported and fail loudly. Tests: TimestampNanosParquetOpsSuite (row-based decode incl. a beyond-2262 range value), ParquetTypeWideningSuite (end-to-end micros->nanos across vectorized and row-based readers, dictionary on/off, and CORRECTED/LEGACY rebase, plus INT96 and TIMESTAMP_MILLIS fail-loud), and ParquetFilterSuite (sub-microsecond pushdown guard). Co-authored-by: Isaac <no-reply@databricks.com>
…comment Addresses code-review feedback on the micros->nanos read support: - Cross-family guard: when reading an INT64 TIMESTAMP(MICROS) column as a nanosecond type, match the file's isAdjustedToUTC to the requested LTZ/NTZ family so a deliberately cross-family explicit read schema (e.g. an NTZ file requested as an LTZ instant) fails loudly instead of reinterpreting the values. Row-based via isMicrosTimestamp(parquetType, expectedAdjustedToUTC); vectorized via a new isTimestampTypeMatched(unit, isAdjustedToUTC) overload. Widening only ever pairs same-family types, so this just rejects a mismatched .schema(...). - Fix the stale getVectorUpdater comment: a TIMESTAMP(MICROS) column is now promoted to nanos by the factory rather than failing loudly. Tests: cross-family rejection (TimestampNanosParquetOpsSuite) and fail-loud cross-family reads across both readers (ParquetTypeWideningSuite). Co-authored-by: Isaac <no-reply@databricks.com>
uros-b
left a comment
uros-b
left a comment
There was a problem hiding this comment.
Looks like this broke a test in ParquetTimestampNanosSuite and CI is red because of that, @stevomitric PTAL and fix accordingly.
This PR intentionally makes reading a TIMESTAMP(MICROS) column under a nanos type succeed, but leaves the pre-existing negative test "SPARK-57102: requesting a nanos type over a non-NANOS Parquet column fails clearly" unchanged. That test writes exactly a TIMESTAMP_NTZ micros column and reads it as TIMESTAMP_NTZ(7) asserting a PARQUET_CONVERSION_FAILURE; with this change the read now succeeds (NTZ-micros -> NTZ-nanos via the new microsAsNanosConverter), so the intercept[SparkException] fails with "Expected exception ... but no exception was thrown".
The test and its now-stale explanatory comment ("the guard requires a NANOS annotation") must be updated to exercise a genuinely-still-unsupported encoding (TIMESTAMP_MILLIS or INT96, both of which still throw here) so the "fails clearly" negative-path coverage is preserved, and CI must go green before merge. The break also indicates no sweep was done for existing tests asserting the old behavior.
…ud test The "requesting a nanos type over a non-NANOS Parquet column fails clearly" test wrote an INT64 TIMESTAMP(MICROS) column and asserted that reading it under a nanos schema throws PARQUET_CONVERSION_FAILURE. That read is now supported (this PR reads an INT64 TIMESTAMP(MICROS) column as nanos -- the read side of widening TIMESTAMP(6) to nanosecond precision), so the assertion no longer holds. Repurpose the test to a still-unsupported non-NANOS encoding (TIMESTAMP(MILLIS)), which is neither NANOS nor same-family MICROS and so still falls through to the generic PARQUET_CONVERSION_FAILURE error, preserving the test's intent. Co-authored-by: Isaac <no-reply@databricks.com>
stevomitric
left a comment
stevomitric
left a comment
There was a problem hiding this comment.
Looks like this broke a test in ParquetTimestampNanosSuite and CI is red because of that, @stevomitric PTAL and fix accordingly.
Thanks @uros-b for noticing - should be fixed now.
2 participants
What changes were proposed in this pull request?
Let a nanosecond timestamp type read a Parquet INT64 TIMESTAMP(MICROS) column, promoting each micros value to the internal (epochMicros, nanosWithinMicro = 0) representation. Three touch points:
Scope: INT64 TIMESTAMP(MICROS) only; INT96 and TIMESTAMP_MILLIS sources remain unsupported.
Why are the changes needed?
It's the read side of widening a microsecond timestamp column (TIMESTAMP(6)) to nanosecond precision without rewriting existing files. Today the nanos reader fails loudly on any non-TIMESTAMP(NANOS) column, so old micros files would be unreadable after such a widening. The promotion is exact and range-complete - micros to (micros, 0) involves no ×1000 encode, so it avoids the INT64 epoch-nanos overflow range (~1677–2262).
Does this PR introduce any user-facing change?
Yes, behind the preview flag spark.sql.timestampNanosTypes.enabled. Previously, reading a TIMESTAMP(MICROS) Parquet column under a nanosecond timestamp type threw (SchemaColumnConvertNotSupportedException / PARQUET_CONVERSION_FAILURE); now it reads the values correctly.
How was this patch tested?
Added TimestampNanosParquetOpsSuite and updated ParquetTypeWideningSuite ParquetFilterSuite.
Was this patch authored or co-authored using generative AI tooling?
Co-authored-by: Claude Opus 4.8