Fix jax formatting of temporal and decimal columns - #8445
Open
LeSingh1 wants to merge 1 commit into
Open
Conversation
Temporal columns are either rejected or silently corrupted by the jax format.
Timestamps, dates, times and decimals raise:
ds = Dataset.from_dict({"t": [datetime.datetime(2020, 1, 1)]})
ds.with_format("jax")[0]["t"]
TypeError: Error interpreting argument to jax.numpy.asarray as an abstract array.
Durations are worse: numpy represents them as timedelta64, which is a subtype of
np.signedinteger, so they take the integer branch and become unit-less integers
cast to the jax default int32. A 5 second duration therefore reads back as 5,
5000, 5000000 or 705032704 depending only on the resolution of the column, the
last one being a silent int32 overflow of 5000000000 nanoseconds.
jax has no dtype for any of these. The formatter already returns strings and
bytes as they are, so do the same for temporal and decimal values, as in the
torch formatter.
The numpy, pandas and polars formatters were already correct.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Temporal columns are either rejected or silently corrupted by the jax format.
1. Timestamps, dates, times and decimals raise
Same for
date32/date64,time32/time64anddecimal128.2. Durations are silently corrupted
numpy represents durations as
timedelta64, andnp.timedelta64is a subtype ofnp.signedinteger. They therefore take the integer branch of_tensorizeand become unit-less integers cast to the jax defaultint32. The same 5 second duration reads back differently depending only on the column's resolution — and the nanosecond one silently overflowsint32:The numpy formatter gets all four right; only jax truncates.
Fix
jax has no dtype for any of these.
JaxFormatter._tensorizealready returnsstr,bytesandNoneunchanged and converts numpy character arrays with.tolist(); this adds temporal and decimal values to that list, before the integer branch that was swallowingtimedelta64.This is the jax counterpart of #8440, which does the same for
TorchFormatter. The two are independent files and can be merged in either order; I kept them separate so each is verifiable on its own. I did not touchTFFormatterbecause I could not run TensorFlow locally to verify it, though it looks like it has the same gap.Note:
.tolist()ontimedelta64[ns]/datetime64[ns]yields anintrather than adatetime, because nanosecond resolution does not fit indatetime.timedelta. That is numpy's own behaviour and, unlike the current code, it is lossless — the test asserts5_000_000_000explicitly for that case.Behaviour change worth calling out:
durationcolumns previously produced a jax integer array and now produce the duration values. That path was inconsistent across resolutions and wrong forns, so I believe returning the values is the right call, but I am happy to adjust if you would rather keep an integer array for durations.Tests
tests/test_formatting.py::test_jax_formatter_keeps_values_jax_cannot_holdcoverstimestamp,date32,duration[s],duration[ns],time64anddecimal128throughformat_row,format_columnandformat_batch. All 6 parametrizations fail onmain.tests/test_formatting.py,tests/features/,tests/test_table.py,tests/test_arrow_dataset.pyandtests/test_iterable_dataset.py: 1530 passed, 0 failures.