Skip to content

Fix jax formatting of temporal and decimal columns - #8445

Open
LeSingh1 wants to merge 1 commit into
huggingface:mainfrom
LeSingh1:fix-jax-format-temporal-columns
Open

Fix jax formatting of temporal and decimal columns#8445
LeSingh1 wants to merge 1 commit into
huggingface:mainfrom
LeSingh1:fix-jax-format-temporal-columns

Conversation

@LeSingh1

@LeSingh1 LeSingh1 commented Aug 9, 2026

Copy link
Copy Markdown

Temporal columns are either rejected or silently corrupted by the jax format.

1. Timestamps, dates, times and decimals raise

import datetime
from datasets import Dataset

ds = Dataset.from_dict({"t": [datetime.datetime(2020, 1, 1)]})
ds.with_format("jax")[0]["t"]
TypeError: Error interpreting argument to jax.numpy.asarray as an abstract array.

Same for date32/date64, time32/time64 and decimal128.

2. Durations are silently corrupted

numpy represents durations as timedelta64, and np.timedelta64 is a subtype of np.signedinteger. They therefore take the integer branch of _tensorize and become unit-less integers cast to the jax default int32. The same 5 second duration reads back differently depending only on the column's resolution — and the nanosecond one silently overflows int32:

for unit in ["s", "ms", "us", "ns"]:
    ds = Dataset.from_dict({"d": [datetime.timedelta(seconds=5)]},
                           features=Features({"d": Value(f"duration[{unit}]")}))
    print(unit, ds.with_format("numpy")[:]["d"], ds.with_format("jax")[:]["d"])
s   [5]           [5]
ms  [5000]        [5000]
us  [5000000]     [5000000]
ns  [5000000000]  [705032704]     <-- int32 overflow, 5000000000 mod 2**32

The numpy formatter gets all four right; only jax truncates.

Fix

jax has no dtype for any of these. JaxFormatter._tensorize already returns str, bytes and None unchanged and converts numpy character arrays with .tolist(); this adds temporal and decimal values to that list, before the integer branch that was swallowing timedelta64.

This is the jax counterpart of #8440, which does the same for TorchFormatter. The two are independent files and can be merged in either order; I kept them separate so each is verifiable on its own. I did not touch TFFormatter because I could not run TensorFlow locally to verify it, though it looks like it has the same gap.

Note: .tolist() on timedelta64[ns]/datetime64[ns] yields an int rather than a datetime, because nanosecond resolution does not fit in datetime.timedelta. That is numpy's own behaviour and, unlike the current code, it is lossless — the test asserts 5_000_000_000 explicitly for that case.

Behaviour change worth calling out: duration columns previously produced a jax integer array and now produce the duration values. That path was inconsistent across resolutions and wrong for ns, so I believe returning the values is the right call, but I am happy to adjust if you would rather keep an integer array for durations.

Tests

tests/test_formatting.py::test_jax_formatter_keeps_values_jax_cannot_hold covers timestamp, date32, duration[s], duration[ns], time64 and decimal128 through format_row, format_column and format_batch. All 6 parametrizations fail on main.

tests/test_formatting.py, tests/features/, tests/test_table.py, tests/test_arrow_dataset.py and tests/test_iterable_dataset.py: 1530 passed, 0 failures.

Temporal columns are either rejected or silently corrupted by the jax format.

Timestamps, dates, times and decimals raise:

    ds = Dataset.from_dict({"t": [datetime.datetime(2020, 1, 1)]})
    ds.with_format("jax")[0]["t"]
    TypeError: Error interpreting argument to jax.numpy.asarray as an abstract array.

Durations are worse: numpy represents them as timedelta64, which is a subtype of
np.signedinteger, so they take the integer branch and become unit-less integers
cast to the jax default int32. A 5 second duration therefore reads back as 5,
5000, 5000000 or 705032704 depending only on the resolution of the column, the
last one being a silent int32 overflow of 5000000000 nanoseconds.

jax has no dtype for any of these. The formatter already returns strings and
bytes as they are, so do the same for temporal and decimal values, as in the
torch formatter.

The numpy, pandas and polars formatters were already correct.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant