Skip to content

GH-50464: [C++][Python] Simplify arrow_to_pandas DateOffset handling for nanoseconds/milliseconds#50465

Open
raulcd wants to merge 2 commits into
apache:mainfrom
raulcd:GH-50464
Open

GH-50464: [C++][Python] Simplify arrow_to_pandas DateOffset handling for nanoseconds/milliseconds#50465
raulcd wants to merge 2 commits into
apache:mainfrom
raulcd:GH-50464

Conversation

@raulcd

@raulcd raulcd commented Jul 10, 2026

Copy link
Copy Markdown
Member

Rationale for this change

Pandas 1.4 fixed the following:

Which made us special case microseconds extracting those from our MonthDayNano and manually building microseconds.

DateOffset can be created with the following:

DateOffset(years=1, months=1,
                   days=1, seconds=1, microseconds=1,
                   minutes=1, hours=1, weeks=1, nanoseconds=1)

We currently only roundtrip Month, Day, Microseconds and Nano due to our MonthDayNano data type. We special cased microseconds but we were not special casing years, seconds, minutos, hours or weeks.

As the initial issue that made us special case has been solved upstream we can stop roundtripping microseconds and just return nanoseconds simplifying our codebase.

What changes are included in this PR?

Remove the unnecessary code to retrieve microseconds from nanoseconds.
Update test as microseconds won't be returned, only nanoseconds.

Are these changes tested?

Yes via CI

Are there any user-facing changes?

Yes, this is a change of the UX.

@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #50464 has been automatically assigned in GitHub to PR creator.

…Year, seconds, microseconds, minutes, hours, weeks
@raulcd

raulcd commented Jul 10, 2026

Copy link
Copy Markdown
Member Author

@jorisvandenbossche would there be any reason to keep roundtriping microseconds on DateOffset(months=1, days=1, microseconds=1, nanoseconds=1) when we don't do it for years, seconds, minutes, hours, weeks?
We can simplify a little here with this PR.

@raulcd raulcd marked this pull request as ready for review July 10, 2026 13:26
@raulcd raulcd requested review from AlenkaF, pitrou and rok as code owners July 10, 2026 13:26
Copilot AI review requested due to automatic review settings July 10, 2026 13:26

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR simplifies the Pandas DateOffset conversion path for month_day_nano_interval in arrow_to_pandas by no longer splitting nanoseconds into microseconds + remainder nanoseconds, and updates Python tests to expect nanoseconds-only roundtripping.

Changes:

  • Simplify MonthDayNanoIntervalType → Pandas DateOffset conversion to pass interval.nanoseconds directly.
  • Update interval/dateoffset-related tests to assert nanoseconds-only behavior.
  • Adjust expected DateOffset construction in tests to reflect the new output shape.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File Description
python/pyarrow/src/arrow/python/arrow_to_pandas.cc Removes microseconds/remainder splitting and passes total nanoseconds into DateOffset.
python/pyarrow/tests/test_pandas.py Updates roundtrip expectations for DateOffset values produced from month-day-nano intervals.
python/pyarrow/tests/test_array.py Updates DateOffset expectations in interval array tests to use nanoseconds-only representation.

Comment on lines 1296 to +1299
PyDict_SetItemString(kwargs.obj(), "months", PyLong_FromLong(interval.months));
PyDict_SetItemString(kwargs.obj(), "days", PyLong_FromLong(interval.days));
PyDict_SetItemString(kwargs.obj(), "microseconds",
PyLong_FromLongLong(microseconds));
PyDict_SetItemString(kwargs.obj(), "nanoseconds", PyLong_FromLongLong(nanoseconds));
PyDict_SetItemString(kwargs.obj(), "nanoseconds",
PyLong_FromLongLong(interval.nanoseconds));
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants