[SPARK-59467][PYTHON] Normalize ArrowDtype timestamps to the session timezone in pandas conversion - #58765
Open
fangchenli wants to merge 6 commits into
Open
[SPARK-59467][PYTHON] Normalize ArrowDtype timestamps to the session timezone in pandas conversion#58765fangchenli wants to merge 6 commits into
fangchenli wants to merge 6 commits into
Conversation
…andas conversion _check_series_convert_timestamps_internal only handled numpy datetime64 and DatetimeTZDtype, so a timezone-naive pd.ArrowDtype timestamp column fell through unnormalized and was interpreted as UTC instead of the session timezone. This affected createDataFrame from pandas and pandas UDF output in non-UTC sessions. Add an ArrowDtype branch that localizes with pc.assume_timezone using ambiguous="latest", matching the standard-time choice of the numpy branch, and falls back to the numpy branch for timezone=None and for zone ids pyarrow cannot parse (e.g. "UTC+01:00"). Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01MNnkeZfwvDybyqry1C3rj2
- Move the migration note to pyspark_upgrade.rst, where pandas conversion notes live, and qualify it for Spark Connect createDataFrame without a schema, which does not infer ArrowDtype columns as TimestampType. - Collapse the unreachable timezone=None sub-branch into the ArrowInvalid fallback and document that nonexistent times take the fallback too. - Route the createDataFrame-level test through create_arrow_array_from_pandas, the path createDataFrame actually uses, and assert timezone=None on values rather than an environment-dependent dtype string. - Fix import ordering flagged by ruff I001; drop a stale nested-timestamp TODO. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01MNnkeZfwvDybyqry1C3rj2
Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01MNnkeZfwvDybyqry1C3rj2
Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01MNnkeZfwvDybyqry1C3rj2
Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01MNnkeZfwvDybyqry1C3rj2
Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01MNnkeZfwvDybyqry1C3rj2
Member
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Add a pd.ArrowDtype branch to _check_series_convert_timestamps_internal so timezone-naive ArrowDtype timestamp columns are localized to the session timezone like numpy datetime64 columns. Localization uses pyarrow.compute.assume_timezone with ambiguous="latest", matching the standard-time choice of the numpy branch, and falls back to the numpy branch for zone ids pyarrow cannot parse (e.g. UTC+01:00). Timezone-aware ArrowDtype columns are converted to UTC and passed through.
This is a precursor to #54518.
Why are the changes needed?
The function only recognized numpy datetime64 and pd.DatetimeTZDtype. A naive timestamp[us][pyarrow] column fell through unchanged and was interpreted as UTC, so in a non-UTC session values were shifted by the session's UTC offset in createDataFrame from pandas and in pandas UDF results. This contradicts the documented behavior of spark.sql.session.timeZone.
Does this PR introduce any user-facing change?
Yes. Timezone-naive ArrowDtype timestamp columns are now interpreted in the session timezone instead of UTC. To keep the previous interpretation, make the column timezone-aware in UTC. A migration guide entry is added.
How was this patch tested?
Unittests added.
Was this patch authored or co-authored using generative AI tooling?
Yes, co-authored with Claude Code Opus 5.