Thanks to visit codestin.com
Credit goes to github.com

Skip to content

comparing time series with index of different units #63466

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Antriksh006 wants to merge 10 commits into
base: main
Choose a base branch
from
+47 −2
Open

comparing time series with index of different units #63466

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
87c158e
comparing time series with index of different units
Antriksh006 Dec 22, 2025
9d21e7c
.
Antriksh006 Dec 23, 2025
cc2668b
.
Antriksh006 Dec 23, 2025
fd19e02
fix Datetimelike.equals unit mismatch and typing errors
Antriksh006 Dec 23, 2025
7566471
.
Antriksh006 Dec 23, 2025
45c435f
Merge branch 'main' into pandas-antriksh-changes
Antriksh006 Dec 24, 2025
cb1e31f
added some tests
Antriksh006 Dec 24, 2025
492614b
Merge remote-tracking branch 'refs/remotes/origin/pandas-antriksh-cha…
Antriksh006 Dec 24, 2025
7459fd0
.
Antriksh006 Dec 24, 2025
0893e86
added comment removed accidently and pushed other dtype kind in iufc …
Antriksh006 Dec 27, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 20 additions & 1 deletion pandas/core/indexes/datetimelike.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -245,8 +245,27 @@ def equals(self, other: Any) -> bool:

if not isinstance(other, Index):
return False
elif other.dtype.kind in "iufc":

if hasattr(other, "dtype") and other.dtype.kind in "iufc":
return False

if len(self) != len(other):
return False

self_unit = getattr(self, "unit", None)
other_unit = getattr(other, "unit", None)

if self_unit is not None and other_unit is not None and self_unit != other_unit:
if getattr(self.dtype, "tz", None) == getattr(other.dtype, "tz", None):
try:
other_values = other._values
if hasattr(other_values, "as_unit") and hasattr(
self._values, "equals"
):
return self._values.equals(other_values.as_unit(self_unit))
Comment on lines +260 to +265
Copy link
Member

@rhshadrach rhshadrach Dec 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In what cases does this raise?

Copy link
Contributor Author

@Antriksh006 Antriksh006 Dec 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

an example could be if we put two date_ranges to equal(), one with frequency "D" and other being "M", then if this scenario is hit, we safely get False as answer rather then raising a ValueError, due to frequency mismatch.

basically first check if other_values has units we must compare, second check if self can have these compared to it, then simply check and return if they are equal or not

Copy link
Member

@rhshadrach rhshadrach Dec 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That makes sense, but why is this code itself in a try-except? This is why I am asking in what case does this code itself raise.

Copy link
Contributor Author

@Antriksh006 Antriksh006 Dec 28, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

during my testing i saw some scenarios where i see the as_unit(self_unit) raising OutOfBoundsDatetime error, so i just put that in a try except to catch error in the return statement, i might as well implement it through a if check if you prefer that.

I am new to pandas contribution, so i didnt know we dont like try-excepts here. You can let me know if i should remove it and i will commit that to this pr.

This is a solution i have came up with, it passes the tests on my system locally. should i push this instead?

        if self_unit is not None and other_unit is not None and self_unit != other_unit:
            if getattr(self.dtype, "tz", None) == getattr(other.dtype, "tz", None):
                other_values = other._values

                if hasattr(other_values, "as_unit") and hasattr(self._values, "equals"):
                    return self._values.equals(other_values.as_unit(self_unit))

except (ValueError, TypeError, AttributeError):
return False

elif not isinstance(other, type(self)):
should_try = False
inferable = self._data._infer_matches
Expand Down
26 changes: 26 additions & 0 deletions pandas/tests/indexes/datetimelike_/test_equals.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -136,6 +136,19 @@ def test_equals2(self):
assert not idx2.equals(oob2)
assert not idx3.equals(oob2)

def test_equals_different_units(self):
# GH#63459
idx1 = date_range("2013-01-01", periods=5).as_unit("ns")
idx2 = idx1.as_unit("us")

assert idx1.equals(idx2) is True
assert idx2.equals(idx1) is True

# Test with mismatching values but same units to ensure we don't
# get false positives
idx3 = date_range("2013-01-02", periods=5).as_unit("us")
assert not idx1.equals(idx3)

@pytest.mark.parametrize("freq", ["B", "C"])
def test_not_equals_bday(self, freq):
rng = date_range("2009-01-01", "2010-01-01", freq=freq)
Expand Down Expand Up @@ -183,3 +196,16 @@ def test_equals2(self):
assert (oob3 == oob).all()
assert not idx.equals(oob3)
assert not idx2.equals(oob3)

def test_equals_different_units(self):
# GH#63459
idx1 = timedelta_range("1 day", periods=10).as_unit("ns")
idx2 = idx1.as_unit("us")

assert idx1.equals(idx2) is True
assert idx2.equals(idx1) is True

# Ensure that NaT equality still holds across units
idx3 = TimedeltaIndex(["1 days", "NaT"]).as_unit("ns")
idx4 = TimedeltaIndex(["1 days", "NaT"]).as_unit("us")
assert idx3.equals(idx4) is True
2 changes: 1 addition & 1 deletion pandas/tests/indexes/datetimes/test_setops.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -246,7 +246,7 @@ def test_intersection_same_timezone_different_units(self):
# Test intersection
result = idx1.intersection(idx2)
expected = date_range("2000-01-01", periods=3, tz=tz).as_unit("ns")
tm.assert_index_equal(result, expected)
tm.assert_index_equal(result, expected, exact=False)
Copy link
Member

@rhshadrach rhshadrach Dec 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does this need to change?

Copy link
Contributor Author

@Antriksh006 Antriksh006 Dec 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It relaxes the dtype strictness. In the context of datetimes, it stops caring whether the storage is in nanoseconds or microseconds, as long as the dates themselves represent the same point in history.

Copy link
Member

@rhshadrach rhshadrach Dec 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Prior to this PR the result and expected were considered equal with exact=True; and even though result and expected here are not changing, they no longer are considered equal with exact=True?

Copy link
Contributor Author

@Antriksh006 Antriksh006 Dec 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test used to pass because the code was implicitly forcing everything into one format (nanoseconds). Now that the code correctly preserves the original units, the test fails because it expects a specific format that no longer matches the output. I changed the test to exact=False so it focuses on whether the dates are correct, rather than worrying about the storage format.

Copy link
Contributor Author

@Antriksh006 Antriksh006 Dec 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rhshadrach


def test_symmetric_difference_same_timezone_different_units(self):
# GH 60080 - fix timezone being changed to UTC when units differ
Expand Down
Loading