Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Improve pandas/xarray/... conversion #22560

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Apr 21, 2022
Merged

Improve pandas/xarray/... conversion #22560

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
7b51044
Improve pandas and xarray conversion
Feb 24, 2022
56af810
CI: add xarray to extra dependencies
tacaswell Apr 13, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions lib/matplotlib/axes/_axes.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7907,8 +7907,8 @@ def violinplot(self, dataset, positions=None, vert=True, widths=0.5,
"""

def _kde_method(X, coords):
if hasattr(X, 'values'): # support pandas.Series
X = X.values
# Unpack in case of e.g. Pandas or xarray object
X = cbook._unpack_to_numpy(X)
# fallback gracefully if the vector contains only one value
if np.all(X[0] == X):
return (X[0] == coords).astype(float)
Expand Down
33 changes: 21 additions & 12 deletions lib/matplotlib/cbook/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1311,9 +1311,8 @@ def _to_unmasked_float_array(x):

def _check_1d(x):
"""Convert scalars to 1D arrays; pass-through arrays as is."""
if hasattr(x, 'to_numpy'):
# if we are given an object that creates a numpy, we should use it...
x = x.to_numpy()
# Unpack in case of e.g. Pandas or xarray object
x = _unpack_to_numpy(x)
if not hasattr(x, 'shape') or len(x.shape) < 1:
return np.atleast_1d(x)
else:
Expand All @@ -1332,15 +1331,8 @@ def _reshape_2D(X, name):
*name* is used to generate the error message for invalid inputs.
"""

# unpack if we have a values or to_numpy method.
try:
X = X.to_numpy()
except AttributeError:
try:
if isinstance(X.values, np.ndarray):
X = X.values
except AttributeError:
pass
# Unpack in case of e.g. Pandas or xarray object
X = _unpack_to_numpy(X)

# Iterate over columns for ndarrays.
if isinstance(X, np.ndarray):
Expand Down Expand Up @@ -2231,3 +2223,20 @@ def _picklable_class_constructor(mixin_class, fmt, attr_name, base_class):
factory = _make_class_factory(mixin_class, fmt, attr_name)
cls = factory(base_class)
return cls.__new__(cls)


def _unpack_to_numpy(x):
"""Internal helper to extract data from e.g. pandas and xarray objects."""
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please document what we intend to support, i.e. everything with .to_numpy() or .values, and what types we expect to catch with it, e.g. values-> older pandas dataframes(?).

if isinstance(x, np.ndarray):
# If numpy, return directly
return x
if hasattr(x, 'to_numpy'):
# Assume that any function to_numpy() do actually return a numpy array
return x.to_numpy()
if hasattr(x, 'values'):
xtmp = x.values
# For example a dict has a 'values' attribute, but it is not a property
# so in this case we do not want to return a function
if isinstance(xtmp, np.ndarray):
return xtmp
return x
5 changes: 2 additions & 3 deletions lib/matplotlib/dates.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -437,9 +437,8 @@ def date2num(d):
The Gregorian calendar is assumed; this is not universal practice.
For details see the module docstring.
"""
if hasattr(d, "values"):
# this unpacks pandas series or dataframes...
d = d.values
# Unpack in case of e.g. Pandas or xarray object
d = cbook._unpack_to_numpy(d)

# make an iterable, but save state to unpack later:
iterable = np.iterable(d)
Expand Down
7 changes: 7 additions & 0 deletions lib/matplotlib/testing/conftest.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,3 +125,10 @@ def pd():
except ImportError:
pass
return pd


@pytest.fixture
def xr():
"""Fixture to import xarray."""
xr = pytest.importorskip('xarray')
return xr
2 changes: 1 addition & 1 deletion lib/matplotlib/tests/conftest.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
from matplotlib.testing.conftest import (mpl_test_settings,
pytest_configure, pytest_unconfigure,
pd)
pd, xr)
25 changes: 24 additions & 1 deletion lib/matplotlib/tests/test_cbook.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -680,14 +680,37 @@ def test_reshape2d_pandas(pd):
for x, xnew in zip(X.T, Xnew):
np.testing.assert_array_equal(x, xnew)


def test_reshape2d_xarray(xr):
# separate to allow the rest of the tests to run if no xarray...
X = np.arange(30).reshape(10, 3)
x = pd.DataFrame(X, columns=["a", "b", "c"])
x = xr.DataArray(X, dims=["x", "y"])
Xnew = cbook._reshape_2D(x, 'x')
# Need to check each row because _reshape_2D returns a list of arrays:
for x, xnew in zip(X.T, Xnew):
np.testing.assert_array_equal(x, xnew)


def test_index_of_pandas(pd):
# separate to allow the rest of the tests to run if no pandas...
X = np.arange(30).reshape(10, 3)
x = pd.DataFrame(X, columns=["a", "b", "c"])
Idx, Xnew = cbook.index_of(x)
np.testing.assert_array_equal(X, Xnew)
IdxRef = np.arange(10)
np.testing.assert_array_equal(Idx, IdxRef)


def test_index_of_xarray(xr):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does xarray get us more coverage here? They have a to_numpy() method the same as pandas I believe.
https://xarray.pydata.org/en/stable/generated/xarray.DataArray.to_numpy.html

So, it seems like a pretty heavy dependency to add for just this one test...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not so much for coverage as for actually testing using data of specified formats. With the discussion about which formats we support, it makes sense to test them as well. Right now some of these are tested in the plots, but it can possibly make sense to simply test them here as these are the core function used to get data that can be plotted.

If we claim (which we actually don't, maybe we should?) that we can plot xarray, we should probably test it as well. And other types that we may want to claim to support. Or maybe fork off a specific dependency test that is not executed on all platforms/version, including pandas (which is 11.7 MB, xarray is 870 kB).

(There is another xarray-test above, so two.)

I can of course remove them, but I think we should discuss if we want to support more formats than pandas and numpy (and Python list/tuple), and, if so, have explicit tests for them.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree, we should probably discuss what we want to support/test. To me, this doesn't seem to add a whole lot of value for adding a new dependency.

There was also a discussion around removing Scipy as a dependency in the docs: #22120

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed the dependencies but kept the tests. Hence, they will run if xarray is available.

I also opened #22645 for discussions (probably should be discussed at a dev-call as well).

# separate to allow the rest of the tests to run if no xarray...
X = np.arange(30).reshape(10, 3)
x = xr.DataArray(X, dims=["x", "y"])
Idx, Xnew = cbook.index_of(x)
np.testing.assert_array_equal(X, Xnew)
IdxRef = np.arange(10)
np.testing.assert_array_equal(Idx, IdxRef)


def test_contiguous_regions():
a, b, c = 3, 4, 5
# Starts and ends with True
Expand Down
5 changes: 3 additions & 2 deletions lib/matplotlib/units.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -180,8 +180,9 @@ class Registry(dict):

def get_converter(self, x):
"""Get the converter interface instance for *x*, or None."""
if hasattr(x, "values"):
x = x.values # Unpack pandas Series and DataFrames.
# Unpack in case of e.g. Pandas or xarray object
x = cbook._unpack_to_numpy(x)

if isinstance(x, np.ndarray):
# In case x in a masked array, access the underlying data (only its
# type matters). If x is a regular ndarray, getdata() just returns
Expand Down
1 change: 1 addition & 0 deletions requirements/testing/extra.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,3 +7,4 @@ pandas!=0.25.0
pikepdf
pytz
pywin32; sys.platform == 'win32'
xarray