✔️ Linting Passed

All linting checks passed. Your pull request is in excellent shape! ☀️

_{Generated for commit: 87a634d. Link to the linter CI: here}

@lesteve @adrinjalali I did fix the error resulting from the mean_absolute_percentage_error function in the example plot_time_series_lagged_features.py and it looks like the tests pass. However if you check the output of ./build_tools/circle/build_doc.sh in the doc-min-dependencies CI, it seems that both mean_squared_error and mean_absolute_error are throwing the same error but probably they are failing silently within the example so the error is not shown in the output.

Indeed no idea what is happening but https://app.circleci.com/pipelines/github/scikit-learn/scikit-learn/59983/workflows/84052e1d-6041-4a3d-89cf-fb076b09ba99/jobs/280390?invite=true#step-105-86409_95 is showing a warning about the scoring failing with a similar stack-trace as before. So I am guessing that the underlying issue is still there?

/home/circleci/project/sklearn/model_selection/_validation.py:997: UserWarning: Scoring failed. The score on this train-test partition for these parameters will be set to nan. Details: 
Traceback (most recent call last):
  File "/home/circleci/project/sklearn/metrics/_scorer.py", line 139, in __call__
    score = scorer._score(
  File "/home/circleci/project/sklearn/metrics/_scorer.py", line 376, in _score
    return self._sign * self._score_func(y_true, y_pred, **scoring_kwargs)
  File "/home/circleci/project/sklearn/utils/_param_validation.py", line 213, in wrapper
    return func(*args, **kwargs)
  File "/home/circleci/project/sklearn/metrics/_regression.py", line 596, in root_mean_squared_error
    mean_squared_error(
  File "/home/circleci/project/sklearn/utils/_param_validation.py", line 186, in wrapper
    return func(*args, **kwargs)
  File "/home/circleci/project/sklearn/metrics/_regression.py", line 516, in mean_squared_error
    dtype = _find_matching_floating_dtype(y_true, y_pred, xp=xp)
  File "/home/circleci/project/sklearn/utils/_array_api.py", line 681, in _find_matching_floating_dtype
    floating_dtypes = [
  File "/home/circleci/project/sklearn/utils/_array_api.py", line 682, in <listcomp>
    a.dtype for a in dtyped_arrays if xp.isdtype(a.dtype, "real floating")
  File "/home/circleci/project/sklearn/utils/_array_api.py", line 442, in isdtype
    return isdtype(dtype, kind, xp=self)
  File "/home/circleci/project/sklearn/utils/_array_api.py", line 198, in isdtype
    return _isdtype_single(dtype, kind, xp=xp)
  File "/home/circleci/project/sklearn/utils/_array_api.py", line 215, in _isdtype_single
    return dtype in supported_float_dtypes(xp)
TypeError: Cannot interpret 'Int64' as a data type

Yes and the error is now showing in functions that were updated before. Do we want to keep the numpy version or can we update it? Otherwise we will need to add some kind of a check in _find_matching_floating_dtype.

Do we want to keep the numpy version or can we update it?

The way I understand the "min" in doc-min-dependencies is that it uses the minimum version for all dependencies. So I think we can't update Numpy. Even if we do update it, I think we should have a good user experience when they use array API support and an older version of Numpy (maybe that version isn't as old as the oldest scikit-learn supports, but older than Numpy 2 I'd say).

I think then we would need to handle this special case in _find_matching_floating_dtype or in _isdtype_single where it actually occurs.

Do you have a small code snippet that reproduces the problem?

I am trying to understand why we are calling array API code from an example. Naively I was expecting that the examples don't use the array API and hence we shouldn't touch any of the related code.

Do you have a small code snippet that reproduces the problem?

I am trying to understand why we are calling array API code from an example. Naively I was expecting that the examples don't use the array API and hence we shouldn't touch any of the related code.

I think to reproduce this we would need to downgrade a number of libraries like numpy, polars which are used in this example. So as @lesteve mentioned follow the instructions in quick doc to set up an environment. After that we can simply run the example. I don't have a smaller snippet because I haven't been able to reproduce this even with the full example on my mac 😄

Here is a snippet that reproduces for me see versions in doc-min-dependencies environment.yml amongst other things polars 0.20.23, numpy 1.19.5.

This one reproduces the issue in main but not in this PR:

import polars as pl

s = pl.Series([1, 2, 3])

from sklearn.metrics import mean_absolute_percentage_error

mean_absolute_percentage_error(s, s)

I need to look more about reproducing the warnings, it looks like this is happening in joblib.Parallel for some reason ...

Thanks for the example. I also struggle to get it to reproduce, but mostly because of difficulty with getting an environment setup with the right versions (I suspect).

Looking around the code base for uses of _find_matching_floating_dtype I found

Basically the series dtype Int64 seems to be throwing an error when we check this line

I also struggle to get it to reproduce, but mostly because of difficulty with getting an environment setup with the right versions (I suspect).

Just curious, I guess the difficulty is that for arm64 macOS it's hard to get older versions, because they have not been built in conda-forge?

Looking a bit further. I think this PR actually fixes the issue for mean_absolute_percentage_error and the warnings comes from the other metrics that are used in the example and we did not notice indeed ...

I think a test should be added in sklearn/metrics/tests/test_common.py to check that the metric called on two polar series does what we expect. This would be tested for our min dependencies. I haven't figured out the polars and numpy combination that makes this issue happen.

As asked by Tim, r2_score has the same issue with polars Series.

I tweaked the example (see tweaked version) and it seems that other metrics are still problematic (root_mean_square_error and mean_absolute_error). Here is the value of cv_results after running the example:

{'fit_time': array([1.8457191 , 1.91570544, 1.12068701]),
 'score_time': array([0.01725125, 0.02016592, 0.01188159]),
 'test_MAPE': array([0.44300752, 0.27772182, 0.3697178 ]),
 'test_RMSE': array([nan, nan, nan]),
 'test_MAE': array([nan, nan, nan]),
 'test_pinball_loss_05': array([15.90543294, 18.45444449, 18.74125985]),
 'test_pinball_loss_50': array([19.07197374, 21.10759731, 18.4528097 ]),
 'test_pinball_loss_95': array([22.23851453, 23.76075013, 18.16435955])}

@lesteve Thank you for reporting. I think the fix in this PR might not be suitable generally. I think we need to handle the error occurring within the functions that cause this error. We cannot actually replace _find_matching_floating_dtype in all the places, as this function is used to find the appropriate float dtype even for cases that might not involve the array api.

I think in a sense this is similar to #29452, as in, while array api dispatch is not enabled, we see side effects. I think we should make sure there is no side effects when dispatch is not enabled.

I haven't figured out the polars and numpy combination that makes this issue happen.

For completeness it seems like numpy 1.20.3 is the latest version that has the issue, numpy>= 1.21.0 works fine. I am guessing that for scikit-learn 1.6 we will be able to bump numpy enough that maybe we will avoid this issue?

Still I share Adrin's concern about the fact that the coverage needs to be increased because it can break in mysterious ways ...

I have removed the warning because it seems unnecessary.

For completeness still, a snippet that shows the issue with numpy 1.20.3 and not 1.21.0 (polars version does not matter):

import numpy as np
import polars as pl

s = pl.Series([1, 2, 3])

numpy_float64_dtype = np.dtype(np.float64)
# False
numpy_float64_dtype == pl.Int64
# False
s.dtype == numpy_float64_dtype
# TypeError: Cannot interpret 'Int64' as a data type
numpy_float64_dtype == s.dtype

I added a small common test to check metric on pandas and polars series. I guess there is probably some room for improvement. This fails locally on main with numpy 1.20.3 and passes on this PR.

The most wonderful thing about it (this is sarcasm just to be clear 😉) is that this is not going to make the CI red because it seems like we don't even have a build for numpy 1.19 or numpy 1.20 ... and even if we had one polars or pandas would not be installed anyway so 😓 ...

I opened #29502 to actually use our minimum supported version in our min-dependencies-build and add pandas and polars into it.

I think we might not be able to fix the codecov issue considering that the TypeError is not raised in the general case.

I am going to approve this but it would be great if someone looked at the test I added because right now, it looks quite minimal.

I think the tests look fine since we are just checking that polars and pandas work without any errors.

@adrinjalali @betatim Could you have a look at this PR, so that we can resolve the CI failures occurring?

I enabled auto-merge, let's make doc-min-dependencies greateen again 💚!