I changed the order of the params in the docs to match the order of reading these into the function, like in all the other places.

_average_binary_score is also used in roc_auc_score. So it's probably best to first discuss how to properly add array API support to it, or wait until this PR is merged before creating a PR for roc_auc_score.

This case handling needs to be added, because average_precision_score handels multiclass inputs by binarising it. As a consequence, y_score needs to be of shape (n_samples, n_classes). I have made a PR with a fix here: #32912.

This is not needed anymore, since average_weight is converted to an array earlier.

This (hopefully) fixes an error occurring on the CI (link to output):

E TypeError: sum() received an invalid combination of arguments - got (out=NoneType, axis=int, ), but expected one of:
E * (*, torch.dtype dtype = None)
E  didn't match because some of the keywords were incorrect: out, axis
E  * (tuple of ints dim, bool keepdim = False, *, torch.dtype dtype = None)
E  * (tuple of names dim, bool keepdim = False, *, torch.dtype dtype = None)

@lesteve and I investigated this together. It seems that numpy.sum() when passed a torch tensor, internally tries to dispatch to torch, but that fails because torch.sum() doesn't have the out argument. The same happens with np.repeat() which has an axis argument, but torch.repeat() hasn't (which had raised in my setup locally).

Yes, I ran into the same issue when I was working on roc_auc_score.

Instead of converting everything to NumPy ahead of time, we should inspect the namespace and device of y_score only. Indeed, we want to allow y_true to have an object dtype for string class labels (hence a NumPy array) even when the predictions are a non-NumPy namespace with a non-CPU device.

See: https://scikit-learn.org/dev/modules/array_api.html#scoring-functions

y_true should thereafter convert to binary indicators (using pos_label when available, and moved to the same device, dtype and namespace as y_score just before doing the arithmetic computation.

sample_weight should similarly follow the namespace, dtype and device of y_score using move_to.

See the discussion and implementation of the common test being worked on in #32755 for mixed input metrics.

I will keep your comment in mind for later, but this here addresses a different problem. It fixes a test failure if all inputs (specifically y_true and y_score) are torch tensors on cpu.

In check_array_api_metric() we have defined a numpy_as_array_works boolean that is True if array api is not enabled, but we get input from a non-numpy namespace that can be converted to numpy (see here):

    # When array API dispatch is disabled, and np.asarray works (for example PyTorch
    # with CPU device), calling the metric function with such numpy compatible inputs
    # should work (albeit by implicitly converting to numpy arrays instead of
    # dispatching to the array library).
    try:
        np.asarray(a_xp)
        np.asarray(b_xp)
        numpy_as_array_works = True

In this case, we feed two torch arrays into the metric, but the test fails (see here):

    if numpy_as_array_works:
        metric_xp = metric(a_xp, b_xp, **metric_kwargs)

If I have understood correctly, then we would always try to convert to numpy, if get_config().get("array_api_dispatch", False) but the input would be convertible to numpy, and making sure that we do is the purpose of this test (added in #30454).

(In this WIP PR I haven't taken care of mixed inputs at all yet, also #32755 is not merged yet.)

_average_binary_score is also used in roc_auc_score. So it's probably best to first discuss how to properly add array API support to it, or wait until this PR is merged before creating a PR for roc_auc_score.

I wasn't aware of that, and maybe that had been discussed somewhere I missed. So I did a bit of searching and I have found xpx imported via _array_api.py in sklearn/model_selection/_search.py and in sklearn/tree/tests/test_tree.py.
And I have found from sklearn.externals import array_api_extra as xpx imports in three other files.

I prefer the more explicit import from sklearn.externals, since it is easier to see what is going on the first glance even if you are unfamiliar with array api, and it will have to be obvious for people unfamilar with array api, as it will be in every function in the future and we also want to make it easy for future contributors.

Yes, I ran into the same issue when I was working on roc_auc_score.