To be conistent with other metrics, we need to offer the np.nan option as well.

I think the docstring as is, is vague and doesn't give enough information.

For instance, this warning is now removed: y_pred contains classes not in y_true.

The diff here seems more complicated than the other instances of adding zero_division, and this parameter is doing more than simply replacing a nan of a division by zero.

Better explaining the actual behavior, it'd be more clear how to proceed.

We could improve the current warning to give more info - both warings (old and new) are correct: when y_pred contains classes not in y_true, tp + fn = 0 (sum of true labels) for that class, this results in zero division and balance accuracy is ill-defined in this case.

How about we improve the warning to give more info, e.g., it is ill-defined because y_pred contains classes not in y_true (resulting in zero division when calculating recall)?

I appreciate it is not similar to precision/recall/f1 where there is explicit averaging/an explicit average parameter, so it's not so obvious that averaging is happening. We do say above: "It is defined as the average of recall obtained on each class" but we could be more explicitly explain in zero_division parameter that the zero division occurs when calculating recall for each class?

Similar to what we do in cohen_kappa_score:

(though cohen_kappa_score is even more different and there is no averaging)

Why are we using record=True here?

We don't need to because we don't use the output indeed.

I'm confused with the semantics of this now. Shouldn't zero_division=np.nan to me would mean "if there's a zero division, give me np.nan", but this is kinda the opposite. To me zero_dicision="ignore" would be easier to mean what we have written here.

Basically, we are consistent with the other metric and with previous definition.

recall_score(y_true, y_pred, average="macro", zero_division=np.nan)

I don't think that we should change this behaviour here.

What make the semantic weird here is that the balanced accuracy have this average without any keyword. However, I would find it weird that the results does not match the definition of the recall_score. So it would mean that if we don't like this semantic, we should change for all zero_division parameters around.

I really feel like we're continuing a wrong path here.

Reading the code

recall_score(y_true, y_pred, average="macro", zero_division=np.nan)

w/o reading the docstring, I really expect to get np.nan if there's any zero_division in calculations, not to have them ignored in an aggregate calculation.

I'd be more in favor of deprecating that usage of zero_division=np.nan in favor of zero_division="ignore" or similar.

This is the situation now:

accuracy_score, classification_report, cohen_kappa_score, jaccard_score, matthews_corrcoef are all variations of this text:

    zero_division : {"warn", 0.0, 1.0, np.nan}, default="warn"
        Sets the value to return when there is a zero division,
        e.g. when `y_true` and `y_pred` are empty.
        If set to "warn", returns 0.0 input, but a warning is also raised.

        versionadded:: 1.6

However, classification_report calls precision_recall_fscore_support, which among other metrics has:

f1_score, fbeta_score, precision_recall_fscore_support, precision_score, recall_score:

    zero_division : {"warn", 0.0, 1.0, np.nan}, default="warn"
        Sets the value to return when there is a zero division, i.e. when all
        predictions and labels are negative.

        Notes:
        - If set to "warn", this acts like 0, but a warning is also raised.
        - If set to `np.nan`, such values will be excluded from the average.

        .. versionadded:: 1.3
           `np.nan` option was added.

I do think we need to change the status quo for the second batch, and make the written code by the user intuitive rather than very surprising.

I agree that there is something wrong here.

I don't like the semantic zero_division="ignore" because the exclusion happen at the aggregation level.

However, I'm wondering if we should not only return an aggregation that take the np.nan into account. Practically speaking, returning an average that does not take into account a certain class is actually dangerous. So I need to go back because I don't remember why this behaviour was chosen.

I more and more convinced that we need both np.nan and "ignore". np.nan will be taken into account in the final aggregation if there is one. "ignore" means that the result from the zero_division is ignore in the aggregation stage. My question know is if we have the "ignore" option, what is the behaviour when we don't aggregate?

Basically what do we expect from the following example:

recall_score([[0, 0], [0, 0]], [[1, 1], [1, 1]], average=None, zero_division="ignore")

Oh yeah, I wasn't saying we should remove np.nan from all of them. In my comment here (#28038 (comment)) to me the semantics of the first batch of the metrics (except classification report) is okay as is. We only need to change for the second batch.

np.nan will be taken into account in the final aggregation if there is one

Can you clarify what this means?

Clarification here: #29048 (comment)