Nb: In #5516 I preferred adding scorers, but not metric functions, for things like fallout.

What are benchmark results looking like?

Also please prefix your pr title with [WIP] until tests and features are complete

This pull request introduces 1 alert when merging cdd619d into a31a906 - view on LGTM.com

new alerts:

• 1 for Wrong name for an argument in a call

Comment posted by LGTM.com

This pull request introduces 1 alert when merging ec82be3 into a31a906 - view on LGTM.com

new alerts:

• 1 for Wrong name for an argument in a call

Comment posted by LGTM.com

The benchmarking result of precision_recall_fscore_support_with_multilabel_confusion_matrix on multiclass case is much slower than the original one, because of the use of confusion_matrix (10 times slower in some cases) (as shown in #10628 ).

I have optimized the speed for multilabel-indicator case (replace bincount with point-wise multiplication, remove unnecessary expensive _check_targets), for this case, the precision_recall_fscore_support_with_multilabel_confusion_matrix has very close performance with the original implementation (slightly faster).

The precision_recall_fscore_support_with_multilabel_confusion_matrix is only for debugging and optimization purpose and will be removed. And this implementation of precision_recall_fscore_support_with_multilabel_confusion_matrix could pass the tests for precision_recall_fscore_support.

Ok, I will ping you when I finished.
I hope to contribute to sth critical, if you find anything suitable, I would love to give it a try.

That issue looks interesting, I am looking into it.
I think the work on multilabel_confusion_matrix is finished, could you review it when you have the time? @jnothman Thanks!

Do you have some references for users?

I invented the name. The function is mostly to help us implement and facilitate the implementation of arbitrary metrics based on set-wise/binary metrics: Precision, Recall, F1, Fβ, Jaccard, Specificity, etc. The point is that it calculates sufficient statistics for these metrics, just as confusion_matrix calculates sufficient statistics for multiclass metrics (ignoring sample_weight perhaps), and contingency_matrix calculates sufficient statistics for clustering metrics.

Because its needs are mostly internal, and it would mostly be used through the metric aggregates, it is unlikely to be referenced in the literature as such.

It makes sense, however, that utiml similarly implements something like this, because from it they can derive all the metrics they need for multilabel evaluation. Their "absolute matrix" is not identical to our confusion_matrix. It is merely the sum over axis 0 of the output of multilabel_confusion_matrix.

The binary case (at least with 1d input) needs to be handled with a 2x2x2 matrix, in order to support those metrics above. One clarifying alternative would be to make a separate function for multiclass input (which needs to be binarised) from that for multilabel input.

I'd be okay calling this binarized_confusion_matrix, in correspondence with LabelBinarizer (but without any affinity in meaning to Binarizer!).

(1) Regarding the name: I'm fine with both nultilabel_confusion_matrix and binarized_confusion_matrix so I'll follow your final decision. Personally, I prefer binarized_confusion_matrix, because if I put binarized_confusion_matrix and confusion_matrix together, I can figure out some of the relationships and differences between them.
(2) Regarding the R pack, apologies I didn't read the doc carefully. I don't think we need to care about their Absolute Matrix and Proportinal Matrix.
(3) Regarding the binary case, I'm fine with current implementation.

So the remaining things @jnothman
(1) your final decision about the name
(2) 4 minor review comments above

And @jnothman another annoying thing :) Do you think it's acceptable?

multilabel_confusion_matrix([0, 0], [0, 0])
array([[[0, 0],
           [0, 2]]], dtype=int64)

Do you think it's acceptable?

I think that is consistent, and don't have a problem with it.

>>> multilabel_confusion_matrix([0, 0], [0, 0])
array([[[0, 0],
        [0, 2]]])
>>> multilabel_confusion_matrix([0, 0], [0, 0], labels=[0, 1])
array([[[0, 0],
        [0, 2]],

       [[2, 0],
        [0, 0]]])

I'm happy to rename to binarized_confusion_matrix, as long as you reckon that makes sense for the case that the input is already multiple labels.

I think that is consistent, and don't have a problem with it.

I see, yes it's reasonable but a bit tricky.

I'm happy to rename to binarized_confusion_matrix

I think both names are fine and will follow your decision. (binarized_confusion_matrix seems more straightforward but multilabel_confusion_matrix seems to be consistent with R utiml).

Still want to know your opinion about the 4 reviews above (#11179 (review)), especially the second and the fourth one :)

@TomDLT what do you think of the name binarized_confusion_matrix vs multilabel_confusion_matrix vs other??

Any further opinions on the name binarized_confusion_matrix vs multilabel_confusion_matrix vs other for a function which returns a 2x2 confusion matrix for each class in multilabel or multiclass data?

FYI test is failing (apologies I don't have time to investigate now)

=================================== FAILURES ===================================
___________________ test_multilabel_confusion_matrix_errors ____________________
    def test_multilabel_confusion_matrix_errors():
        y_true = np.array([[1, 0, 1], [0, 1, 0], [1, 1, 0]])
        y_pred = np.array([[1, 0, 0], [0, 1, 1], [0, 0, 1]])
    
        # Bad sample_weight
        assert_raise_message(ValueError, "inconsistent numbers of samples",
                             multilabel_confusion_matrix,
                             y_true, y_pred, sample_weight=[1, 2])
        assert_raise_message(ValueError, "could not be broadcast",
                             multilabel_confusion_matrix,
                             y_true, y_pred,
                             sample_weight=[[1, 2, 3],
                                            [2, 3, 4],
>                                           [3, 4, 5]])
y_pred     = array([[1, 0, 0],
       [0, 1, 1],
       [0, 0, 1]])
y_true     = array([[1, 0, 1],
       [0, 1, 0],
       [1, 1, 0]])
/home/travis/build/scikit-learn/scikit-learn/sklearn/metrics/tests/test_classification.py:484: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
exceptions = <type 'exceptions.ValueError'>, message = 'could not be broadcast'
function = <function multilabel_confusion_matrix at 0x7f3042ff0c80>
args = (array([[1, 0, 1],
       [0, 1, 0],
       [1, 1, 0]]), array([[1, 0, 0],
       [0, 1, 1],
       [0, 0, 1]]))
kwargs = {'sample_weight': [[1, 2, 3], [2, 3, 4], [3, 4, 5]]}
e = ValueError('a.shape[axis] != len(repeats)',)
error_message = 'a.shape[axis] != len(repeats)'
    def assert_raise_message(exceptions, message, function, *args, **kwargs):
        """Helper function to test the message raised in an exception.
    
        Given an exception, a callable to raise the exception, and
        a message string, tests that the correct exception is raised and
        that the message is a substring of the error thrown. Used to test
        that the specific message thrown during an exception is correct.
    
        Parameters
        ----------
        exceptions : exception or tuple of exception
            An Exception object.
    
        message : str
            The error message or a substring of the error message.
    
        function : callable
            Callable object to raise error.
    
        *args : the positional arguments to `function`.
    
        **kwargs : the keyword arguments to `function`.
        """
        try:
            function(*args, **kwargs)
        except exceptions as e:
            error_message = str(e)
            if message not in error_message:
                raise AssertionError("Error message does not include the expected"
                                     " string: %r. Observed error message: %r" %
>                                    (message, error_message))
E               AssertionError: Error message does not include the expected string: 'could not be broadcast'. Observed error message: 'a.shape[axis] != len(repeats)'
args       = (array([[1, 0, 1],
       [0, 1, 0],
       [1, 1, 0]]), array([[1, 0, 0],
       [0, 1, 1],
       [0, 0, 1]]))
e          = ValueError('a.shape[axis] != len(repeats)',)
error_message = 'a.shape[axis] != len(repeats)'
exceptions = <type 'exceptions.ValueError'>
function   = <function multilabel_confusion_matrix at 0x7f3042ff0c80>
kwargs     = {'sample_weight': [[1, 2, 3], [2, 3, 4], [3, 4, 5]]}
message    = 'could not be broadcast'

• multilabel_confusion_matrix makes it clear on which problem you can use the function.
• binarized_confusion_matrix makes it clear what is actually computed.

Both names are fine. I would be slightly in favor of multilabel_confusion_matrix, since this may help users find the function. The question of what is actually computed can be found in the docstring: Multiclass data will be treated as if binarized under a one-vs-rest transformation..

I think we can merge. Thanks all for the great work!

And thanks @ShangwuYao for making this happen!