@untom The Pascal VOC is multi-class multi-label, right? Pixels are not evaluated individually, but the whole image is. And then people usually look at per-class measures.

But you're right, it looks like the original definition is different from ours, see https://encrypted.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=3&cad=rja&uact=8&ved=0ahUKEwjT1-XGvY_PAhXGeD4KHSFLBUAQFggwMAI&url=http%3A%2F%2Fwww.informatica.si%2Findex.php%2Finformatica%2Farticle%2Fdownload%2F753%2F608&usg=AFQjCNFd6KF03h8j6Yfk_6hQbx6oeBZP8g&sig2=1OOs_6BFsyaPcxGvPbEtLQ
Where is it in the original paper?

According to the evaluation code (http://host.robots.ox.ac.uk/pascal/VOC/voc2012/VOCdevkit_18-May-2011.tar), the multi-class VOC score is the average of per-class Jaccard scores.
The relevant part is from line 74 to line 90 in VOCevalseg.m

@untom hi, i think you are right.

Currently jaccard_similarity_score just counts the samples in intersection of pred and true, which is not correct for the definition of Jaccard similarity coefficient. We need to calculate union.

https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/metrics/classification.py#L174

I am not sure what it looks like for multi label.

I agree that this seems to be strange even for the binary case. I would have thought Jaccard is an alternative to precision or recall or F1 (= Dice coefficient) in evaluating performance, in the binary case, on a single positive class, i.e. "true positives / (true positives + false positives + false negatives)". In particular, the binary implementation in our case does not seem to equate to the multilabel implementation run over a single class.

Regarding @untom's initial contention that multiclass implementation is incorrect, I agree that the multiclass implementation is useless. I don't think that the macro averaging he suggests is the only way to go about it, and as with P/R/F, micro-averaging excluding a majority negative class is still meaningful; weighted macro-average may also be feasible.

So yes, multiple strange things in our jaccard implementation IMO, and at a glance I don't see how the reference given in #1795 tells us about the multiclass case.

Labelling this a bug.

Any update on this issue? I believe this issue should be assigned as a high priority bug. Incorrect evaluation metrics could mislead many people in interpreting and reporting their experimental results. I may consider submitting a patch if no one else is going to work on it.

Yes, probably. PR welcome

@jacobdang polite ping. Are you still working? It is very much fine if you are still working but if you are not, then this is looks like a really good issue I would like to work on.

@gxyd I am catching a deadline so may not be able to work on it immediately. So if you could help that will be highly appreciated. Thanks. 👍

Any progress about this issue yet?

Waiting for a second review at #10083

Unfortunately the code there is quite complicated, and perhaps could be simplified if something like #10628 existed, but I haven't had time to make that work let alone bring it up to mergeable standard.

So, is the Jaccard similarity score a valid metric to use for multilabel classification problems?

@agamemnonc Only if you want the average score across the samples. It is totally different than intersection over union.

This problem in scikit-learn has just recently caused some big headache for me in my research.
I work with segmentation and was surprised to see how scikit-learn interprets the jaccard index metric.

@TSchattschneider I feel you. I remember how frustrated I was. I had the same problem. This bug should be flagged in the documentation, unless is fixed.