Add references for multiclass balanced-accuracy definitions #9982
Conversation
|
@amueller |
doc/modules/model_evaluation.rst
Outdated
| @@ -467,6 +467,8 @@ given binary ``y_true`` and ``y_pred``: | |||
| * Normalized class-wise accuracy average as described in [Guyon2015]_ | |||
| * Macro-average recall as described in [Mosley2013]_ | |||
|
|
|||
| Note that none of these different definitions are currently implemented in scikit-learn. | |||
There was a problem hiding this comment.
But macro-average recall is implemented? @jnothman
There was a problem hiding this comment.
I meant not implemented within balanced_accuracy_score
|
I would also mention Class balanced accuracy from here: #6747 (comment) because that's closely related to macro average recall. |
|
The book "MAchine Learning for Predictive Data Analytics" by Kelleher et al also uses macro average recall as average class accuracy. Might be another good reference. |
|
I'm trying to remember if somewhere in this discussion I saw something that takes the recall of each class and the recall of the complement of each class and macro-averages... I.e. the average of balanced accuracy for each binarized class. |
|
@amueller You say it's closely related to the macro average recall. But in the definition given it says:
Isn't this the macro-average recall in itself (i.e. the average over all classes of the recall for each class)? |
|
No, 2.9 is just reiterating what I think is one common definition of balanced accuracy, which is macro-average recall. I guess we could add that to the references that use this definition. I was referring to 3.4, which is class balanced accuracy. |
doc/modules/model_evaluation.rst
Outdated
| * Normalized class-wise accuracy average as described in [Guyon2015]_ | ||
| * Macro-average recall as described in [Mosley2013]_ and [Kelleher2015]_ | ||
|
|
||
| Note that none of these different definitions are currently implemented within |
There was a problem hiding this comment.
I would add either here or above that macro-average recall is implemented in sklearn.
doc/modules/model_evaluation.rst
Outdated
| There is no clear consensus on the definition of a balanced accuracy for the | ||
| multiclass setting. Here are some definitions that can be found in the literature: | ||
|
|
||
| * Normalized class-wise accuracy average as described in [Guyon2015]_ |
There was a problem hiding this comment.
can you give a definition? maybe for both?
|
Oh, sorry. I was too quick. I thought his comments were addressed! Yes,
please fix up my poor work!
|
…ns (scikit-learn#9982)" This reverts commit 9172a59.
|
@jnothman |
|
Pull master onto your local machine and git merge with it.
…On 30 October 2017 at 10:35, Mohamed Maskani ***@***.***> wrote:
@jnothman <https://github.com/jnothman>
I think I mistakenly reverted the changes.
Is there any way to fix that?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#9982 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAEz61QbJwxojiKQZx9d93RjrZAxCP4Vks5sxQuvgaJpZM4QDepx>
.
|
|
Here is the correcting PR #10040 |
Reference Issues/PRs
In addition to #8066
What does this implement/fix? Explain your changes.
I added some references from the literature for the multiclass balanced accuracy. The threads #6747 and #8066 cited some papers and their different implementations of the metric. There was no real consensus about what was the definition to use, so the user might want to implement the one of his/her/their choice.