cc: @jnothman

So, this is effectively a reversion of @jblackburne's #3268, to which @arjoly objected, and which I and @ogrisel agreed was acceptable. You're right that there are solutions with dynamic tolerance. It seems like at least the fixed tolerance should be configurable.

So what to do? Should I add function parameters to the top-most roc_curve function? That's a significant interface change, I didn't want to start that bold, but it's the only option out of the listed that does not obscure other use cases.

As you already stated in the corresponding issue, the default should be 0.0, like I implemented it in this PR, the reason being that the user does not even know that some potential thresholds are just discarded because they don't fulfill a more or less arbitrary tolerance threshold.

I just checked the duplicate PR. I don't think that @arjoly's (understandable!) arguments apply in this case. In my real problem where this bug occured, the reduced number of thresholds suddenly yields artifacts in the ROC curve.

I don't want to get into the details of my actual problem, but here you can see what happens. The input gradient norm line (light blue) should look like in this file correct
However, this can only be achieved by scaling the scores (identical in both plots!) with a factor of 10. If I don't do this, it looks like this: incorrect
You cannot see it here, but I also did a scatter plot of the incorrect curve and found that the vertical part consists of a lot of points, while the diagonal part consists of no points except for its two endpoints.

And here, it's not just plotting artifacts, but the evaluation of my classifier is just plaing wrong (from close to perfect to suboptimal...).

I don't think there's a big problem with adding new parameters. It's better
than having the mechanism hidden away and biting people at random. Making
this issue more obvious is also an option: warn if isclose is going to
conflate neighboring scores, and give a suggested solution to the user.

On 21 April 2016 at 23:56, Maximilian Soelch [email protected]
wrote:

I just checked the duplicate PR. I don't think that @arjoly
https://github.com/arjoly's (understandable!) arguments apply in this
case. In my real problem where this bug occured, the reduced number of
thresholds suddenly yields artifacts in the ROC curve.

I don't want to get into the details of my actual problem, but here you
can see what happens. The input gradient norm line (light blue) should look
like in this file correct
https://github.com/scikit-learn/scikit-learn/files/230022/on_roc.pdf
However, this can only be achieved by scaling the scores (identical in
both plots!) with a factor of 10. If I don't do this, it looks like this:
incorrect
https://github.com/scikit-learn/scikit-learn/files/230018/on_roc.pdf
You cannot see it here, but I also did a scatter plot of the incorrect
curve and found that the vertical part consists of a lot of points, while
the diagonal part consists of no points except for its two endpoints.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#6693 (comment)

OK, will do later.

Hi, I caused all this with my previous PR. Sorry.

Looking back, I could have solved my problem more easily by just rounding my y_score before passing it into roc_curve(). It wasn't really fair to ask sklearn to solve what was essentially a problem in my client code.

So instead of making the threshold dropping more clever or adding kwargs that almost nobody will ever use, I propose just dropping the isclose altogether. This would shift the responsibility for controlling roundoff back to the client where it belongs. I could even put together a PR to do that if you like.

Thanks for clarification.

Upon reconsideration, I don't think extra arguments are the best way. Way too much needs to be changed for little improvement, and documenting it properly is very hard.

I second @jblackburne's: It should be left to the user to check y_score (which was the very first thing I did when I ran into the error). However, I don't think a new PR is necessary, because applying isclose to delete real duplicates is fine, which is achieved by the current status of this PR (which sets tolerances to 0).

Well, once atol and rtol are zero, isclose is a no-op. That will be confusing for readers of the code. Also note that we can delete the isclose function from utils/fixes.py if we get rid of it here.

EDIT: That's inaccurate, it's not a no-op. What I mean is that where and diff eliminate the exactly equal scores by themselves without needing logical_not and isclose.

That's true, it's redundant. If @jnothman agrees, I will adjust this PR accordingly by simply deleting isclose entirely.

Okay. Let's revert, assuming @ogrisel's happy with that. :/

And thanks for the input @jblackburne, even as I'm sorry we believed you the first time :P

fixed in #7353.