DOC Fix det_curve fnr docstring threshold direction - #34390
Conversation
The Returns entry for `fnr` described the false negative rate over predictions with `score >= thresholds[i]`, copied from the `fpr` entry. A false negative is a positive sample predicted negative, i.e. one scoring below the threshold, so FNR at element i is the rate of positives with `score < thresholds[i]`. The function's own example confirms FNR uses score < threshold (thresholds=[0.35, 0.4, 0.8], fnr=[0., 0.5, 0.5]). Distinct from scikit-learn#34325, which fixes the separate thresholds-direction wording.
|
Thank you for opening your first pull request to scikit-learn! 🎉 To help get your contribution reviewed, please make sure that:
|
eeshsaxena
left a comment
There was a problem hiding this comment.
This looks correct to me. A false negative is a positive sample that scores below the threshold (so it's predicted negative), so fnr[i] should indeed be described in terms of score < thresholds[i], not >=.
I checked it against the actual output to be sure:
import numpy as np
from sklearn.metrics import det_curve
y_true = np.array([0, 0, 1, 1, 1, 0])
y_score = np.array([0.1, 0.4, 0.35, 0.8, 0.6, 0.55])
fpr, fnr, thr = det_curve(y_true, y_score)
pos = y_score[y_true == 1]
# fnr matches "fraction of positives with score < threshold"
assert np.allclose(fnr, [np.mean(pos < t) for t in thr])The fnr values (0, 0.333, 0.333, 0.333) line up with score < thresholds[i] and clearly do not match score >= thresholds[i] (1, 0.667, 0.667, 0.667), which confirms the sentence was copied from the fpr entry above (whose >= wording is correct for the false positive rate). Nice, focused fix.
The `thresholds` return of det_curve is documented as "Decreasing" but the function returns them in increasing order, as its own example shows. Correct the wording and assert monotonic non-decreasing thresholds in test_det_curve_toydata. Incorporates scikit-learn#34325 at the maintainer's request; combined with the fnr threshold-direction fix already in this PR.
|
Done @jeremiedbb, added the threshold-direction fix too. So this PR now covers both the fnr sentence and the thresholds direction. |
eeshsaxena
left a comment
There was a problem hiding this comment.
Thanks for the ping. I went through the change and it looks correct to me.
I reproduced the docstring example to double-check both edits:
y_true = np.array([0, 0, 1, 1]); y_score = np.array([0.1, 0.4, 0.35, 0.8])
fpr, fnr, thr = det_curve(y_true, y_score)
# fpr [0.5 0.5 0. ] fnr [0. 0.5 0.5] thr [0.35 0.4 0.8]fnr->score < thresholds[i]: a false negative is a positive sample predicted negative, i.e. scoring below the threshold, so this is the right direction. Thethr[0] = 0.35point is the one that disambiguates it: the two positives score0.35and0.8, none are< 0.35, givingfnr = 0, which matches. The old>= thresholds[i]wording would give1.0there, so the previous text was indeed copied fromfprand wrong.thresholds->Increasing: the returnedthris[0.35, 0.4, 0.8], consistent with the 1.7 change (arbitrary threshold at +inf appended for thefpr=0/fnr=1case), soIncreasingmatches the current behaviour.
Nice that the fpr entry is left as >= thresholds[i] - I checked and that one is still correct (fpr counts negatives at/above the threshold), so keeping the fpr/fnr boundary asymmetric is intentional and right, not an oversight.
The added assert np.all(np.diff(thresholds) >= 0) in test_det_curve_toydata is a nice guard for the increasing-order claim across the parametrized cases.
LGTM from me - deferring to the maintainers for the final call.
Reference Issues/PRs
Issue #34310 and PR #34325 concern the adjacent
thresholdsincreasing/decreasing wording; this PR fixes the separatefnrthreshold-direction sentence.What does this implement/fix? Explain your changes.
The
Returnssection ofsklearn.metrics.det_curvedescribesfnras "the false negative rate of predictions with score >= thresholds[i]." That text is copied from thefprentry and is wrong for the false negative rate: a false negative is a positive sample predicted negative, i.e. one scoring below the threshold, so FNR at elementiis the rate of positives withscore < thresholds[i]. The function's own example confirms this: withthresholds=[0.35, 0.4, 0.8]andfnr=[0., 0.5, 0.5], the positive scoring0.35becomes a false negative at threshold0.4(0.35 < 0.4). This is a one-token docstring fix (>=to<) on thefnrline only; thefprline is left unchanged (its>=is correct), there is no behavior change, and the existing doctest still passes.First time contributor introduction
Hi, I'm Christian, and I'm a rising sophomore studying CS at Johns Hopkins. I used scikit-learn for binary classification for a project where a false negative (a missed positive) can potentially be dangerous for the user/client. I ran into this while using
det_curveto choose an operating threshold on the miss-rate vs false-alarm tradeoff, and the invertedfnrthreshold direction in the docs pointed me the wrong way until the worked example set me straight.AI usage disclosure
I used AI assistance for:
Any other comments?
The correctness proof is already encoded in the function's own doctest (the
fnr=[0., 0.5, 0.5]example), so no new test is needed.