ROC curves upside down #31683

pedro-w · 2025-06-30T14:25:16Z

pedro-w
Jun 30, 2025

I've been fitting an SVC to experimental data and came across an issue where RocCurveDisplay sometimes was putting the curves upside down.
I know this happens if the positive label gets mixed up between fitting and plotting but here it is happening randomly.

It only happens when I specify probability=True to the model which the docs say requires randomness to compute the probabilities. I did a bit more investigation to get to a much smaller code example. I found that passing in a RandomState gives repeatable behaviour with some random states plot the ROC curve correctly and some wrongly.
From this I found that predict_proba seems to do one of two things depending on the random state and this leads to one of the two ROC curves.
I couldn't get any further because it seems very data dependent; just deleting a few data rows made the problem go away but I was able to manually truncate my data to 2 decimal places and nothing changed. Sorting the data by increasing value also seems to be necessary.

Here is an illustration - the first column is a 'correct' ROC curve and the corresponding probabilities, and the second is a 'wrong' one. Note that the y-scale of probabilities is very different.

Any comments welcome, especially if I am doing something daft here.

Thanks.

Here is the code, from a Jupyter notebook

from matplotlib import pyplot as plt
from sklearn.metrics import  RocCurveDisplay
from sklearn.svm import SVC
import numpy as np
from numpy.random import RandomState
import pandas as pd

rndst1 = RandomState(1)
rndst2 = RandomState(2)

def load_data():
    negs = [
        0.14,
        0.59,
        1.13,
        2.60,
        2.92,
        2.98,
        3.99,
        4.08,
        4.43,
        7.73,
        10.98,
    ]
    poss = [
        1.84,
        2.15,
        2.73,
        3.46,
        3.59,
        3.63,
        3.67,
        3.75,
        4.49,
        5.22,
        5.33,
        5.35,
        5.51,
        5.69,
        5.72,
        5.90,
        5.98,
        6.29,
        7.96,
        7.98,
        8.21,
        8.62,
        9.27,
        10.88,
        11.84,
        13.11,
        19.12,
        20.09,
        21.99,
        25.00,
        35.00,
    ]

    return pd.DataFrame(
        {
            "Class": np.concat([["NEG"] * len(negs), ["POS"] * len(poss)]),
            "ScoreA": np.concat([negs, poss]),
        }
    )

model1 = SVC(probability=True, kernel="rbf", random_state=rndst1)
model2 = SVC(probability=True, kernel="rbf", random_state=rndst2)

data = load_data()
data.sort_values(by="ScoreA", inplace=True)

Xs = data[["ScoreA"]]
ys = data["Class"]

model1.fit(Xs, ys)
model2.fit(Xs, ys)

probs1 = model1.predict_proba(Xs)[:, 1]
probs2 = model2.predict_proba(Xs)[:, 1]
probs=pd.DataFrame({"probs1":probs1, "probs2":probs2})
display(probs.describe().style.format(precision=2))
fig, axs = plt.subplots(2, 2, figsize=(10,10))
fig.tight_layout()
RocCurveDisplay.from_predictions(y_pred=probs1, y_true=ys, pos_label="POS", ax=axs[0,0])
RocCurveDisplay.from_predictions(y_pred=probs2, y_true=ys, pos_label="POS", ax=axs[0,1])
xv = np.arange(len(probs1))
axs[1,0].plot(xv, probs1)
axs[1,1].plot(xv, probs2)

I am using scikit-learn 1.5.2, numpy 2.1.3 and python 3.12.10.

Kxrma47 · 2026-08-31T13:04:57Z

Kxrma47
Aug 31, 2026

This can happen with SVC(probability=True) on a small dataset. The SVM decision function and the probability model are fitted by different procedures: libsvm fits a Platt-scaling model using internal cross-validation, and that calibration step uses randomness. With only 42 samples, different folds can produce an unstable sigmoid and probabilities whose ordering disagrees with the decision scores.

For an ROC curve, probabilities are not required; ROC only needs a ranking score. Use the decision function directly:

scores = model1.decision_function(Xs)
RocCurveDisplay.from_predictions(
    y_true=ys, y_pred=scores, pos_label="POS"
)

Also check model.classes_ before selecting a probability column rather than assuming column 1 is always the desired class:

pos_col = np.flatnonzero(model1.classes_ == "POS")[0]
probs = model1.predict_proba(Xs)[:, pos_col]

If calibrated probabilities are genuinely needed, fit the SVC without probability=True, reserve a separate calibration set, and use CalibratedClassifierCV (or nested cross-validation). Evaluate both discrimination and calibration on untouched data. Plotting the ROC on the same samples used to fit such a small model will otherwise give a very noisy picture.

0 replies

Navigation Menu

Uh oh!

Uh oh!

ROC curves upside down #31683

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Uh oh!

Uh oh!

ROC curves upside down #31683

Uh oh!

pedro-w Jun 30, 2025

Replies: 1 comment

Uh oh!

Kxrma47 Aug 31, 2026

pedro-w
Jun 30, 2025

Kxrma47
Aug 31, 2026