AUC ROC Changing after Calibration #31044

This is not necessarily a bug. Calibration optimizes probability quality, not ROC AUC, and it does not promise to preserve every sample's ranking.

With isotonic calibration, ties and local reorderings are normal. A single sigmoid applied to one fixed score is monotonic and should preserve ranking, but CalibratedClassifierCV normally fits several estimator/calibrator pairs and averages their probabilities. That average is not simply a monotonic transform of one original classifier score, so AUC can change as well.

To diagnose it, evaluate on a test set used for neither training nor calibration and compare:

roc_auc_score(y_test, clf.decision_function(X_test))
roc_auc_score(y_test, calibrated.predict_proba(X_test)[:, 1])

If the goal is to calibrate one already-fitted model while changing its ranking as little as possible, fit that model on the training split and calibrate it on a separate split using FrozenEstimator with CalibratedClassifierCV. Then report both ROC AUC and a calibration metric such as log loss or Brier score. A small AUC decrease can be an acceptable trade if probability calibration improves materially.