Codestin Search App

ogrisel · 2026-06-16T14:12:58Z

Fixes: #34312

Note: this draft PR was started a long time ago. Work is needed to realign it with the current main branch.

…ssing strategies

…cal study

antoinebaker · 2026-06-19T14:59:25Z

I agree, we should definitively use the logits for sigmoid recalibration / Platt scaling.

@ogrisel Could you grant me rights to push to your PR ? I'll try to merge main and update this draft PR.

ogrisel · 2026-06-19T15:18:22Z

@antoinebaker you were already registered as a collaborator on my scikit-learn fork :)

Please go ahead with the merge and update of this PR.

antoinebaker · 2026-06-25T14:57:42Z

Hi @ogrisel and @virchan, the PR is currently passing the CI.

However I would like to refactor the code, in line with your code comments @ogrisel and your comment @virchan. I would greatly appreciate your feedback (see Comments section) to discuss future implementations !

With the help of cursor, here a summary of the current logic of _enforce_logits (which role is to preprocess the predictions as logits before feeding them to the calibrators).

`predict_proba`

Inputs are always probabilities p_k = P(y=k).

Calibration method	Target logit space	Transformation
sigmoid / isotonic	OvR (“Bernoulli”) logits	`z_k = logit(p_k)`
temperature	Multinomial logits	`z_k = log(p_k) - log(p_mean)`

`decision_function`

Inputs are scores s whose meaning varies by estimator.

Calibration method	Target logit space	Current transformation
sigmoid / isotonic	OvR (“Bernoulli”) logits	Binary (1D): pass-through `s` (assumed positive-class log-odds). Multiclass (K>2): `softmax(s) → p_k`, then `z_k = logit(p_k)`.
temperature	Multinomial logits	Pass-through `s` (assumed joint logits with `p = softmax(s)`).

Comments

For me, the handling of predict_proba is correct and solves #34312. The handling of decision_function is more debatable. It seems to always assume "Multinomial logits"-like scores, and in particular converts them to Bernoulli logits for the sigmoid/isotonic calibration method.

Here are some implementation options I'd like to discuss:

get rid of the logit_preprocessing parameter. Instead we always decide based on the calibration method (sigmoid vs softmax) and response_method_name (predict_proba vs decision_function)
only convert predict_proba for now, let decision_function passthrough. This would make the code much easier to understand. This would already fixes # 34312.
try to determine (for an estimator) if the decision_function scores are "morally" more Bernoulli vs Multinomial logits and convert based on that. But this seems difficult to do in general.
always convert the decision_function scores to the target logit space, even if they are in the correct logit space to start with. Currently done for the sigmoid method, but could be done for the temperature method as well.

ogrisel · 2026-06-25T15:33:38Z

get rid of the logit_preprocessing parameter. Instead we always decide based on the calibration method (sigmoid vs softmax) and response_method_name (predict_proba vs decision_function)

I think I originally introduced the logit_preprocessing parameter in order to be able to conduct an empirical evaluation similarly to what was done in the CalArena paper. This evaluation was added at the end of examples/calibration/plot_calibration_multiclass.py but it wasn't meant to stay after the review of the PR. Since the results of the CalArena paper are clear, I think we can indeed drop this parameter.

only convert predict_proba for now, let decision_function passthrough. This would make the code much easier to understand. This would already fixes # 34312.

I agree.

ogrisel · 2026-06-25T15:59:22Z

@antoinebaker Let's first add temperature scaling to examples/calibration/plot_calibration_multiclass.py and then check then results collected at the end of this notebook that "softmax" logits preprocessing is always better than (or very similar to) sigmoid logits preprocessing for temperature scaling.

Let's also check that "sigmoid" logits preprocessing is always better or similar to "softmax" logits preprocessing for the two OvR methods.

If all goes according to our plan, we can hard code the logits preprocessing logic and remove the empirical evaluation of the logits_processing parameter from that example.

lorentzenchr · 2026-07-27T15:28:00Z

+implements :term:`predict_proba`, :math:`f_i = \text{logit}(\hat{p}_i)` is the
+logit of the predicted probability; otherwise, :math:`f_i` is the score from
+:term:`decision_function`. :math:`A` and :math:`B` are real numbers to be
+determined when fitting the regressor via maximum likelihood.


"fitting the regressor" is a bit weird as we speak of classifiers

Done in 7ebe1a8

lorentzenchr · 2026-07-27T15:51:50Z

+_CLASSIFIER_RESPONSE_METHODS = ("predict_proba", "decision_function")
+
+
+def _ensure_logits(predictions, response_method_name, method):


We already have _convert_to_logits in this file. Why a new function.

_ensure_logits and _convert_to_logits both do some sort of "logit preprocessing", but in a different way, I tried to merge them at some point but the resulting code was too messy.

_ensure_logits is only used inside CalibratedClassifierCV while _convert_to_logits is only used inside _TemperatureScaling.

Co-authored-by: Christian Lorentzen <[email protected]>

virchan

Both calibration.py and test_calibration.py LGTM. I will review the examples later.

ogrisel · 2026-08-31T12:45:14Z

Merging main to re-trigger an HTML rendering of the doc changed in this PR.

ogrisel

I did another pass of review and tried to mark as resolved all previous comments by @lorentzenchr on outdated sections of the diff that seem to have been actually addressed by subsequent commits. This was quite painful because I think the comment navigation of the diff view of github seems to be badly broken on this PR...

From my point of view this seems good to go.

Co-authored-by: Olivier Grisel <[email protected]>

lorentzenchr

Further partial review. What remains is reviewing the actual fix 😏

lorentzenchr · 2026-09-11T15:43:24Z

+# indicated by the diagonal calibration curve. This is expected since we
+# adjusted the regularization parameter to minimize the :ref:`log_loss` via
+# internal cross-validation.


This is expected

Is it? I would put it less strongly.

lorentzenchr · 2026-09-11T15:44:27Z

+# by default because of the redundant features which violate the assumption of
+# feature-independence and result in an **overly confident** classifier, which
+# is indicated by the typical transposed-sigmoid calibration curve.


Can you split the sentence? It is too long and complex.

lorentzenchr · 2026-09-11T15:46:05Z

+#
+# Post-hoc calibration of the predicted probabilities of
+# :class:`~sklearn.naive_bayes.GaussianNB` with :ref:`isotonic` or
+# :ref:`Sigmoid regression <sigmoid_regressor>` can fix this issue as can be


Suggested change

# :ref:`Sigmoid regression <sigmoid_regressor>` can fix this issue as can be

# :ref:`Sigmoid regression <sigmoid_regressor> fix this issue to a good extent as can be

lorentzenchr · 2026-09-11T15:47:09Z

+# because of the limited amount of training and calibration data (1,000 samples
+# for both).
 #
 # Below we will make a quantitative analysis considering several classification


Suggested change

# Below we will make a quantitative analysis considering several classification

# Below we make a quantitative analysis considering several classification

Same as section dataset.

lorentzenchr · 2026-09-11T15:52:29Z

+# Post-hoc calibration improves the :ref:`brier_score_loss` (a metric composed
+# of calibration term and refinement term) and :ref:`log_loss` as expected


Both metrics, Brier score and log loss, are composed of a calibration (reliability) and refinement (resolution) term.

lorentzenchr · 2026-09-11T16:06:15Z

+# According to the Brier score, the calibrated classifier is also slightly
+# better than the original model.
+#
+# Be aware that an improvement in log-loss or Brier score on a test set is not


Suggested change

# Be aware that an improvement in log-loss or Brier score on a test set is not

# Be aware that an improvement in log loss or Brier score on a test set is not

lorentzenchr · 2026-09-11T16:08:32Z

+# always guaranteed since the calibration set is finite and sometimes the base
+# classifier is already well calibrated.


We could just state the fact and remove the "because".
Or we add that calibration might decrease the resolution/discrimination term of the loss.

lorentzenchr · 2026-09-11T16:09:16Z

+
+# %%
+#
+# Let's now do the same for various classifiers with different mis-calibration


Suggested change

# Let's now do the same for various classifiers with different mis-calibration

# Let's now repeat the above for various classifiers with different mis-calibration

lorentzenchr · 2026-09-11T16:12:28Z

+# - Some estimators such as highly regularized polynomial classifiers and
+#   shallow tree-based models tend to be under-confident by default and each


Suggested change

# - Some estimators such as highly regularized polynomial classifiers and

# shallow tree-based models tend to be under-confident by default and each

# - Some estimators such as the highly regularized polynomial classifier and

# the shallow tree-based model tend to be under-confident by default and each

Similar below.

lorentzenchr · 2026-09-11T16:14:59Z

+# - The temperature scaling method applies a single global rescaling of the
+#   multinomial logits. This induces a smooth mapping that preserves the
+#   ranking of the predicted classes while adjusting their confidence.


It's not so much the multinomial logits, it is that it minimized the multinomial log loss (instead of ovr binary log loss + normalization).

lorentzenchr

Remaining part of the review.

lorentzenchr · 2026-09-12T07:48:01Z

+        # TODO: once we have a calibration loss, use it instead of the
+        # brier score to check recalibration.


This is absolute not related to the fix of this PR. But ok, let's keep it.

lorentzenchr · 2026-09-12T08:05:02Z

-    # Check probabilities sum to 1
-    assert_allclose(np.sum(probas, axis=1), np.ones(len(X_test)))
+    y_pred_cal = cal_clf.predict_proba(X_test)
+    assert_allclose(np.sum(y_pred_cal, axis=1), np.ones(len(X_test)))


Suggested change

assert_allclose(np.sum(y_pred_cal, axis=1), np.ones(len(X_test)))

assert_allclose(np.sum(y_pred_cal, axis=1), 1)

lorentzenchr · 2026-09-12T08:06:08Z

+    labels = np.arange(n_classes)
+    bs_uncal = brier_score_loss(y_test, y_pred_uncal, labels=labels)
+    bs_cal = brier_score_loss(y_test, y_pred_cal, labels=labels)


Suggested change

labels = np.arange(n_classes)

bs_uncal = brier_score_loss(y_test, y_pred_uncal, labels=labels)

bs_cal = brier_score_loss(y_test, y_pred_cal, labels=labels)

bs_uncal = brier_score_loss(y_test, y_pred_uncal, labels=cal_clf.classes_)

bs_cal = brier_score_loss(y_test, y_pred_cal, labels=cal_clf.classes_)

This is cleaner and more secure.

lorentzenchr · 2026-09-12T08:11:00Z

+    All calibration methods are expected to improve the log-loss of the base
+    classifiers given enough calibration data because the log-loss is a
+    strictly proper scoring rule.


I am not 100% convinced of the premise. A proper scoring rule is decomposed as:
loss = mis-calibration - discrimination + entropy. Entropy is constant (given by data only, not the model). If we change the predictions, we change the mis-calibration term, but also the discrimination term.

lorentzenchr · 2026-09-12T08:11:49Z

+def test_ensure_logits_decision_function(method, predictions):
+    # Apart from reshaping, this is a passthrough.
+    logits = _ensure_logits(predictions, "decision_function", method)
+    assert_allclose(logits.ravel(), predictions.ravel())


Is the ravel needed?

lorentzenchr · 2026-09-12T08:27:59Z

+
+    When the response method is ``predict_proba``:
+
+    - For ``method='sigmoid'``, Bernoulli logits are computed per class


Suggested change

- For ``method='sigmoid'``, Bernoulli logits are computed per class

- For ``method='sigmoid'``, logits are computed per class

lorentzenchr · 2026-09-12T08:29:52Z

+    eps = xp.finfo(predictions.dtype).eps
+    eps_ = xp.asarray(eps, dtype=predictions.dtype, device=device_)


Suggested change

eps = xp.finfo(predictions.dtype).eps

eps_ = xp.asarray(eps, dtype=predictions.dtype, device=device_)

eps = xp.asarray(xp.finfo(predictions.dtype).eps, dtype=predictions.dtype, device=device_)

Or just renaming eps_ to eps.

lorentzenchr · 2026-09-12T08:31:19Z

+            return xp.reshape(predictions, (-1, 1))
+        return predictions
+
+    if method == "isotonic":


Suggested change

if method == "isotonic":

elif method == "isotonic":

and so on. Not really needed, but a bit cleaner, imo.

lorentzenchr · 2026-09-12T08:31:47Z

+    raise ValueError(
+        f"Unknown calibration method: {method}. "
+        "Expected 'sigmoid', 'isotonic', or 'temperature'."
+    )


Suggested change

raise ValueError(

f"Unknown calibration method: {method}. "

"Expected 'sigmoid', 'isotonic', or 'temperature'."

)

else:

raise ValueError(

f"Unknown calibration method: {method}. "

"Expected 'sigmoid', 'isotonic', or 'temperature'."

)

lorentzenchr · 2026-09-12T08:33:25Z

+    )
+
+
+def _to_calibration_logits(predictions, *, response_method_name, method, classes=None):


This function is only used once: Do we needed it or better put the code in place?

lorentzenchr · 2026-09-12T08:46:28Z

For a bugfix this is quite a large PR. I understand that examples are improved, too.

ogrisel added 9 commits April 17, 2025 10:07

wip

4a21122

wip

1259c82

wip

44a614c

Add analysis of the results in the multiclass example

c34f4c7

Make it possible to select different kinds of logit preprocessing

dabfe7c

Improve example wording and run extended evaluation of logit preproce…

cc1f6ef

…ssing strategies

Small fixes

561169d

wip

ad215c2

Also include the impact of ensembling or not in the multiclass empiri…

837b776

…cal study

ogrisel mentioned this pull request Jun 16, 2026

CalibratedClassifierCV(method="sigmoid") should always take the raw logits as inputs instead of the predicted probabilities #34312

Open

antoinebaker moved this to In progress in Labs Jun 19, 2026

antoinebaker added this to Labs Jun 19, 2026

virchan added the module:calibration label Jun 20, 2026

virchan reviewed Jun 20, 2026

View reviewed changes

Comment thread sklearn/calibration.py Outdated

antoinebaker added 9 commits June 23, 2026 11:31

Merge branch 'main' into fix-always-use-logits-in-CalibratedClassifierCV

ff56f7d

fix logits / reshape order

4044dff

changelog

4402cf5

docstrings

fc92a22

explicit l1_ratios

7a3a53a

specify use_legacy_attributes

41349c6

docstring MultinomialLogit

d5f2c72

increase code coverage

7711421

remove deprecated cv=prefit

983008e

ogrisel commented Jun 25, 2026

View reviewed changes

Comment thread examples/calibration/plot_calibration_curve.py Outdated

check best logit preprocessing temperature

e50032b

doc

219814a

antoinebaker reviewed Jul 24, 2026

View reviewed changes

Comment thread examples/calibration/plot_calibration_multiclass.py Outdated

antoinebaker and others added 4 commits July 24, 2026 16:18

reverse diff

4303a86

update doc

7c49c38

remove verbose discussion

48eac72

Merge branch 'main' into fix-always-use-logits-in-CalibratedClassifierCV

8006e69

lorentzenchr reviewed Jul 27, 2026

View reviewed changes

antoinebaker and others added 2 commits July 28, 2026 16:32

docs

a9775de

Apply suggestions from code review

20f0dcf

Co-authored-by: Christian Lorentzen <[email protected]>

github-actions Bot added the CI:Linter failure The linter CI is failing on this PR label Jul 28, 2026

fix renaming

de9881a

github-actions Bot removed the CI:Linter failure The linter CI is failing on this PR label Jul 28, 2026

antoinebaker added 2 commits July 28, 2026 16:51

fix renaming again

99f7355

add todo

83117e6

antoinebaker moved this from In progress to PR waiting for reviews in Labs Jul 29, 2026

virchan reviewed Aug 1, 2026

View reviewed changes

ogrisel added the CUDA CI label Aug 10, 2026

github-actions Bot removed the CUDA CI label Aug 10, 2026

Merge branch 'main' into fix-always-use-logits-in-CalibratedClassifierCV

c5ec6d7

increase rtol

645e5c0

ogrisel commented Sep 1, 2026

View reviewed changes

Comment thread doc/modules/calibration.rst Outdated

ogrisel added the CUDA CI label Sep 1, 2026

github-actions Bot removed the CUDA CI label Sep 1, 2026

antoinebaker and others added 2 commits September 1, 2026 17:32

Update doc/modules/calibration.rst

7ebe1a8

Co-authored-by: Olivier Grisel <[email protected]>

Merge branch 'main' into fix-always-use-logits-in-CalibratedClassifierCV

ef18067

lorentzenchr reviewed Sep 11, 2026

View reviewed changes

lorentzenchr changed the title ~~Fix always use logits in calibrated classifier cv~~ FIX always use logits in CalibratedClassifierCV Sep 11, 2026

lorentzenchr reviewed Sep 12, 2026

View reviewed changes

		_CLASSIFIER_RESPONSE_METHODS = ("predict_proba", "decision_function")


		def _ensure_logits(predictions, response_method_name, method):

	# :ref:`Sigmoid regression <sigmoid_regressor>` can fix this issue as can be
	# :ref:`Sigmoid regression <sigmoid_regressor> fix this issue to a good extent as can be

	# Below we will make a quantitative analysis considering several classification
	# Below we make a quantitative analysis considering several classification

		# Post-hoc calibration improves the :ref:`brier_score_loss` (a metric composed
		# of calibration term and refinement term) and :ref:`log_loss` as expected

	# Be aware that an improvement in log-loss or Brier score on a test set is not
	# Be aware that an improvement in log loss or Brier score on a test set is not

		# always guaranteed since the calibration set is finite and sometimes the base
		# classifier is already well calibrated.

	# Let's now do the same for various classifiers with different mis-calibration
	# Let's now repeat the above for various classifiers with different mis-calibration

		# - Some estimators such as highly regularized polynomial classifiers and
		# shallow tree-based models tend to be under-confident by default and each

		# TODO: once we have a calibration loss, use it instead of the
		# brier score to check recalibration.

	assert_allclose(np.sum(y_pred_cal, axis=1), np.ones(len(X_test)))
	assert_allclose(np.sum(y_pred_cal, axis=1), 1)


		When the response method is ``predict_proba``:

		- For ``method='sigmoid'``, Bernoulli logits are computed per class

	- For ``method='sigmoid'``, Bernoulli logits are computed per class
	- For ``method='sigmoid'``, logits are computed per class

		eps = xp.finfo(predictions.dtype).eps
		eps_ = xp.asarray(eps, dtype=predictions.dtype, device=device_)

	eps = xp.finfo(predictions.dtype).eps
	eps_ = xp.asarray(eps, dtype=predictions.dtype, device=device_)
	eps = xp.asarray(xp.finfo(predictions.dtype).eps, dtype=predictions.dtype, device=device_)

		)


		def _to_calibration_logits(predictions, *, response_method_name, method, classes=None):

Uh oh!

Uh oh!

Conversation

ogrisel commented Jun 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

antoinebaker commented Jun 19, 2026

Uh oh!

ogrisel commented Jun 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

antoinebaker commented Jun 25, 2026

predict_proba

decision_function

Comments

Uh oh!

ogrisel commented Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

ogrisel commented Jun 25, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

virchan left a comment

Choose a reason for hiding this comment

Uh oh!

ogrisel commented Aug 31, 2026

Uh oh!

ogrisel left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

lorentzenchr left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lorentzenchr left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ogrisel commented Jun 16, 2026 •

edited

Loading

ogrisel commented Jun 19, 2026 •

edited

Loading

`predict_proba`

`decision_function`

ogrisel commented Jun 25, 2026 •

edited

Loading

ogrisel left a comment •

edited

Loading

lorentzenchr left a comment •

edited

Loading