Unrelated to this PR, but now that MAE criterion is at much 10x slower than the MSE, and usually more like 5x slower. Is it really significantly slower? It will still fit fairly fast for most tabular datasets (less than let's say 10M points).

I am inclined to remove this sentence. The reason for choosing either squared error or absolute error should not be fit time, but use case / statistically driven.

Let's remove.

Happy to 😄

I propose to delete this test, as the changes proposed by this PR are not exactly backward compatible, given that the criterion calculations now use a different but equivalent formula.

Another option would be to update it with the current values, but I feel that a such test prevents legitimate changes/improvements that slightly affect any calculation.

I wouldn't skip. What's the difference after the change in this test? What should be the tolerance for it to pass?

What should be the tolerance for it to pass?

rtol=0.1, which doesn't make much sense for a test named test_huber_exact_backward_compat

In this test, the model 100% overfits, so the values checked in the asserts are mostly 0 + some float precision noise I think 😅 , at least the one for which rtol=0.1 would be needed.

So I propose to transform this test into test_huber_overfit, see my commit 4009fd3

I wouldn't skip. What's the difference after the change in this test? What should be the tolerance for it to pass?

Is this change cause by rounding error discrepancies when switching from criterion="friedman_mse" to criterion="squared_error"? Or is this a consequence of #32707?

It's caused by rounding error discrepancies. #32707 only affects the scale of the impurity I think, see the fix I made temporarily in PR #32699, and here the scale of impurity shouldn't impact the results (trees are controlled by max_depth only).

But the calculation of impurity (criterion.impurity_improvement) being implemented differently, the rounding errors differ, and when you have several features with very similar (or the same) splits, the selected split might differ.

This happens for all the losses, except for loss="absolute_error": I think the gradient being -1 or 1 for this loss, there are no rounding errors (integers everywhere, and integers small enough to fit in the mantissa of a float64).

I think we should rather document this a fix and rephrase this entry accordingly, since this PR changes the default behavior of the estimators by changing the underlying implementation from the buggy "friedman_mse" implementation to the correct "squared_error" implementation (#32707).

Hum... I wrote both a .fix.rst and a .api.rst, each one focuses on a different aspect. Let me know what you think.

Please undo the changes of this test. I should still pass. If not, this PR introduces a regression.

This test doesn't pass after the changes in this PR because of round errors. As mentioned above, in this test (test_huber_exact_backward_compat) the model 100% overfits, so the values checked in the asserts are mostly 0/<some_int> + some float precision noise. Checking the exact values means checking the exact float precision noise, preventing almost any change, even meaningful and valid, from passing it.

If not, this PR introduces a regression

No. This test was just not a meaningful test (as I already found and fixed many since I started working on scikit-learn...).

Typically, this test fails when you change:

median = _weighted_percentile(y_true, sample_weight, 50) to median = _weighted_percentile(y_true, sample_weight, 50, average=True) in HuberLoss.fit_intercept_only. Even though I think we consider that average=True is a better option for computing the weighted median. At least that's what I assumed when I wrote the PR #32100 and was, in a sense, confirmed by @ogrisel who asked me to test my logic against _weighted_percentile(..., average=True).

I agree that we will need to open another PR for to use average=True in. HuberLoss.fit_intercept_only and check that optimizing the HuberLoss on a dataset with symmetrically distributed target data and constant features returns the same as np.median, whether the sum of integer sample weights is even or odd and remove this arbitrary bias.

I'd be happy to do that 😄 We have the same thing for the losses AbsoluteError and PinballLoss.

Can this test be modified instead of removed? Or is it tested elsewhere?

This test was mostly testing that the string "friedman_mse" was present in appropriate places in the output. We have some much more complete tests in this file (for the squared error criterion). So I think it's fine removing it.

Alternatively, I can keep this test and just ignore the deprecation warnings. And we'll remove it when we remove "friedman_mse" completely.

I would rename the test to test_criterion_in_gradient_boosting_graphviz and check that the name of the new criterion ("squared_error") is present in the nodes.

I think this would be redundant with test_graphviz_toy where we compare with exact outputs (including examples including "squared_error".

Edit: ah no, this test fits GradientBoostingClassifier, ok let's do what you suggest.

Though TBH, I don't think this test makes a lot of sense: why testing the export of GB.estimators_[0] and not RF.estimators_[0], ET.estimators_[0], etc.?

I would rather suggest a test (parametrized by the criterion) that fits a decision tree, and tests the presence of the criterion name in the exported string. So we test all criteria. And we avoid importing from sklearn.ensemble in a test from sklearn.tree which looks like a bad pattern in most cases.