Does the proxy_impurity_improvement return value need to be scaled by number of instances? Not exactly sure how sklearn operates, but you might want to check this scaled correctly so that any 'min_impurity' type checks are consistent with the loss function.

@glemaitre @thomasjpfan Friendly ping.

For the Poisson case, the weighted_n_* gets canceled out.
Is this your understanding of the bug as well?

Yes, the weights of the (candidate) left and right nodes get cancelled out in the lines

Example: impurity = MSE = 1/n sum_i (y_i - y_pred_i)**2

• For a single tree node, y_pred_i = const = y_mean = 1/n sum_i y_i and therefore MSE = 1/n sum_i (y_i - y_mean)**2 = var(y)
• If one divides the node into left and right, one has different values for y_pred (left/right) and so the MSE of the parent can be rewritten as MSE(parent) = 1/n * (n_L * MSE(left) + n_R * MSE(right)) which can then be further simplified. The point is that n_L * MSE = sum of squares instead of mean of squares.

I also see a comment at the beginning of the class regarding the proxy computation. Do you want to remove it then?

For information, I tried to test the impact of this PR on the poisson regression example : https://scikit-learn.org/stable/auto_examples/linear_model/plot_poisson_regression_non_normal_loss.html

I defined a poisson_rf model, similar to the poisson_gbrt` pipeline but with:

from sklearn.ensemble import RandomForestRegressor


poisson_rf = Pipeline(
    [
        ("preprocessor", tree_preprocessor),
        (
            "regressor",
            RandomForestRegressor(criterion="poisson", min_samples_leaf=10, n_jobs=-1),
        ),
    ]
)
poisson_rf.fit(
    df_train, df_train["Frequency"], regressor__sample_weight=df_train["Exposure"]
)

print("Poisson Random Forest evaluation:")
score_estimator(poisson_rf, df_test)

and then computed the deviance when training on 10% of the data (due to longer training times of RF compared to linear models and GBDT) and I found that:

• a this PR can improve the Poisson deviance a bit (compared to main) but not by much (not sure if it's in the statistical noise);
• b Poisson RF cannot compete with Poisson GBRT nor even linear Poisson regression on this data (or even Ridge regression), both in terms of Poisson deviance or Gini....
• c MSE RF is not necessarily worse than Poisson RF in terms of Poisson deviance...

I find b a bit worrying but maybe this is expected?

Disclaimer: I did not run a full hyper-parameter optimization for RF models. In particular, it's possible that the RF models would perform better with more trees (but those are slow...).

Edit: I made a mistake in my first batch of experiments where I forgot to recompile when switching branch... but after re-compiling, the previous comments still mostly hold.

But in any case, I made the following surprising observation with RFs: on this example, the following RF:

            RandomForestRegressor(
                criterion="poisson", n_estimators=5, min_samples_leaf=1000, n_jobs=-1
            )

performs as good as the same RF with hundreds of trees and much better than deep RF with lower values for min_samples_leaf. With those hyperparameters, RF are competitive with linear models and GBRT.

I wonder if a simple averaging the y_hats in RF is optimal for Poisson regression.

I confirm that this PR fixes the case originally reported as #22186 (comment):