Do you use these because of computation time, or for a bias / variance tradeoff?

I suppose you could expect some speed ups. However,
Since the HistGradientBoosting estimator first bin the data, these parameters might not have the same effect regarding bias / variance

It was to prevent overfitting. I have read LightGBM has issues with overfitting so I thought this would be a good solution. If things are different in this method please explain.

Looks like LightGBM has bagging_fraction for subsampling, and feature_fraction for dealing with overfitting.

I think it's worth implementing, WDYT @ogrisel @adrinjalali?

I believe that if we implement both bagging_fraction and feature_fraction and set a low learning rate (and large number of boosting iterations) you can get boosted trees to behave more like random forests, hence can probably better deal with overfitting.

This is probably also interesting to get better calibration of the predict_proba output (in classification problems) which can be useful to report meaningful confidence levels when deploying the system.

Having the ability to resample each boosted tree independently could also be a very efficient way to deal with imbalanced classification problems: the majority class could be subsampled. This would warrant additional parameters though to deal with class-wise bagging fractions. Related to: #13227.

For the feature_fraction, is that on a tree-basis or a per-split basis? Our GradientBoosting does "max_features" which is on a per-split basis....

Doing it on per-split basis is the preferred method

I agree. My intuition is that if max_depth / max_leaf_nodes are high enough, the tree still has a chance to recover from a bad feature subset draw at a given split nodes while if we do it a tree-based level, the whole tree is wasted.

@DrEhrfurchtgebietend or anyone: for prosperity, is that also what the other libs do?

LightGBM uses a per-tree approach for max_features. https://github.com/microsoft/LightGBM/blob/master/src/treelearner/serial_tree_learner.cpp#L325

Is anyone working on the sampling part of HistGradientBoostingClassifier/Regressor? Apart from acceleration, sampling strategy also provides regularization on the entire GBDT system and improves its performance.

Here are some commonly-used sampling strategies on a tree-basis in other GBDT libraries:

• Stochastic Gradient Boosting (XGBoost/LightGBM/CatBoost)(https://astro.temple.edu/~msobel/courses_files/StochasticBoosting(gradient).pdf)
• Gradient-based One-Side Sampling (LightGBM)
  (https://papers.nips.cc/paper/6907-lightgbm-a-highly-efficient-gradient-boosting-decision-tree.pdf)
• Minimum Variance Sampling (CatBoost)
  (https://papers.nips.cc/paper/9645-minimal-variance-sampling-in-stochastic-gradient-boosting)

@gbolmier is currently working on feature and sample subsampling.

However the GOSS strategy from LightGBM is out of scope (at least for now) as it departs quite significantly from traditional GBDTs.

Doing it on per-split basis is the preferred method

@mayer79 Do you think the same for column/feature subsampling?

@lorentzenchr : Ideally, we could have both options. But "per split" is also my favourite. Given that it is not interferring too much with interaction constraints!

LightGBM has 2 feature_fraction alias colsample_bytree and feature_fraction_bynode alias colsample_bynode. The last one is for split-based subsampling.

XGBoost even has 3 different ones: colsample_bytree, colsample_bylevel, colsample_bynode. The last one is for split-base subsampling.

I propose we go with colsample_bynode, i.e. per-split subsampling.

Feature subsampling has been merged. Next step: sample-wise subsampling.

Also, it would be great if someone wanted to open a PR to randomize tie breaking when considering splits with equal newton steps. This could be done by shuffling the order of the features as suggested in #26428 or with flipping an RNG-coin each time an exact tie is detected when pushing to the heap queue.