Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Commit 36ebf3e

Browse files
NicolasHugNicolasHug
authored
[MRG] Monotonic constraints for GBDT (#15582)
1 parent 942001a commit 36ebf3e

File tree

11 files changed

+874
-99
lines changed

11 files changed

+874
-99
lines changed

‎doc/modules/ensemble.rst

Lines changed: 46 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -897,7 +897,7 @@ based on permutation of the features.
897897
Histogram-Based Gradient Boosting
898898
=================================
899899

900-
Scikit-learn 0.21 introduces two new experimental implementations of
900+
Scikit-learn 0.21 introduced two new experimental implementations of
901901
gradient boosting trees, namely :class:`HistGradientBoostingClassifier`
902902
and :class:`HistGradientBoostingRegressor`, inspired by
903903
`LightGBM <https://github.com/Microsoft/LightGBM>`__ (See [LightGBM]_).
@@ -1050,6 +1050,51 @@ multiplying the gradients (and the hessians) by the sample weights. Note that
10501050
the binning stage (specifically the quantiles computation) does not take the
10511051
weights into account.
10521052

1053+
.. _monotonic_cst_gbdt:
1054+
1055+
Monotonic Constraints
1056+
---------------------
1057+
1058+
Depending on the problem at hand, you may have prior knowledge indicating
1059+
that a given feature should in general have a positive (or negative) effect
1060+
on the target value. For example, all else being equal, a higher credit
1061+
score should increase the probability of getting approved for a loan.
1062+
Monotonic constraints allow you to incorporate such prior knowledge into the
1063+
model.
1064+
1065+
A positive monotonic constraint is a constraint of the form:
1066+
1067+
:math:`x_1 \leq x_1' \implies F(x_1, x_2) \leq F(x_1', x_2)`,
1068+
where :math:`F` is the predictor with two features.
1069+
1070+
Similarly, a negative monotonic constraint is of the form:
1071+
1072+
:math:`x_1 \leq x_1' \implies F(x_1, x_2) \geq F(x_1', x_2)`.
1073+
1074+
Note that monotonic constraints only constraint the output "all else being
1075+
equal". Indeed, the following relation **is not enforced** by a positive
1076+
constraint: :math:`x_1 \leq x_1' \implies F(x_1, x_2) \leq F(x_1', x_2')`.
1077+
1078+
You can specify a monotonic constraint on each feature using the
1079+
`monotonic_cst` parameter. For each feature, a value of 0 indicates no
1080+
constraint, while -1 and 1 indicate a negative and positive constraint,
1081+
respectively::
1082+
1083+
>>> from sklearn.experimental import enable_hist_gradient_boosting # noqa
1084+
>>> from sklearn.ensemble import HistGradientBoostingRegressor
1085+
1086+
... # positive, negative, and no constraint on the 3 features
1087+
>>> gbdt = HistGradientBoostingRegressor(monotonic_cst=[1, -1, 0])
1088+
1089+
In a binary classification context, imposing a monotonic constraint means
1090+
that the feature is supposed to have a positive / negative effect on the
1091+
probability to belong to the positive class. Monotonic constraints are not
1092+
supported for multiclass context.
1093+
1094+
.. topic:: Examples:
1095+
1096+
* :ref:`sphx_glr_auto_examples_ensemble_plot_monotonic_constraints.py`
1097+
10531098
Low-level parallelism
10541099
---------------------
10551100

‎doc/whats_new/v0.23.rst

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -184,6 +184,11 @@ Changelog
184184
samples in the training set. :pr:`14516` by :user:`Johann Faouzi
185185
<johannfaouzi>`.
186186

187+
- |Feature| :class:`ensemble.HistGradientBoostingClassifier` and
188+
:class:`ensemble.HistGradientBoostingRegressor` now support monotonic
189+
constraints, useful when features are supposed to have a positive/negative
190+
effect on the target. :pr:`15582` by `Nicolas Hug`_.
191+
187192
- |Fix| Fixed a bug in :class:`ensemble.BaggingClassifier`,
188193
:class:`ensemble.BaggingRegressor` and :class:`ensemble.IsolationForest`
189194
where the attribute `estimators_samples_` did not generate the proper indices

‎examples/ensemble/plot_monotonic_constraints.py

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
"""
2+
=====================
3+
Monotonic Constraints
4+
=====================
5+
6+
This example illustrates the effect of monotonic constraints on a gradient
7+
boosting estimator.
8+
9+
We build an artificial dataset where the target value is in general
10+
positively correlated with the first feature (with some random and
11+
non-random variations), and in general negatively correlated with the second
12+
feature.
13+
14+
By imposing a positive (increasing) or negative (decreasing) constraint on
15+
the features during the learning process, the estimator is able to properly
16+
follow the general trend instead of being subject to the variations.
17+
18+
This example was inspired by the `XGBoost documentation
19+
<https://xgboost.readthedocs.io/en/latest/tutorials/monotonic.html>`_.
20+
"""
21+
from sklearn.experimental import enable_hist_gradient_boosting # noqa
22+
from sklearn.ensemble import HistGradientBoostingRegressor
23+
from sklearn.inspection import plot_partial_dependence
24+
import numpy as np
25+
import matplotlib.pyplot as plt
26+
27+
28+
print(__doc__)
29+
30+
rng = np.random.RandomState(0)
31+
32+
n_samples = 5000
33+
f_0 = rng.rand(n_samples) # positive correlation with y
34+
f_1 = rng.rand(n_samples) # negative correlation with y
35+
X = np.c_[f_0, f_1]
36+
noise = rng.normal(loc=0.0, scale=0.01, size=n_samples)
37+
y = (5 * f_0 + np.sin(10 * np.pi * f_0) -
38+
5 * f_1 - np.cos(10 * np.pi * f_1) +
39+
noise)
40+
41+
fig, ax = plt.subplots()
42+
43+
44+
# Without any constraint
45+
gbdt = HistGradientBoostingRegressor()
46+
gbdt.fit(X, y)
47+
disp = plot_partial_dependence(
48+
gbdt, X, features=[0, 1],
49+
line_kw={'linewidth': 4, 'label': 'unconstrained'},
50+
ax=ax)
51+
52+
# With positive and negative constraints
53+
gbdt = HistGradientBoostingRegressor(monotonic_cst=[1, -1])
54+
gbdt.fit(X, y)
55+
56+
plot_partial_dependence(
57+
gbdt, X, features=[0, 1],
58+
feature_names=('First feature\nPositive constraint',
59+
'Second feature\nNegtive constraint'),
60+
line_kw={'linewidth': 4, 'label': 'constrained'},
61+
ax=disp.axes_)
62+
63+
for f_idx in (0, 1):
64+
disp.axes_[0, f_idx].plot(X[:, f_idx], y, 'o', alpha=.3, zorder=-1)
65+
disp.axes_[0, f_idx].set_ylim(-6, 6)
66+
67+
plt.legend()
68+
fig.suptitle("Monotonic constraints illustration")
69+
70+
plt.show()

‎sklearn/ensemble/_hist_gradient_boosting/common.pxd

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,3 +30,9 @@ cdef packed struct node_struct:
3030
unsigned int depth
3131
unsigned char is_leaf
3232
X_BINNED_DTYPE_C bin_threshold
33+
34+
35+
cpdef enum MonotonicConstraint:
36+
NO_CST = 0
37+
POS = 1
38+
NEG = -1

‎sklearn/ensemble/_hist_gradient_boosting/gradient_boosting.py

Lines changed: 32 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -29,8 +29,9 @@ class BaseHistGradientBoosting(BaseEstimator, ABC):
2929
@abstractmethod
3030
def __init__(self, loss, learning_rate, max_iter, max_leaf_nodes,
3131
max_depth, min_samples_leaf, l2_regularization, max_bins,
32-
warm_start, early_stopping, scoring, validation_fraction,
33-
n_iter_no_change, tol, verbose, random_state):
32+
monotonic_cst, warm_start, early_stopping, scoring,
33+
validation_fraction, n_iter_no_change, tol, verbose,
34+
random_state):
3435
self.loss = loss
3536
self.learning_rate = learning_rate
3637
self.max_iter = max_iter
@@ -39,6 +40,7 @@ def __init__(self, loss, learning_rate, max_iter, max_leaf_nodes,
3940
self.min_samples_leaf = min_samples_leaf
4041
self.l2_regularization = l2_regularization
4142
self.max_bins = max_bins
43+
self.monotonic_cst = monotonic_cst
4244
self.warm_start = warm_start
4345
self.early_stopping = early_stopping
4446
self.scoring = scoring
@@ -82,6 +84,12 @@ def _validate_parameters(self):
8284
raise ValueError('max_bins={} should be no smaller than 2 '
8385
'and no larger than 255.'.format(self.max_bins))
8486

87+
if self.monotonic_cst is not None and self.n_trees_per_iteration_ != 1:
88+
raise ValueError(
89+
'monotonic constraints are not supported for '
90+
'multiclass classification.'
91+
)
92+
8593
def fit(self, X, y, sample_weight=None):
8694
"""Fit the gradient boosting model.
8795
@@ -352,12 +360,12 @@ def fit(self, X, y, sample_weight=None):
352360

353361
# Build `n_trees_per_iteration` trees.
354362
for k in range(self.n_trees_per_iteration_):
355-
356363
grower = TreeGrower(
357364
X_binned_train, gradients[k, :], hessians[k, :],
358365
n_bins=n_bins,
359366
n_bins_non_missing=self.bin_mapper_.n_bins_non_missing_,
360367
has_missing_values=has_missing_values,
368+
monotonic_cst=self.monotonic_cst,
361369
max_leaf_nodes=self.max_leaf_nodes,
362370
max_depth=self.max_depth,
363371
min_samples_leaf=self.min_samples_leaf,
@@ -790,6 +798,11 @@ class HistGradientBoostingRegressor(RegressorMixin, BaseHistGradientBoosting):
790798
Features with a small number of unique values may use less than
791799
``max_bins`` bins. In addition to the ``max_bins`` bins, one more bin
792800
is always reserved for missing values. Must be no larger than 255.
801+
monotonic_cst : array-like of int of shape (n_features), default=None
802+
Indicates the monotonic constraint to enforce on each feature. -1, 1
803+
and 0 respectively correspond to a positive constraint, negative
804+
constraint and no constraint. Read more in the :ref:`User Guide
805+
<monotonic_cst_gbdt>`.
793806
warm_start : bool, optional (default=False)
794807
When set to ``True``, reuse the solution of the previous call to fit
795808
and add more estimators to the ensemble. For results to be valid, the
@@ -867,16 +880,18 @@ class HistGradientBoostingRegressor(RegressorMixin, BaseHistGradientBoosting):
867880
def __init__(self, loss='least_squares', learning_rate=0.1,
868881
max_iter=100, max_leaf_nodes=31, max_depth=None,
869882
min_samples_leaf=20, l2_regularization=0., max_bins=255,
870-
warm_start=False, early_stopping='auto', scoring='loss',
871-
validation_fraction=0.1, n_iter_no_change=10, tol=1e-7,
883+
monotonic_cst=None, warm_start=False, early_stopping='auto',
884+
scoring='loss', validation_fraction=0.1,
885+
n_iter_no_change=10, tol=1e-7,
872886
verbose=0, random_state=None):
873887
super(HistGradientBoostingRegressor, self).__init__(
874888
loss=loss, learning_rate=learning_rate, max_iter=max_iter,
875889
max_leaf_nodes=max_leaf_nodes, max_depth=max_depth,
876890
min_samples_leaf=min_samples_leaf,
877891
l2_regularization=l2_regularization, max_bins=max_bins,
878-
warm_start=warm_start, early_stopping=early_stopping,
879-
scoring=scoring, validation_fraction=validation_fraction,
892+
monotonic_cst=monotonic_cst, early_stopping=early_stopping,
893+
warm_start=warm_start, scoring=scoring,
894+
validation_fraction=validation_fraction,
880895
n_iter_no_change=n_iter_no_change, tol=tol, verbose=verbose,
881896
random_state=random_state)
882897

@@ -978,6 +993,11 @@ class HistGradientBoostingClassifier(BaseHistGradientBoosting,
978993
Features with a small number of unique values may use less than
979994
``max_bins`` bins. In addition to the ``max_bins`` bins, one more bin
980995
is always reserved for missing values. Must be no larger than 255.
996+
monotonic_cst : array-like of int of shape (n_features), default=None
997+
Indicates the monotonic constraint to enforce on each feature. -1, 1
998+
and 0 respectively correspond to a positive constraint, negative
999+
constraint and no constraint. Read more in the :ref:`User Guide
1000+
<monotonic_cst_gbdt>`.
9811001
warm_start : bool, optional (default=False)
9821002
When set to ``True``, reuse the solution of the previous call to fit
9831003
and add more estimators to the ensemble. For results to be valid, the
@@ -1058,17 +1078,18 @@ class HistGradientBoostingClassifier(BaseHistGradientBoosting,
10581078

10591079
def __init__(self, loss='auto', learning_rate=0.1, max_iter=100,
10601080
max_leaf_nodes=31, max_depth=None, min_samples_leaf=20,
1061-
l2_regularization=0., max_bins=255, warm_start=False,
1062-
early_stopping='auto', scoring='loss',
1081+
l2_regularization=0., max_bins=255, monotonic_cst=None,
1082+
warm_start=False, early_stopping='auto', scoring='loss',
10631083
validation_fraction=0.1, n_iter_no_change=10, tol=1e-7,
10641084
verbose=0, random_state=None):
10651085
super(HistGradientBoostingClassifier, self).__init__(
10661086
loss=loss, learning_rate=learning_rate, max_iter=max_iter,
10671087
max_leaf_nodes=max_leaf_nodes, max_depth=max_depth,
10681088
min_samples_leaf=min_samples_leaf,
10691089
l2_regularization=l2_regularization, max_bins=max_bins,
1070-
warm_start=warm_start, early_stopping=early_stopping,
1071-
scoring=scoring, validation_fraction=validation_fraction,
1090+
monotonic_cst=monotonic_cst, warm_start=warm_start,
1091+
early_stopping=early_stopping, scoring=scoring,
1092+
validation_fraction=validation_fraction,
10721093
n_iter_no_change=n_iter_no_change, tol=tol, verbose=verbose,
10731094
random_state=random_state)
10741095

0 commit comments

Comments
 (0)