Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Commit 8edd9f9

Browse files
thomasjpfanjnothman
thomasjpfan
authored and
jnothman
committed
ENH Add verbose option to Pipeline, FeatureUnion, and ColumnTransformer (#11364)
1 parent 58927cf commit 8edd9f9

12 files changed

Lines changed: 429 additions & 77 deletions

File tree

‎doc/modules/compose.rst‎

Lines changed: 14 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -60,7 +60,7 @@ is an estimator object::
6060
>>> pipe # doctest: +NORMALIZE_WHITESPACE, +ELLIPSIS
6161
Pipeline(memory=None,
6262
steps=[('reduce_dim', PCA(copy=True,...)),
63-
('clf', SVC(C=1.0,...))])
63+
('clf', SVC(C=1.0,...))], verbose=False)
6464

6565
The utility function :func:`make_pipeline` is a shorthand
6666
for constructing pipelines;
@@ -75,7 +75,8 @@ filling in the names automatically::
7575
steps=[('binarizer', Binarizer(copy=True, threshold=0.0)),
7676
('multinomialnb', MultinomialNB(alpha=1.0,
7777
class_prior=None,
78-
fit_prior=True))])
78+
fit_prior=True))],
79+
verbose=False)
7980

8081
Accessing steps
8182
...............
@@ -106,9 +107,9 @@ permitted). This is convenient for performing only some of the transformations
106107
(or their inverse):
107108

108109
>>> pipe[:1] # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS
109-
Pipeline(memory=None, steps=[('reduce_dim', PCA(copy=True, ...))])
110+
Pipeline(memory=None, steps=[('reduce_dim', PCA(copy=True, ...))],...)
110111
>>> pipe[-1:] # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS
111-
Pipeline(memory=None, steps=[('clf', SVC(C=1.0, ...))])
112+
Pipeline(memory=None, steps=[('clf', SVC(C=1.0, ...))],...)
112113

113114
Nested parameters
114115
.................
@@ -119,7 +120,8 @@ Parameters of the estimators in the pipeline can be accessed using the
119120
>>> pipe.set_params(clf__C=10) # doctest: +NORMALIZE_WHITESPACE, +ELLIPSIS
120121
Pipeline(memory=None,
121122
steps=[('reduce_dim', PCA(copy=True, iterated_power='auto',...)),
122-
('clf', SVC(C=10, cache_size=200, class_weight=None,...))])
123+
('clf', SVC(C=10, cache_size=200, class_weight=None,...))],
124+
verbose=False)
123125

124126
This is particularly important for doing grid searches::
125127

@@ -202,7 +204,7 @@ object::
202204
>>> pipe # doctest: +NORMALIZE_WHITESPACE, +ELLIPSIS
203205
Pipeline(...,
204206
steps=[('reduce_dim', PCA(copy=True,...)),
205-
('clf', SVC(C=1.0,...))])
207+
('clf', SVC(C=1.0,...))], verbose=False)
206208
>>> # Clear the cache directory when you don't need it anymore
207209
>>> rmtree(cachedir)
208210

@@ -219,7 +221,8 @@ object::
219221
>>> pipe.fit(digits.data, digits.target)
220222
... # doctest: +NORMALIZE_WHITESPACE, +ELLIPSIS
221223
Pipeline(memory=None,
222-
steps=[('reduce_dim', PCA(...)), ('clf', SVC(...))])
224+
steps=[('reduce_dim', PCA(...)), ('clf', SVC(...))],
225+
verbose=False)
223226
>>> # The pca instance can be inspected directly
224227
>>> print(pca1.components_) # doctest: +NORMALIZE_WHITESPACE, +ELLIPSIS
225228
[[-1.77484909e-19 ... 4.07058917e-18]]
@@ -241,7 +244,8 @@ object::
241244
>>> cached_pipe.fit(digits.data, digits.target)
242245
... # doctest: +NORMALIZE_WHITESPACE, +ELLIPSIS
243246
Pipeline(memory=...,
244-
steps=[('reduce_dim', PCA(...)), ('clf', SVC(...))])
247+
steps=[('reduce_dim', PCA(...)), ('clf', SVC(...))],
248+
verbose=False)
245249
>>> print(cached_pipe.named_steps['reduce_dim'].components_)
246250
... # doctest: +NORMALIZE_WHITESPACE, +ELLIPSIS
247251
[[-1.77484909e-19 ... 4.07058917e-18]]
@@ -376,7 +380,7 @@ and ``value`` is an estimator object::
376380
FeatureUnion(n_jobs=None,
377381
transformer_list=[('linear_pca', PCA(copy=True,...)),
378382
('kernel_pca', KernelPCA(alpha=1.0,...))],
379-
transformer_weights=None)
383+
transformer_weights=None, verbose=False)
380384

381385

382386
Like pipelines, feature unions have a shorthand constructor called
@@ -391,7 +395,7 @@ and ignored by setting to ``'drop'``::
391395
FeatureUnion(n_jobs=None,
392396
transformer_list=[('linear_pca', PCA(copy=True,...)),
393397
('kernel_pca', 'drop')],
394-
transformer_weights=None)
398+
transformer_weights=None, verbose=False)
395399

396400
.. topic:: Examples:
397401

‎doc/whats_new/v0.21.rst‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -565,6 +565,13 @@ Support for Python 3.4 and below has been officially dropped.
565565
therefore ``len(pipeline)`` returns the number of steps in the pipeline.
566566
:issue:`13439` by :user:`Lakshya KD <LakshKD>`.
567567

568+
- |Feature| Added optional parameter ``verbose`` in :class:`pipeline.Pipeline`,
569+
:class:`compose.ColumnTransformer` and :class:`pipeline.FeatureUnion`
570+
and corresponding ``make_`` helpers for showing progress and timing of
571+
each step. :issue:`11364` by :user:`Baze Petrushev <petrushev>`,
572+
:user:`Karan Desai <karandesai-96>`, `Joel Nothman`_, and
573+
:user:`Thomas Fan <thomasjpfan>`.
574+
568575
:mod:`sklearn.preprocessing`
569576
............................
570577

‎examples/compose/plot_column_transformer.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -117,7 +117,7 @@ def transform(self, posts):
117117

118118
# Use a SVC classifier on the combined features
119119
('svc', LinearSVC()),
120-
])
120+
], verbose=True)
121121

122122
# limit the list of categories to make running this example faster.
123123
categories = ['alt.atheism', 'talk.religion.misc']

‎sklearn/compose/_column_transformer.py‎

Lines changed: 36 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -101,6 +101,10 @@ class ColumnTransformer(_BaseComposition, TransformerMixin):
101101
transformer is multiplied by these weights. Keys are transformer names,
102102
values the weights.
103103
104+
verbose : boolean, optional(default=False)
105+
If True, the time elapsed while fitting each transformer will be
106+
printed as it is completed.
107+
104108
Attributes
105109
----------
106110
transformers_ : list
@@ -160,13 +164,19 @@ class ColumnTransformer(_BaseComposition, TransformerMixin):
160164
"""
161165
_required_parameters = ['transformers']
162166

163-
def __init__(self, transformers, remainder='drop', sparse_threshold=0.3,
164-
n_jobs=None, transformer_weights=None):
167+
def __init__(self,
168+
transformers,
169+
remainder='drop',
170+
sparse_threshold=0.3,
171+
n_jobs=None,
172+
transformer_weights=None,
173+
verbose=False):
165174
self.transformers = transformers
166175
self.remainder = remainder
167176
self.sparse_threshold = sparse_threshold
168177
self.n_jobs = n_jobs
169178
self.transformer_weights = transformer_weights
179+
self.verbose = verbose
170180

171181
@property
172182
def _transformers(self):
@@ -377,6 +387,11 @@ def _validate_output(self, result):
377387
"The output of the '{0}' transformer should be 2D (scipy "
378388
"matrix, array, or pandas DataFrame).".format(name))
379389

390+
def _log_message(self, name, idx, total):
391+
if not self.verbose:
392+
return None
393+
return '(%d of %d) Processing %s' % (idx, total, name)
394+
380395
def _fit_transform(self, X, y, func, fitted=False):
381396
"""
382397
Private function to fit and/or transform on demand.
@@ -385,12 +400,19 @@ def _fit_transform(self, X, y, func, fitted=False):
385400
on the passed function.
386401
``fitted=True`` ensures the fitted transformers are used.
387402
"""
403+
transformers = list(
404+
self._iter(fitted=fitted, replace_strings=True))
388405
try:
389406
return Parallel(n_jobs=self.n_jobs)(
390-
delayed(func)(clone(trans) if not fitted else trans,
391-
_get_column(X, column), y, weight)
392-
for _, trans, column, weight in self._iter(
393-
fitted=fitted, replace_strings=True))
407+
delayed(func)(
408+
transformer=clone(trans) if not fitted else trans,
409+
X=_get_column(X, column),
410+
y=y,
411+
weight=weight,
412+
message_clsname='ColumnTransformer',
413+
message=self._log_message(name, idx, len(transformers)))
414+
for idx, (name, trans, column, weight) in enumerate(
415+
self._iter(fitted=fitted, replace_strings=True), 1))
394416
except ValueError as e:
395417
if "Expected 2D array, got 1D array instead" in str(e):
396418
raise ValueError(_ERR_MSG_1DCOLUMN)
@@ -775,6 +797,10 @@ def make_column_transformer(*transformers, **kwargs):
775797
``-1`` means using all processors. See :term:`Glossary <n_jobs>`
776798
for more details.
777799
800+
verbose : boolean, optional(default=False)
801+
If True, the time elapsed while fitting each transformer will be
802+
printed as it is completed.
803+
778804
Returns
779805
-------
780806
ct : ColumnTransformer
@@ -800,18 +826,20 @@ def make_column_transformer(*transformers, **kwargs):
800826
['numerical_column']),
801827
('onehotencoder',
802828
OneHotEncoder(...),
803-
['categorical_column'])])
829+
['categorical_column'])], verbose=False)
804830
805831
"""
806832
# transformer_weights keyword is not passed through because the user
807833
# would need to know the automatically generated names of the transformers
808834
n_jobs = kwargs.pop('n_jobs', None)
809835
remainder = kwargs.pop('remainder', 'drop')
810836
sparse_threshold = kwargs.pop('sparse_threshold', 0.3)
837+
verbose = kwargs.pop('verbose', False)
811838
if kwargs:
812839
raise TypeError('Unknown keyword arguments: "{}"'
813840
.format(list(kwargs.keys())[0]))
814841
transformer_list = _get_transformer_list(transformers)
815842
return ColumnTransformer(transformer_list, n_jobs=n_jobs,
816843
remainder=remainder,
817-
sparse_threshold=sparse_threshold)
844+
sparse_threshold=sparse_threshold,
845+
verbose=verbose)

‎sklearn/compose/tests/test_column_transformer.py‎

Lines changed: 59 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
"""
22
Test the ColumnTransformer.
33
"""
4+
import re
45

56
import numpy as np
67
from scipy import sparse
@@ -596,7 +597,8 @@ def test_column_transformer_get_set_params():
596597
'trans2__with_mean': True,
597598
'trans2__with_std': True,
598599
'transformers': ct.transformers,
599-
'transformer_weights': None}
600+
'transformer_weights': None,
601+
'verbose': False}
600602

601603
assert_dict_equal(ct.get_params(), exp)
602604

@@ -613,7 +615,8 @@ def test_column_transformer_get_set_params():
613615
'trans2__with_mean': True,
614616
'trans2__with_std': True,
615617
'transformers': ct.transformers,
616-
'transformer_weights': None}
618+
'transformer_weights': None,
619+
'verbose': False}
617620

618621
assert_dict_equal(ct.get_params(), exp)
619622

@@ -944,7 +947,8 @@ def test_column_transformer_get_set_params_with_remainder():
944947
'trans1__with_mean': True,
945948
'trans1__with_std': True,
946949
'transformers': ct.transformers,
947-
'transformer_weights': None}
950+
'transformer_weights': None,
951+
'verbose': False}
948952

949953
assert ct.get_params() == exp
950954

@@ -960,7 +964,8 @@ def test_column_transformer_get_set_params_with_remainder():
960964
'sparse_threshold': 0.3,
961965
'trans1': 'passthrough',
962966
'transformers': ct.transformers,
963-
'transformer_weights': None}
967+
'transformer_weights': None,
968+
'verbose': False}
964969

965970
assert ct.get_params() == exp
966971

@@ -981,6 +986,56 @@ def test_column_transformer_no_estimators():
981986
assert ct.transformers_[-1][2] == [0, 1, 2]
982987

983988

989+
@pytest.mark.parametrize(
990+
['est', 'pattern'],
991+
[(ColumnTransformer([('trans1', Trans(), [0]), ('trans2', Trans(), [1])],
992+
remainder=DoubleTrans()),
993+
(r'\[ColumnTransformer\].*\(1 of 3\) Processing trans1.* total=.*\n'
994+
r'\[ColumnTransformer\].*\(2 of 3\) Processing trans2.* total=.*\n'
995+
r'\[ColumnTransformer\].*\(3 of 3\) Processing remainder.* total=.*\n$'
996+
)),
997+
(ColumnTransformer([('trans1', Trans(), [0]), ('trans2', Trans(), [1])],
998+
remainder='passthrough'),
999+
(r'\[ColumnTransformer\].*\(1 of 3\) Processing trans1.* total=.*\n'
1000+
r'\[ColumnTransformer\].*\(2 of 3\) Processing trans2.* total=.*\n'
1001+
r'\[ColumnTransformer\].*\(3 of 3\) Processing remainder.* total=.*\n$'
1002+
)),
1003+
(ColumnTransformer([('trans1', Trans(), [0]), ('trans2', 'drop', [1])],
1004+
remainder='passthrough'),
1005+
(r'\[ColumnTransformer\].*\(1 of 2\) Processing trans1.* total=.*\n'
1006+
r'\[ColumnTransformer\].*\(2 of 2\) Processing remainder.* total=.*\n$'
1007+
)),
1008+
(ColumnTransformer([('trans1', Trans(), [0]),
1009+
('trans2', 'passthrough', [1])],
1010+
remainder='passthrough'),
1011+
(r'\[ColumnTransformer\].*\(1 of 3\) Processing trans1.* total=.*\n'
1012+
r'\[ColumnTransformer\].*\(2 of 3\) Processing trans2.* total=.*\n'
1013+
r'\[ColumnTransformer\].*\(3 of 3\) Processing remainder.* total=.*\n$'
1014+
)),
1015+
(ColumnTransformer([('trans1', Trans(), [0])], remainder='passthrough'),
1016+
(r'\[ColumnTransformer\].*\(1 of 2\) Processing trans1.* total=.*\n'
1017+
r'\[ColumnTransformer\].*\(2 of 2\) Processing remainder.* total=.*\n$'
1018+
)),
1019+
(ColumnTransformer([('trans1', Trans(), [0]), ('trans2', Trans(), [1])],
1020+
remainder='drop'),
1021+
(r'\[ColumnTransformer\].*\(1 of 2\) Processing trans1.* total=.*\n'
1022+
r'\[ColumnTransformer\].*\(2 of 2\) Processing trans2.* total=.*\n$')),
1023+
(ColumnTransformer([('trans1', Trans(), [0])], remainder='drop'),
1024+
(r'\[ColumnTransformer\].*\(1 of 1\) Processing trans1.* total=.*\n$'))])
1025+
@pytest.mark.parametrize('method', ['fit', 'fit_transform'])
1026+
def test_column_transformer_verbose(est, pattern, method, capsys):
1027+
X_array = np.array([[0, 1, 2], [2, 4, 6], [8, 6, 4]]).T
1028+
1029+
func = getattr(est, method)
1030+
est.set_params(verbose=False)
1031+
func(X_array)
1032+
assert not capsys.readouterr().out, 'Got output for verbose=False'
1033+
1034+
est.set_params(verbose=True)
1035+
func(X_array)
1036+
assert re.match(pattern, capsys.readouterr()[0])
1037+
1038+
9841039
def test_column_transformer_no_estimators_set_params():
9851040
ct = ColumnTransformer([]).set_params(n_jobs=2)
9861041
assert ct.n_jobs == 2

‎sklearn/model_selection/_validation.py‎

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -19,11 +19,11 @@
1919
import scipy.sparse as sp
2020

2121
from ..base import is_classifier, clone
22-
from ..utils import indexable, check_random_state, safe_indexing
22+
from ..utils import (indexable, check_random_state, safe_indexing,
23+
_message_with_time)
2324
from ..utils.validation import _is_arraylike, _num_samples
2425
from ..utils.metaestimators import _safe_split
2526
from ..utils._joblib import Parallel, delayed
26-
from ..utils._joblib import logger
2727
from ..metrics.scorer import check_scoring, _check_multimetric_scoring
2828
from ..exceptions import FitFailedWarning
2929
from ._split import check_cv
@@ -572,8 +572,7 @@ def _fit_and_score(estimator, X, y, scorer, train, test, verbose,
572572

573573
if verbose > 1:
574574
total_time = score_time + fit_time
575-
end_msg = "%s, total=%s" % (msg, logger.short_format_time(total_time))
576-
print("[CV] %s %s" % ((64 - len(end_msg)) * '.', end_msg))
575+
print(_message_with_time('CV', msg, total_time))
577576

578577
ret = [train_scores, test_scores] if return_train_score else [test_scores]
579578

‎sklearn/model_selection/tests/test_validation.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -189,7 +189,7 @@ def fit(self, X, Y=None, sample_weight=None, class_prior=None,
189189
raise ValueError('X cannot be d')
190190
if sample_weight is not None:
191191
assert sample_weight.shape[0] == X.shape[0], (
192-
'MockClassifier extra fit_param '
192+
'MockClassifier extra fit_param '
193193
'sample_weight.shape[0] is {0}, should be {1}'
194194
.format(sample_weight.shape[0], X.shape[0]))
195195
if class_prior is not None:

0 commit comments

Comments
 (0)