[MRG] Add verbose option to VotingClassifier - #10974
Conversation
| verbose: int, optional | ||
| The verbosity level: if non zero, progress messages are printed. | ||
| Above 50, the output is sent to stdout. The frequency of the messages | ||
| increases with the verbosity level. If it more than 10, all iterations |
There was a problem hiding this comment.
"iterations" does not really apply here.
How many estimators does one need to have in a VotingClassifier for anything lower than 10 to be helpful here?
There was a problem hiding this comment.
Good point! Thanks a lot for your review.
I replace "iterations" by "fit calls" in fcbfbcd.
I would say that a value lower than 10 is useful when:
- the number of estimator is big (let say 500)
- each estimator is pretty quick to fit (let say 1 second)
- n_jobs is strictly greater than 1
I'm thinking of a situation similar to this call:
from time import sleep
from sklearn.externals.joblib import Parallel, delayed
r = Parallel(n_jobs=2, verbose=5)(delayed(sleep)(1.) for _ in range(500))In this case, the user may want to have feedback to know that (s)he has 4.16 minutes to get a coffee (500 / 2 * 1 / 60), but without being spammed by 500 lines. Does it make sense?
|
is VotingClassifier often used for that number of estimators? that's really
what I was wondering.
|
|
I don't think it is, except maybe for some isolated cases. Do you suggest that the
|
|
@jnothman I think you're right. In a second thought, I suggest another solution, implemented by 3fa6b91: a boolean value should be passed to
I think it's simpler and most suitable to the use of |
|
sounds like a good start, certainly!
|
jnothman
left a comment
There was a problem hiding this comment.
Could you please provide a sample output to remind me what Parallel's verbose output looks like? Thanks.
|
Please add an entry to the change log at |
|
Thanks a lot for your review. I have added this change to
Sure. Please see the snippet below: >>> import numpy as np
>>> from sklearn.ensemble import VotingClassifier
>>> from sklearn.linear_model import LogisticRegression
>>> from sklearn.ensemble import RandomForestClassifier
>>> from sklearn.naive_bayes import GaussianNB
>>> clf1 = LogisticRegression(random_state=123)
>>> clf2 = RandomForestClassifier(random_state=123)
>>> clf3 = GaussianNB()
>>> X = np.array([[-1.1, -1.5], [-1.2, -1.4], [-3.4, -2.2], [1.1, 1.2]])
>>> y = np.array([1, 1, 2, 2])
>>> VotingClassifier(estimators=[
... ('lr', clf1), ('rf', clf2), ('gnb', clf3)],
... voting='soft', verbose=True).fit(X, y)
[Parallel(n_jobs=1)]: Done 1 out of 1 | elapsed: 0.0s remaining: 0.0s
[Parallel(n_jobs=1)]: Done 2 out of 2 | elapsed: 0.0s remaining: 0.0s
[Parallel(n_jobs=1)]: Done 3 out of 3 | elapsed: 0.0s remaining: 0.0s
[Parallel(n_jobs=1)]: Done 3 out of 3 | elapsed: 0.0s finished |
| the sums of the predicted probabilities, which is recommended for | ||
| an ensemble of well-calibrated classifiers. | ||
|
|
||
| verbose : bool, optional (default=False) |
There was a problem hiding this comment.
why not use an int here? I find it weird to remove a level of control here.
There was a problem hiding this comment.
@jnothman was suggesting in previous comments (see 1 and 2) that VotingClassifier isn't used for a huge number of estimators. And I think that the int level of control is useless if the number of estimators is smaller than, let's say, 50. But it's really helpful for 500 classifiers. Do you think that it is used for large amounts of classifiers? And that we should keep it as int?
There was a problem hiding this comment.
I'm not sure what the number of estimators has to do with having an int vs a bool. An int allows us to change the reporting level and add more details at higher levels at any point.
There was a problem hiding this comment.
For few estimators, if the int is only passed to the verbose option of Parallel, the difference will be minor between having bool or int. For example, see the difference between the verbose options in the following snippet:
>>> from time import sleep
>>> from sklearn.externals.joblib import Parallel, delayed
>>> r = Parallel(n_jobs=2, verbose=1)(delayed(sleep)(1.) for _ in range(10))
[Parallel(n_jobs=2)]: Done 10 out of 10 | elapsed: 5.0s finished
>>> r = Parallel(n_jobs=2, verbose=5)(delayed(sleep)(1.) for _ in range(10))
[Parallel(n_jobs=2)]: Done 10 out of 10 | elapsed: 5.0s remaining: 0.0s
[Parallel(n_jobs=2)]: Done 10 out of 10 | elapsed: 5.0s finished
>>> r = Parallel(n_jobs=2, verbose=10)(delayed(sleep)(1.) for _ in range(10))
[Parallel(n_jobs=2)]: Done 1 tasks | elapsed: 1.0s
[Parallel(n_jobs=2)]: Done 4 tasks | elapsed: 2.0s
[Parallel(n_jobs=2)]: Done 10 out of 10 | elapsed: 5.0s finished
# value used in the MR if the boolean value is True:
>>> r = Parallel(n_jobs=2, verbose=100)(delayed(sleep)(1.) for _ in range(10))
[Parallel(n_jobs=2)]: Done 1 tasks | elapsed: 1.0s
[Parallel(n_jobs=2)]: Done 2 tasks | elapsed: 1.0s
[Parallel(n_jobs=2)]: Done 3 tasks | elapsed: 2.0s
[Parallel(n_jobs=2)]: Done 4 tasks | elapsed: 2.0s
[Parallel(n_jobs=2)]: Done 5 tasks | elapsed: 3.0s
[Parallel(n_jobs=2)]: Done 6 tasks | elapsed: 3.0s
[Parallel(n_jobs=2)]: Done 7 tasks | elapsed: 4.0s
[Parallel(n_jobs=2)]: Done 8 out of 10 | elapsed: 4.0s remaining: 1.0s
[Parallel(n_jobs=2)]: Done 10 out of 10 | elapsed: 5.0s remaining: 0.0s
[Parallel(n_jobs=2)]: Done 10 out of 10 | elapsed: 5.0s finishedIf there is a high number of classifiers, having higher control is interesting to have the first output (few messages) instead of the second one (which spams stdout):
>>> r = Parallel(n_jobs=2, verbose=5)(delayed(sleep)(.3) for _ in range(500))
[Parallel(n_jobs=2)]: Done 14 tasks | elapsed: 2.1s
[Parallel(n_jobs=2)]: Done 68 tasks | elapsed: 10.2s
[Parallel(n_jobs=2)]: Done 158 tasks | elapsed: 23.7s
[Parallel(n_jobs=2)]: Done 284 tasks | elapsed: 42.7s
[Parallel(n_jobs=2)]: Done 446 tasks | elapsed: 1.1min
[Parallel(n_jobs=2)]: Done 500 out of 500 | elapsed: 1.3min finished
>>> r = Parallel(n_jobs=2, verbose=100)(delayed(sleep)(.3) for _ in range(500))
[Parallel(n_jobs=2)]: Done 1 tasks | elapsed: 0.3s
[Parallel(n_jobs=2)]: Done 2 tasks | elapsed: 0.3s
[Parallel(n_jobs=2)]: Done 3 tasks | elapsed: 0.6s
[Parallel(n_jobs=2)]: Done 4 tasks | elapsed: 0.6s
[Parallel(n_jobs=2)]: Done 5 tasks | elapsed: 0.9s
[Parallel(n_jobs=2)]: Done 6 tasks | elapsed: 0.9s
[Parallel(n_jobs=2)]: Done 7 tasks | elapsed: 1.2s
[Parallel(n_jobs=2)]: Done 8 tasks | elapsed: 1.2s
[Parallel(n_jobs=2)]: Done 9 tasks | elapsed: 1.5s
[Parallel(n_jobs=2)]: Done 10 tasks | elapsed: 1.5s
[Parallel(n_jobs=2)]: Done 11 tasks | elapsed: 1.8s
[...]
[Parallel(n_jobs=2)]: Done 495 tasks | elapsed: 1.2min
[Parallel(n_jobs=2)]: Done 496 tasks | elapsed: 1.2min
[Parallel(n_jobs=2)]: Done 497 tasks | elapsed: 1.2min
[Parallel(n_jobs=2)]: Done 500 out of 500 | elapsed: 1.3min finishedDo you suggest that having an int option is better in order to control the verbosity level somewhere else than in the Parallel call?
There was a problem hiding this comment.
yes. the estimator itself can have different verbosity levels
|
are you still interested in finishing this up? |
|
Closed by #16069 |
Reference Issues/PRs
Fixes #10360
Closes #10367 #10365
What does this implement/fix? Explain your changes.
verboseoption toVotingClassifier. If True,100is passed to theverboseoption ofParallelin order to print all progress messages to stdout.VotingClassifier.False, andTrue.Any other comments?
I used the
capfdpytest fixture in my unit test. Is it ok? Is there a particular reason for not using it in other verbose tests (found withgit grep test_.*verbose)? Because it seems quite convenient.