[MRG+1] Raise warning in scikit-learn/sklearn/linear_model/cd_fast.pyx for cases when the main loop exits without reaching the desired tolerance #11754
Conversation
| n_classes = 2 | ||
| X = np.ones([n_samples, n_features]) * 1e50 | ||
| y = np.ones([n_samples, n_classes]) | ||
| assert_warns(ConvergenceWarning, clf.fit, X, y) |
There was a problem hiding this comment.
to test this use tiny data and set max_iter to a very tiny number. it will make testing faster.
besides this is already done in the estimator eg:
why is it not enough for you? bug?
There was a problem hiding this comment.
For the cases identified in #10813, the estimator raises no warning. Essentially, there are numerical issues causing the duality gap and tolerance to be equal to zero - as such the warning won't be raised in the estimator.
sklearn/linear_model/cd_fast.pyx
Outdated
| else: | ||
| with gil: | ||
| warnings.warn("Objective did not converge." | ||
| " You might want to increase the number of iterations.", |
There was a problem hiding this comment.
Can we include the desired and the achieved tolerance in the warning message?
| clf = Lasso(precompute=True) | ||
| n_samples = 15500 | ||
| n_features = 500 | ||
| X = np.ones([n_samples, n_features]) * 1e50 |
There was a problem hiding this comment.
in np.ones(shape), typically shape is a tuple not a list.
sklearn/linear_model/cd_fast.pyx
Outdated
| @@ -302,6 +303,13 @@ def enet_coordinate_descent(np.ndarray[floating, ndim=1] w, | |||
| if gap < tol: | |||
| # return if we reached desired tolerance | |||
| break | |||
| else: | |||
There was a problem hiding this comment.
Shouldn't this be an else clause of the for loop, not of the if statement?
Please add a test that no warning is raised if the optimisation reaches convergence
There was a problem hiding this comment.
Since the function returns gap, can't we do this outside of the function??
There was a problem hiding this comment.
Issue #10813 suggested raising the warning when the max number of iterations is reached and the desired tolerance has yet to be achieved. It's not obvious to me how to test for that outside of the function because it doesn't return max_iter.
Should this be changed to look for instances where gap and tolerance are both equal to zero - indicating possible numerical issues?
There was a problem hiding this comment.
max_iter is passed in, so surely it is available to the caller test
|
Please rename this PR to describe what it is actually changing. It is not doing what the title says |
There was a problem hiding this comment.
Actually, this check is already implemented in a factorized way for a number of enet_coordinate_descent* functions here, similarly to @jnothman's suggestion in https://github.com/scikit-learn/scikit-learn/pull/11754/files#r208071463.
What is missing is to find other places where these functions are used and implement a similar check, namely just,
sklearn/covariance/graph_lasso_.py
225: coefs, _, _, _ = cd_fast.enet_coordinate_descent_gram(
|
The strict inequality in those functions misses the cases raised in the original issue. Essentially, there are numerical issues causing the duality gap and tolerance to be equal to zero - as such the warning won't be raised in the estimator. Should the check in those functions be changed? |
|
rebased. Now ConvergeWarning comes from the cython code for those who use directly the cython functions. |
|
@rth can you have a look? |
|
We do have a bunch of warnings in examples now (cf. https://circleci.com/gh/scikit-learn/scikit-learn/48650?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link) |
| @@ -794,5 +817,11 @@ def enet_coordinate_descent_multi_task(floating[::1, :] W, floating l1_reg, | |||
| if gap < tol: | |||
| # return if we reached desired tolerance | |||
| break | |||
| else: | |||
There was a problem hiding this comment.
after this warning message was printed, the for-loop goes on if max_iter is not reached, right?
And if max_iter is reached before the condition in 767 happens then it won't converge but never warn?
…r cases when the main loop exits without reaching the desired tolerance (scikit-learn#11754)
…t.pyx for cases when the main loop exits without reaching the desired tolerance (scikit-learn#11754)" This reverts commit 97a1b0e.
…t.pyx for cases when the main loop exits without reaching the desired tolerance (scikit-learn#11754)" This reverts commit 97a1b0e.
…r cases when the main loop exits without reaching the desired tolerance (scikit-learn#11754)
Reference Issues/PRs
Fixes #10813.
What does this implement/fix? Explain your changes.
This pull request adds
ConvergenceWarningsto theenet_coordinate_descent*solvers found in scikit-learn/sklearn/linear_model/cd_fast.pyx for cases when the main loop exits without reaching the desired tolerance.Any other comments?
Tests have been included in both sklearn/linear_model/tests/test_coordinate_descent.py and sklearn/linear_model/tests/test_sparse_coordinate_descent.py