[MRG + 1] Fix BayesianRidge() and ARDRegression() for constant target vectors #10095
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
I would have thought that |
|
@glemaitre Yea I like that solution better as well, will implement it. But note that then |
glemaitre
reviewed
Nov 9, 2017
sklearn/linear_model/bayes.py
Outdated
| @@ -162,7 +162,8 @@ def fit(self, X, y): | |||
| n_samples, n_features = X.shape | |||
|
|
|||
| # Initialization of the values of the parameters | |||
| alpha_ = 1. / np.var(y) | |||
| eps = np.spacing(1) | |||
There was a problem hiding this comment.
You might want to add a small comment to explain why we add eps
There was a problem hiding this comment.
Thx for the suggestion. I added a comment, and additionally applied the same fix to ARDRegression
There was a problem hiding this comment.
I would prefer you use:
eps = np.finfo(np.float64).eps
as here you're not taking into account the dtype of X or y anyway
agramfort
reviewed
Nov 9, 2017
| # constant target vectors | ||
| n_samples = 4 | ||
| n_features = 5 | ||
| constant_value = np.random.rand() |
agramfort
reviewed
Nov 9, 2017
| # The standard dev. should be relatively small (< 0.1 is tested here) | ||
| n_samples = 4 | ||
| n_features = 5 | ||
| constant_value = np.random.rand() |
jnothman
reviewed
Nov 9, 2017
|
|
||
| def test_std_ard_with_constant_input(): | ||
| # Test ARDRegression standard dev. for edge case of constant target vector | ||
| # The standard dev. should be relatively small (< 0.1 is tested here) |
There was a problem hiding this comment.
That seems quite a large standard dev...
There was a problem hiding this comment.
tl;dr: I decreased the upper bound to 0.01.
Playing around with a couple of random initializations, the std was around 0.005. To make sure that the test won't fail by chance, I originally chose 0.1 as an upper bound. Now that I use a random state in the test, it's safe to reduce the bound to 0.01.
agramfort
approved these changes
Nov 10, 2017
glemaitre
approved these changes
Nov 10, 2017
|
thx @jdoepfert |
|
awesome! |
maskani-moh
pushed a commit
to maskani-moh/scikit-learn
that referenced
this pull request
Nov 15, 2017
… vectors (scikit-learn#10095) * add test for issue scikit-learn#10092 * add comment to test * split into two tests * add tests for scores, alpha and beta * adapt tests: n_samples != n_features * add test when no intercept is fitted * add handling of constant target vector when intercept is fitted * fix typo in comments * fix format issues * replace original fix with simpler fix * add comment * increase upper boundary for test * increase upper boundary for test * merge tests for ARDRegression and BayesianRidge * use random state in tests * decrease upper bound for std * replace np.spacing(1) -> np.finfo(np.float64).eps
jwjohnson314
pushed a commit
to jwjohnson314/scikit-learn
that referenced
this pull request
Dec 18, 2017
… vectors (scikit-learn#10095) * add test for issue scikit-learn#10092 * add comment to test * split into two tests * add tests for scores, alpha and beta * adapt tests: n_samples != n_features * add test when no intercept is fitted * add handling of constant target vector when intercept is fitted * fix typo in comments * fix format issues * replace original fix with simpler fix * add comment * increase upper boundary for test * increase upper boundary for test * merge tests for ARDRegression and BayesianRidge * use random state in tests * decrease upper bound for std * replace np.spacing(1) -> np.finfo(np.float64).eps
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Reference Issues/PRs
Fixes #10092
What does this implement/fix? Explain your changes.
This PR fixes the issue that when fitting a
BayesianRidgeorARDRegressionclassifier with a constant target vectory, thepredictmethod yields NaN arrays for both predictions and the respective standard devitions. This occurs since the estimated precision for the noisealpha_is initialized with1/np.var(y)=inf.This PR fixes this issue by initializing
alpha_with1/(np.var(y) + eps), whereeps=np.spacing(1).Any other comments?
The returned standard deviation
stdfor the predictions will not be zero for a constant target vector (as suggested in #10092), but a small number instead. I added a test for this. However,stdapproaches zero as the number of samples increase, as expected: