Thanks to visit codestin.com
Credit goes to github.com

Skip to content

[MRG+1] Fix BIC/AIC for Lasso #9022

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 8 commits into from
Jun 11, 2017
Merged

[MRG+1] Fix BIC/AIC for Lasso #9022

merged 8 commits into from
Jun 11, 2017

Conversation

agramfort
Copy link
Member

Reference Issue

taking over #6080

What does this implement/fix? Explain your changes.

AIC/BIC criterion in LassoLarsIC was buggy ...

Any other comments?

Nope

@agramfort agramfort mentioned this pull request Jun 6, 2017
Closed
agramfort
agramfort commented Jun 6, 2017
View reviewed changes
sklearn/linear_model/least_angle.py
with np.errstate(divide='ignore'):
self.criterion_ = n_samples * np.log(mean_squared_error) + K * df
eps64 = np.finfo('float64').eps
self.criterion_ = (n_samples * mean_squared_error / (sigma2 + eps64) +
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added the eps64 to avoid the division by zero

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, it took me a while to understand that the reason that we don't need the log is that the MSE is actually already the log-likelihood.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe add a comment pointing any future reader to eqn 53/54 of https://web.stanford.edu/~hastie/Papers/dflasso.pdf? It's true the paper is already in the docstring though, so feel free to just ignore this comment.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also maybe document somewhere that the criterion is off by a constant and scaled by n, compared to the actual value, but that this doesn't affect relative comparisons? The wikipedia page for AIC has a section calling this "software unreliability"...

agramfort
agramfort commented Jun 6, 2017
View reviewed changes
sklearn/linear_model/tests/test_least_angle.py Outdated
X = diabetes.data
y = diabetes.target
X = np.c_[X, rng.randn(X.shape[0], 4)] # add 4 bad features
X = np.c_[X, rng.randn(X.shape[0], 5)] # add 4 bad features
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I need this to have enough alpha on the grid so alpha_bic > alpha_aic (otherwise it was equal)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

comment is now out of sync with code

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

comment is out of sync with code

agramfort
agramfort commented Jun 6, 2017
View reviewed changes
sklearn/linear_model/tests/test_least_angle.py
X = y.reshape(-1, 1)
lars = linear_model.LassoLarsIC(normalize=False)
assert_no_warnings(lars.fit, X, y)
assert_true(np.any(np.isinf(lars.criterion_)))
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not needed anymore. There is no log anymore

@agramfort agramfort changed the title Fix BIC/AIC for Lasso [MRG] Fix BIC/AIC for Lasso Jun 6, 2017
Mehmet Basbug and others added 3 commits June 7, 2017 09:54
@agramfort
correcting information criterion calculation in least_angle.py
0f81cc4 
The information criterion calculation is not compatible with the original paper 
Zou, Hui, Trevor Hastie, and Robert Tibshirani. "On the “degrees of freedom” of the lasso." The Annals of Statistics 35.5 (2007): 2173-2192.
APA
@agramfort
FIX : fix AIC/BIC computation in LassoLarsIC
bf165ff
@agramfort
update what's new
ee9802f
@agramfort agramfort force-pushed the fix_bic_aic_lasso branch from 581c3d0 to ee9802f Compare June 7, 2017 07:56
@agramfort
Copy link
Member Author

good to go on my end. Travis is happy

vene
vene reviewed Jun 8, 2017
View reviewed changes
doc/whats_new.rst Outdated

- Fixed a memory leak in our LibLinear implementation. :issue:`9024` by
:user:`Sergei Lebedev <superbobry>`
- Fix AIC/BIC criterion computation in LassoLarsIC by `Alexandre Gramfort`_
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

space before?

@GaelVaroquaux
Copy link
Member

LGTM.

+1 for merge

@GaelVaroquaux GaelVaroquaux changed the title [MRG] Fix BIC/AIC for Lasso [MRG+1] Fix BIC/AIC for Lasso Jun 10, 2017
@agramfort
Copy link
Member Author

@vene you give me the other +1 plz?

vene
vene reviewed Jun 10, 2017
View reviewed changes
sklearn/linear_model/least_angle.py
raise ValueError('criterion should be either bic or aic')

R = y[:, np.newaxis] - np.dot(X, coef_path_) # residuals
mean_squared_error = np.mean(R ** 2, axis=0)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here we take the mean of sum of squares, only to multiply by n_samples afterwards. It seems mean_squared_error is not used elsewhere in the local scope, we could save some cycles.

@vene
Copy link
Member

vene commented Jun 10, 2017

Hmm I'd like a stronger test but I don't have any good ideas of how to get one... this was subtle until I found the equations in the paper, the wikipedia page was not super helpful...

@vene
Copy link
Member

vene commented Jun 10, 2017

LGTM apart from the minor comments. @agramfort , if you're busy but agree with my comments I'll be happy to make the changes myself and merge.

@agramfort
Copy link
Member Author

agramfort commented Jun 11, 2017 via email

vene and others added 2 commits June 11, 2017 11:46
@vene
Copy link
Member

vene commented Jun 11, 2017

My comment have been addressed and @GaelVaroquaux 's +1 should still stand. Travis passes; Appveyor failures seem irrelevant and also present on master. (AFAIK appveyor is just back from an outage.)

Merging. Thanks @agramfort and @mehmetbasbug !

@vene vene merged commit 6a35622 into scikit-learn:master Jun 11, 2017
Sundrique pushed a commit to Sundrique/scikit-learn that referenced this pull request Jun 14, 2017
@agramfort @Sundrique
[FIX] BIC/AIC for Lasso (scikit-learn#9022)
e516e02 
* correcting information criterion calculation in least_angle.py

The information criterion calculation is not compatible with the original paper 
Zou, Hui, Trevor Hastie, and Robert Tibshirani. "On the “degrees of freedom” of the lasso." The Annals of Statistics 35.5 (2007): 2173-2192.
APA

* FIX : fix AIC/BIC computation in LassoLarsIC

* update what's new

* fix test

* fix test

* address comments

* DOC comments and docstring on criterion computation
dmohns pushed a commit to dmohns/scikit-learn that referenced this pull request Aug 7, 2017
@agramfort @dmohns
[FIX] BIC/AIC for Lasso (scikit-learn#9022)
6c40a36 
* correcting information criterion calculation in least_angle.py

The information criterion calculation is not compatible with the original paper 
Zou, Hui, Trevor Hastie, and Robert Tibshirani. "On the “degrees of freedom” of the lasso." The Annals of Statistics 35.5 (2007): 2173-2192.
APA

* FIX : fix AIC/BIC computation in LassoLarsIC

* update what's new

* fix test

* fix test

* address comments

* DOC comments and docstring on criterion computation
dmohns pushed a commit to dmohns/scikit-learn that referenced this pull request Aug 7, 2017
@agramfort @dmohns
[FIX] BIC/AIC for Lasso (scikit-learn#9022)
047a6c3 
* correcting information criterion calculation in least_angle.py

The information criterion calculation is not compatible with the original paper 
Zou, Hui, Trevor Hastie, and Robert Tibshirani. "On the “degrees of freedom” of the lasso." The Annals of Statistics 35.5 (2007): 2173-2192.
APA

* FIX : fix AIC/BIC computation in LassoLarsIC

* update what's new

* fix test

* fix test

* address comments

* DOC comments and docstring on criterion computation
NelleV pushed a commit to NelleV/scikit-learn that referenced this pull request Aug 11, 2017
@agramfort @NelleV
[FIX] BIC/AIC for Lasso (scikit-learn#9022)
c0addef 
* correcting information criterion calculation in least_angle.py

The information criterion calculation is not compatible with the original paper 
Zou, Hui, Trevor Hastie, and Robert Tibshirani. "On the “degrees of freedom” of the lasso." The Annals of Statistics 35.5 (2007): 2173-2192.
APA

* FIX : fix AIC/BIC computation in LassoLarsIC

* update what's new

* fix test

* fix test

* address comments

* DOC comments and docstring on criterion computation
paulha pushed a commit to paulha/scikit-learn that referenced this pull request Aug 19, 2017
@agramfort @paulha
[FIX] BIC/AIC for Lasso (scikit-learn#9022)
e094e3c 
* correcting information criterion calculation in least_angle.py

The information criterion calculation is not compatible with the original paper 
Zou, Hui, Trevor Hastie, and Robert Tibshirani. "On the “degrees of freedom” of the lasso." The Annals of Statistics 35.5 (2007): 2173-2192.
APA

* FIX : fix AIC/BIC computation in LassoLarsIC

* update what's new

* fix test

* fix test

* address comments

* DOC comments and docstring on criterion computation
AishwaryaRK pushed a commit to AishwaryaRK/scikit-learn that referenced this pull request Aug 29, 2017
@agramfort @AishwaryaRK
[FIX] BIC/AIC for Lasso (scikit-learn#9022)
014ed11 
* correcting information criterion calculation in least_angle.py

The information criterion calculation is not compatible with the original paper 
Zou, Hui, Trevor Hastie, and Robert Tibshirani. "On the “degrees of freedom” of the lasso." The Annals of Statistics 35.5 (2007): 2173-2192.
APA

* FIX : fix AIC/BIC computation in LassoLarsIC

* update what's new

* fix test

* fix test

* address comments

* DOC comments and docstring on criterion computation
maskani-moh pushed a commit to maskani-moh/scikit-learn that referenced this pull request Nov 15, 2017
@agramfort @maskani-moh
[FIX] BIC/AIC for Lasso (scikit-learn#9022)
c24533d 
* correcting information criterion calculation in least_angle.py

The information criterion calculation is not compatible with the original paper 
Zou, Hui, Trevor Hastie, and Robert Tibshirani. "On the “degrees of freedom” of the lasso." The Annals of Statistics 35.5 (2007): 2173-2192.
APA

* FIX : fix AIC/BIC computation in LassoLarsIC

* update what's new

* fix test

* fix test

* address comments

* DOC comments and docstring on criterion computation
jwjohnson314 pushed a commit to jwjohnson314/scikit-learn that referenced this pull request Dec 18, 2017
@agramfort
[FIX] BIC/AIC for Lasso (scikit-learn#9022)
93d694a 
* correcting information criterion calculation in least_angle.py

The information criterion calculation is not compatible with the original paper 
Zou, Hui, Trevor Hastie, and Robert Tibshirani. "On the “degrees of freedom” of the lasso." The Annals of Statistics 35.5 (2007): 2173-2192.
APA

* FIX : fix AIC/BIC computation in LassoLarsIC

* update what's new

* fix test

* fix test

* address comments

* DOC comments and docstring on criterion computation
@martin-hahn martin-hahn mentioned this pull request Feb 17, 2018
Closed
@flyjuice flyjuice mentioned this pull request Aug 5, 2019
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@GaelVaroquaux GaelVaroquaux GaelVaroquaux left review comments

@vene vene vene left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@agramfort @GaelVaroquaux @vene