[MRG] FIX and ENH in _RidgeGCV #15648

qinhanmin2014 · 2019-11-18T10:06:33Z

Fixes #4667 Fixes #4790 Fixes #13998 Fixes #15182 Fixes #15183

This PR focus on RidgeCV

add best_score_ to RidgeCV, in order to write tests (RidgeCV should provide best_score_ #4667 [MRG+1] Add best_score_ attribute to RidgeCV and RidgeClassifierCV #4790)
do not store all cv values when store_cv_values == False (_RidgeGCV stores LOO predictions even when not required #15182 [MRG] Does not store all cv values nor all dual coef in _RidgeGCV fit #15183 )
when scoring = None
- correct the errors (previously they are erroneously weighted)
- correct the scores (previously they are erroneously weighted)
when scoring != None
- correct the predictions (previously they are erroneously weighted and scaled)
- correct the scores (previously they are erroneously weighted and scaled)
- handle multioutput correctly when calculating the scores

TODO in this PR: update the doc, update what's new

TODO: issues regarding RidgeClassifierCV
ping @glemaitre feel free to edit or push

Co-Authored-By: Thomas J Fan <[email protected]>

glemaitre · 2019-11-18T11:23:38Z

sklearn/linear_model/_ridge.py

-        self.dual_coef_ = C[best]
+                if y.ndim == 2:
+                    y_true = y / sqrt_sw[:, np.newaxis] + y_offset
+                    y_pred = predictions / sqrt_sw[:, np.newaxis] + y_offset


you don't need to create new axis. you can ravel and use np.repeat(sqrt_sw, n_y)

Is this memory efficient? I guess broadcast is better?

glemaitre · 2019-11-18T11:25:17Z

sklearn/linear_model/_ridge.py

+                if y.ndim == 2:
+                    squared_errors /= sample_weight[:, np.newaxis]
+                else:
+                    squared_errors /= sample_weight


It is not that easy. There is a test ridge_sample_weight which will fail.
Right now it was thought that repeating 3 times a sample will lead to an error 3 times bigger.
Normalizing the sample_weight will not lead to this results

this part makes sure that RidgeCV() is equivalent to GridSearchCV(Ridge(), cv=LeaveOneOut())

glemaitre · 2019-11-18T11:31:20Z

sklearn/linear_model/tests/test_ridge.py

+
+    assert (
+        ridge_cv_no_score.cv_values_.mean() == pytest.approx(
+            mean_squared_error(y, ridge_cv_score.cv_values_.ravel()))


From my above message, we could expect to pass sample_weight to mean_squared_error. However, the np.average will normalize by the sum of the weight. It is equivalent to normalize the weight such that they sum to 1.

However, the loss internally will not do such thing because we want to impose that 3 times a sample correspond to seeing 3 times a sample increasing the loss x3.

So this is not straightforward what to implement.

Basically, with the initial semantic the assert would be

ridge_cv_no_score.cv_values_.mean() == pytest.approx( (((y - ridge_cv_score.cv_values_.ravel()) ** 2) * sample_weight).sum() )

while the mean_squared_error would be equivalent to

ridge_cv_no_score.cv_values_.mean() == pytest.approx( (((y - ridge_cv_score.cv_values_.ravel()) ** 2) * sample_weight / sample_weight.sum()).sum() )

However, the loss internally will not do such thing because we want to impose that 3 times a sample correspond to seeing 3 times a sample increasing the loss x3.

Sorry but I can't understand your discussions, this PR is not related to the loss function?

I got confused as well so please ignore my previous comment. As far as I can tell the changes made in this PR yield the correct behaviour for _RidgeGCV

qinhanmin2014 · 2019-11-18T11:37:36Z

TODO:
(1) debug test_ridge_gcv_sample_weights (I comment out that test)
(2) more tests

qinhanmin2014 · 2019-11-18T11:43:52Z

@glemaitre Let me summarize my solution when scoring = None
(1) For the errors, I think we should not report weighted errors.
(2) For the scores, I think we should keep consistent with GridSearchCV (i.e., RidgeCV() should be equivalent to GridSearchCV(Ridge(), cv=LeaveOneOut()))
Do you agree?

jeromedockes · 2019-11-18T12:55:27Z

doc/whats_new/v0.22.rst

+  `store_cv_values` is `True`.
+  :pr:`15183` by :user:`Jérôme Dockès <jeromedockes>`.
+
+- |Fix| In :class:`linear_model.RidgeCV`, the predicitons reported by


I think there is a confusion in this whatsnew entry. the problem mentioned is described in issue #13998 and is not fixed in this PR

Sorry, I saw you fixed it too -- then maybe it is the pr number that needs to change?

Let's focus on the PR itself and ignore what's new now.

jeromedockes · 2019-11-18T13:38:31Z

(1) debug test_ridge_gcv_sample_weights (I comment out that test)

it checks that giving a sample weight of 3 gives the same score as repeating the sample 3 times. that is no longer the case if sample weights are not used to compute the scores. therefore instead of repeating samples and using GroupKFold this test should now simply compare the GCV with LOO GridSearch as you suggest in #15648 (comment)

jeromedockes · 2019-11-18T13:40:57Z

(1) debug test_ridge_gcv_sample_weights (I comment out that test)

it checks that giving a sample weight of 3 gives the same score as repeating the sample 3 times. that is no longer the case if sample weights are not used to compute the scores. therefore instead of repeating samples and using GroupKFold this test should now simply compare the GCV with LOO GridSearch as you suggest in #15648 (comment)

however this test should probably be kept but applied with only one hyperparameter in the grid, to check that for the coefficients and intercept giving sample weights is indeed equivalent to repeating samples -- for a fixed hyperparameter, not for computing the score

qinhanmin2014 · 2019-11-18T13:45:10Z

it checks that giving a sample weight of 3 gives the same score as repeating the sample 3 times. that is no longer the case if sample weights are not used to compute the scores. therefore instead of repeating samples and using GroupKFold this test should now simply compare the GCV with LOO GridSearch as you suggest in #15648 (comment)

thanks a lot :) I've figured out this reason and mentioned it in gitter.

qinhanmin2014 · 2019-11-22T03:40:44Z

ping @glemaitre @jeromedockes I add some tests here, perhaps its worthwhile for you to have a look.

jeromedockes and others added 15 commits October 11, 2019 16:20

do not store all cv values nor all dual coef in _RidgeGCV fit

3aac2e5

Apply suggestions from code review

51b9add

Co-Authored-By: Thomas J Fan <[email protected]>

whatsnew entry

a78be8c

add tests for different scorers in _RidgeGCV

bfb6d98

add note in _RidgeGCV docstring

c7baa7c

move whatsnew entry

61782b3

git move ridge -> _ridge

89dde17

Merge remote-tracking branch 'origin/master' into pr/jeromedockes/15183

02e694e

TST add additional tests

b4cf560

fix

98d2f07

TST check equivalence scoring none and mse

097e783

MNT FIX and ENH in _RidgeGCV

5188494

trigger CI

2de9c67

small bug

ff0a5ab

small bug

485a72f

glemaitre reviewed Nov 18, 2019

View reviewed changes

jeromedockes reviewed Nov 18, 2019

View reviewed changes

jeromedockes mentioned this pull request Nov 18, 2019

[MRG] Does not store all cv values nor all dual coef in _RidgeGCV fit #15183

Closed

glemaitre mentioned this pull request Nov 18, 2019

ENH do not allocate local arrays in Ridge*CV of store_cv_values is False #15652

Merged

qinhanmin2014 added 4 commits November 21, 2019 22:05

Merge remote-tracking branch 'upstream/master' into 15183

34a56b4

clean what's new

fef0b46

remove tests

4e183e6

remove redundant test (we no longer return weighted error)

82ed115

qinhanmin2014 added 4 commits November 21, 2019 22:22

clearer solution

703a93e

add some tests

90b7c68

flake8

72c1d1b

more tests

cb3c379

qinhanmin2014 changed the title ~~[WIP] FIX and ENH in _RidgeGCV~~ [MRG] FIX and ENH in _RidgeGCV Nov 22, 2019

github-actions bot added the module:linear_model label Mar 2, 2020

Base automatically changed from master to main January 22, 2021 10:51

glemaitre mentioned this pull request Sep 16, 2024

FIX scoring != None for RidgeCV should used unscaled y for evaluation #29842

Merged

thomasjpfan closed this in #29842 Sep 18, 2024

Uh oh!

[MRG] FIX and ENH in _RidgeGCV #15648

[MRG] FIX and ENH in _RidgeGCV #15648

Uh oh!

Conversation

qinhanmin2014 commented Nov 18, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

qinhanmin2014 commented Nov 18, 2019

Uh oh!

qinhanmin2014 commented Nov 18, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jeromedockes commented Nov 18, 2019

Uh oh!

jeromedockes commented Nov 18, 2019

Uh oh!

qinhanmin2014 commented Nov 18, 2019

Uh oh!

qinhanmin2014 commented Nov 22, 2019

Uh oh!

Uh oh!

qinhanmin2014 commented Nov 18, 2019 •

edited

Loading