GroupKFold fails in nested cross-validation (similar to #2879) #7646

davidslater · 2016-10-11T21:15:02Z

Description

groups parameter in model_selection.cross_val_score() is not propagated in to RandomSearchCV.fit() call. This is similar to #2879 and probably best addressed in #4497.

Steps/Code to Reproduce

import numpy as np
from sklearn.utils.validation import indexable
from sklearn import linear_model
from sklearn import model_selection

# generate data with simple decision boundary, with 2 labels and 2 groups per label
X = np.array(range(20)).reshape(-1, 1)
y = np.array([0] * 10 + [1] * 10)
groups = np.array([0] * 5 + [1] * 5 + [2] * 5 + [3] * 5)

# run nested cross-validation (works with StratifiedKFold, but not GroupKFold)
clf = linear_model.LogisticRegression()
#cv = model_selection.StratifiedKFold(n_splits=2)
cv = model_selection.GroupKFold(n_splits=2)
param_dist = {'penalty': ['l1', 'l2'], 'C': np.logspace(-3, 3, 13)}
random_search = model_selection.RandomizedSearchCV(clf, cv=cv, param_distributions=param_dist, n_iter=20)
print model_selection.cross_val_score(random_search, X, y=y, groups=groups, cv=cv)

Expected Results

When StratifiedKFold is used, the output is [ 0.8 0.7]. In general, it should be an array of 2 floats.

Actual Results

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/davidslater/.virtualenvs/davidslater/lib/python2.7/site-packages/sklearn/model_selection/_validation.py", line 140, in cross_val_score
    for train, test in cv.split(X, y, groups))
  File "/Users/davidslater/.virtualenvs/davidslater/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.py", line 758, in __call__
    while self.dispatch_one_batch(iterator):
  File "/Users/davidslater/.virtualenvs/davidslater/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.py", line 608, in dispatch_one_batch
    self._dispatch(tasks)
  File "/Users/davidslater/.virtualenvs/davidslater/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.py", line 571, in _dispatch
    job = self._backend.apply_async(batch, callback=cb)
  File "/Users/davidslater/.virtualenvs/davidslater/lib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.py", line 109, in apply_async
    result = ImmediateResult(func)
  File "/Users/davidslater/.virtualenvs/davidslater/lib/python2.7/site-packages/sklearn/externals/joblib/_parallel_backends.py", line 322, in __init__
    self.results = batch()
  File "/Users/davidslater/.virtualenvs/davidslater/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.py", line 131, in __call__
    return [func(*args, **kwargs) for func, args, kwargs in self.items]
  File "/Users/davidslater/.virtualenvs/davidslater/lib/python2.7/site-packages/sklearn/model_selection/_validation.py", line 238, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File "/Users/davidslater/.virtualenvs/davidslater/lib/python2.7/site-packages/sklearn/model_selection/_search.py", line 1185, in fit
    return self._fit(X, y, groups, sampled_params)
  File "/Users/davidslater/.virtualenvs/davidslater/lib/python2.7/site-packages/sklearn/model_selection/_search.py", line 562, in _fit
    for parameters in parameter_iterable
  File "/Users/davidslater/.virtualenvs/davidslater/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.py", line 758, in __call__
    while self.dispatch_one_batch(iterator):
  File "/Users/davidslater/.virtualenvs/davidslater/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.py", line 603, in dispatch_one_batch
    tasks = BatchedCalls(itertools.islice(iterator, batch_size))
  File "/Users/davidslater/.virtualenvs/davidslater/lib/python2.7/site-packages/sklearn/externals/joblib/parallel.py", line 127, in __init__
    self.items = list(iterator_slice)
  File "/Users/davidslater/.virtualenvs/davidslater/lib/python2.7/site-packages/sklearn/model_selection/_search.py", line 563, in <genexpr>
    for train, test in cv.split(X, y, groups))
  File "/Users/davidslater/.virtualenvs/davidslater/lib/python2.7/site-packages/sklearn/model_selection/_split.py", line 321, in split
    for train, test in super(_BaseKFold, self).split(X, y, groups):
  File "/Users/davidslater/.virtualenvs/davidslater/lib/python2.7/site-packages/sklearn/model_selection/_split.py", line 90, in split
    for test_index in self._iter_test_masks(X, y, groups):
  File "/Users/davidslater/.virtualenvs/davidslater/lib/python2.7/site-packages/sklearn/model_selection/_split.py", line 102, in _iter_test_masks
    for test_index in self._iter_test_indices(X, y, groups):
  File "/Users/davidslater/.virtualenvs/davidslater/lib/python2.7/site-packages/sklearn/model_selection/_split.py", line 474, in _iter_test_indices
    raise ValueError("The groups parameter should not be None")
ValueError: The groups parameter should not be None

Versions

Darwin-15.6.0-x86_64-i386-64bit
('Python', '2.7.11 (default, Jan 22 2016, 08:29:18) \n[GCC 4.2.1 Compatible Apple LLVM 7.0.2 (clang-700.1.81)]')
('NumPy', '1.11.2')
('SciPy', '0.18.1')
('Scikit-Learn', '0.18')

The text was updated successfully, but these errors were encountered:

davidslater · 2016-10-11T21:18:39Z

In particular, the estimator.fit(X_train, y_train, **fit_params) call in model_selection._validation.py does not include "groups" in fit_params, so it defaults to None.

jnothman · 2016-10-13T12:52:29Z

Thanks for the report @davidslater. I'm not sure if this is to be fixed for 0.18.1 (it only applies to 0.18, but AFAIK this kind of nested CV wasn't possible before, so it's not exactly a regression), but I've labelled as such.

amueller · 2016-10-14T16:41:20Z

I agree we need #4497 to fix this, and I don't see how we could do it for 0.18.1 - except special-casing the groups parameter to be passed to the cross-validation but not the estimator.

amueller · 2016-10-14T16:41:48Z

"need contributor"? I don't think we agree on a fix, do we?

jnothman · 2016-10-15T09:39:05Z

Or just some kind of routing parameter specific to CV, seeing as we already
accept a groups param to GridSearchCV.fit?

On 15 October 2016 at 03:41, Andreas Mueller [email protected]
wrote:

"need contributor"? I don't think we agree on a fix, do we?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#7646 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAEz66TUo2v46mTWvenCuQoKpOgxTvNAks5qz7DNgaJpZM4KUHR0
.

GaelVaroquaux · 2016-10-17T08:19:32Z

I really do think that we need #4497 to address this. It's an issue that
is very dear to my heart, by I don't think that rushing it is a good
idea. This is probably material for a sprint discussion.

raghavrv · 2016-10-21T14:22:07Z

Yes, I don't think this should be tagged 0.18.1...

aryamccarthy · 2017-03-26T01:27:26Z

Bump. Is there an agreement on how to approach this?

jnothman · 2017-03-26T05:01:42Z

Is there an easy fix (special casing 'groups') for this? I feel like a proper one is a while off yet.

…

On 26 Mar 2017 12:27 pm, "Arya McCarthy" ***@***.***> wrote: Bump. Is there an agreement on how to approach this? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#7646 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEz68AU5I1TSRAIjQ3YvawaG8w6heyZks5rpb7_gaJpZM4KUHR0> .

s96lam · 2020-05-09T21:29:09Z

I encountered what seems to be another form of the same error running the example code for nested CV with GroupKFold as cv technique:
ValueError: The 'groups' parameter should not be None.

This is no problem, as there are other ways to implement nested CV without using cross_val_score, but the example provided specifically suggests using GroupKFold as an possible method. If not implemented in upcoming versions, could the nested CV example and the cross_val_score documentation (this seems to be where it fails?) be updated to reflect that this is currently not possible?

Sorry if this is the wrong place, but I'm pretty new to this and didn't want to open a new Issue.

staticdev · 2020-05-20T14:41:33Z

Any updated on this? It has a WIP PR since 2017 =/

jnothman · 2020-05-21T01:15:41Z

Yes, and the general problem is making some good progress, but it's a hard one.

lkev · 2021-05-14T14:06:09Z

Would it be possible to raise a NotImplementedError or similar on this? It fails silently as-is and just outputs NaN vals for scores. Took a lot of searching to find this issue.

collinb9 · 2021-05-28T17:06:42Z

In particular, the estimator.fit(X_train, y_train, **fit_params) call in model_selection._validation.py does not include "groups" in fit_params, so it defaults to None.

So unless I'm mistaken, it looks like passing fit_params={"groups": groups} to cross_validate() works for now?

staticdev · 2021-05-28T18:46:04Z

This is a serious issue for running this kind of splitting. I don't know how it took so many years and still is not fixed.

jhn-nt · 2021-10-05T12:53:51Z

Hi All, is there any fix or work around on this one?

ivezakis · 2021-10-21T14:38:28Z

So unless I'm mistaken, it looks like passing fit_params={"groups": groups} to cross_validate() works for now?

Curiously, this does seem to work. I am saying curiously because normally, I'd expect an error due to shape mismatch - we'd be passing a subset of the data to the nested CV, but with the full group and not its corresponding subset.
However, there are no errors. I am unfamiliar with the source code to reliably check this myself. Can somebody confirm this is a valid workaround?

jnothman added the Bug label Oct 13, 2016

jnothman added this to the 0.18.1 milestone Oct 13, 2016

jnothman added the Need Contributor label Oct 13, 2016

amueller added the API label Oct 14, 2016

raghavrv removed this from the 0.18.1 milestone Oct 21, 2016

raghavrv mentioned this issue Oct 22, 2016

permutation_test_score does not handle output from GridSearchCV #7722

Open

cbrummitt mentioned this issue Dec 4, 2016

Add TimeSeriesCV and HomogeneousTimeSeriesCV #6322

Open

rhiever mentioned this issue Feb 23, 2017

TimeSeriesSplit EpistasisLab/tpot#342

Closed

aryamccarthy mentioned this issue Mar 26, 2017

Hyperparameter tuning with MLP regressor SMU-Ubicomp-Lab/OpenCycle#21

Open

jnothman mentioned this issue Aug 16, 2017

[WIP] Sample property routing #9566

Closed

11 tasks

lesteve added help wanted and removed Need Contributor labels Oct 18, 2017

DanLeopold mentioned this issue Apr 3, 2019

CalibratedClassifierCV doesn't support groups parameter in fit method #12052

Closed

cmarmo removed the help wanted label Aug 24, 2020

adrinjalali mentioned this issue Oct 29, 2020

[WIP] sample props (proposal 4) #16079

Closed

adrinjalali mentioned this issue Jun 24, 2021

sample-props alternate implementation #20350

Closed

aretor mentioned this issue Nov 13, 2021

Error when using “Group” cv instances manuel-calzolari/sklearn-genetic#28

Closed

cmarmo added the module:model_selection label Dec 14, 2021

adrinjalali mentioned this issue Aug 18, 2022

FEAT SLEP006: metadata routing infrastructure #24027

Merged

adrinjalali mentioned this issue Apr 26, 2023

SLEP006 - Metadata Routing task list #22893

Open

28 tasks

adrinjalali mentioned this issue Aug 3, 2023

ENH add metadata routing to cross_val* #26896

Merged

glemaitre closed this as completed in #26896 Aug 9, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GroupKFold fails in nested cross-validation (similar to #2879) #7646

GroupKFold fails in nested cross-validation (similar to #2879) #7646

davidslater commented Oct 11, 2016

davidslater commented Oct 11, 2016

jnothman commented Oct 13, 2016

amueller commented Oct 14, 2016

amueller commented Oct 14, 2016

jnothman commented Oct 15, 2016

GaelVaroquaux commented Oct 17, 2016

raghavrv commented Oct 21, 2016

aryamccarthy commented Mar 26, 2017

jnothman commented Mar 26, 2017 via email

s96lam commented May 9, 2020

staticdev commented May 20, 2020

jnothman commented May 21, 2020 via email

lkev commented May 14, 2021

collinb9 commented May 28, 2021

staticdev commented May 28, 2021 •

edited

Loading

jhn-nt commented Oct 5, 2021

ivezakis commented Oct 21, 2021

GroupKFold fails in nested cross-validation (similar to #2879) #7646

GroupKFold fails in nested cross-validation (similar to #2879) #7646

Comments

davidslater commented Oct 11, 2016

Description

Steps/Code to Reproduce

Expected Results

Actual Results

Versions

davidslater commented Oct 11, 2016

jnothman commented Oct 13, 2016

amueller commented Oct 14, 2016

amueller commented Oct 14, 2016

jnothman commented Oct 15, 2016

GaelVaroquaux commented Oct 17, 2016

raghavrv commented Oct 21, 2016

aryamccarthy commented Mar 26, 2017

jnothman commented Mar 26, 2017 via email

s96lam commented May 9, 2020

staticdev commented May 20, 2020

jnothman commented May 21, 2020 via email

lkev commented May 14, 2021

collinb9 commented May 28, 2021

staticdev commented May 28, 2021 • edited Loading

jhn-nt commented Oct 5, 2021

ivezakis commented Oct 21, 2021

staticdev commented May 28, 2021 •

edited

Loading