Also one should add a non-regression test based on the digits dataset as highlighted in the notebook.

When you say that StratifiedKMeans should preserve individual dependency, do you mean each individual fold should contain samples of each classes in the same order as in the original set? If yes, I think using stable sorting should the problem, that is, replace line in cross_validation.py
from: idx = np.argsort(self.y)
to: idx = np.argsort(self.y, kind='mergesort')

The samples would be sorted, but the samples for each class would be taken in the same order as it is in the original set.

Or do you mean to asses dependency of individual classes in some more quantitative way?

Thanks a lot!

What I mean is that StratifiedKFold should minimally impact the original order of the samples when computing the folds.

The current behavior is:

>>> from sklearn.cross_validation import StratifiedKFold
>>> for train, test in StratifiedKFold([0, 0, 0, 1, 1, 1, 1, 1]):
...     print "train={}, test={}".format(train, test)
...
train=[1 2 4 5 7], test=[0 3 6]
train=[0 2 3 5 6], test=[1 4 7]
train=[0 1 3 4 6 7], test=[2 5]

Ideally I will like to have:

train=[1 2 5 6 7], test=[0 3 4]
train=[0 2 3 4 7], test=[1 5 6]
train=[0 1 3 4 5 6], test=[2 7]

I don't care much about preserving the ordering inside the folds (although it would be better if it is easy to implement).

I think idx = np.argsort(self.y, kind='mergesort') should do the trick.

No it does not, with kind='mergesort' you get:

>>> for train, test in StratifiedKFold([0, 0, 0, 1, 1, 1, 1, 1]):
...     print "train={}, test={}".format(train, test)
...
train=[1 2 4 5 7], test=[0 3 6]
train=[0 2 3 5 6], test=[1 4 7]
train=[0 1 3 4 6 7], test=[2 5]

I don't consider this a bug at all. We don't do structured learning, so we can assume there are no dependencies between samples.

It's inconsistent with the behavior of KFold and thus StratifiedKFold hides bad iid assumption problems, therefore misleading our users (my-self included).

We don't do structured learning, so we can assume there are no dependencies between samples.

But in real life there is always a bit of dependency and if you are not able to quantify the impact of the IID-breakage then your are just lying to yourself with too optimistic scores.

Furthermore we already provide LeaveOneLabelOut and LeavePLabelOut when the user has additional info on the dependency structure (e.g. experiments with distinct subjects).

Alright, I didn't fully grasp the issues. Never mind.

This line breaks for multilabel classifiers since it fails for both arrays of tuples as well as the 2D numpy array produced by MultiLabelBinarizer:

label_test_folds = test_folds[y == label]

StratifiedKFolddoesn't work for multilabel data.
There is pr that fix it the issue with cross_val_score.

Stratified cross-validation for the multilabel case is a much harder problem than for multiclass, and requires special algorithms.

What I mean is that StratifiedKFold should minimally impact the original order of the samples when computing the folds.

Couldn't we have just used a stable sort to do this? I know this is ancient but imbalanced test set sizes in #10154 makes this look like a pretty strange strategy!