Ah, it's actually equivalent with

            to_n_i = np.random.choice(n_classes, missing_train, replace=False,
                                      p=left_over_per_class / 
                                      left_over_per_class.sum()),

I think

Is this a fix that should go back to cross_validation.py (and also be backported to the recently released version?)

numpy.random.RandomState.choice was added in numpy 1.7, according to the numpy doc, so you get some errors like this:

    choice = rng.choice(n_classes, n_samples, replace=False,

AttributeError: 'mtrand.RandomState' object has no attribute 'choice'

I do agree there was a problem that train was getting too many samples as you noted in the associated issue and it's great that you managed to remove the dodgy second-pass "unassigned samples" logic.

numpy.random.RandomState.choice was added in numpy 1.7,

Use sklearn.utils.random.choice

To find such a thing, don't hesitate to do a git grep in the scikit-learn
codebase.

@amueller What exactly should I be looking for here ?

cc @MechCoder

hm maybe the best choice would be to remove the randomness in picking t_i and n_i and using a mode of the multinomial hypergeometric instead of sampling from it. Not sure if it is easy to compute that. I asked on stackoverflow ^^

ok so I can't spend that much more time on it, but I think the newest version without sampling is ok.
I'm using a bad approximation of the mode of the multivariate hypergeometric. Here "bad approximation" means that it can be off by one for each class. So that should not be that bad for us. This is all about how to do the rounding.

What I'm doing now is computing the most likely draw from the original class_counts with n_train many point. Then, from the class_counts minus what is in the training part, I get the most likely outcome for drawing n_test many points.

There is an alternative, which is "draw n_train points from class_counts, draw n_test points from class_counts, and make sure that we they don't sum up to more than class_counts". While this behavior might be a bit more "intuitive", the "ensure that they don't sum up to more than class_counts" part is more or less what was buggy in #6121.

So I'd rather stay with the simpler semantics of doing one draw and then the other draw from what's left over.

I have no idea what to do with the failing tests, though.
The first test is now failing because of the following situation:

(Pdb) p np.bincount(y[test])
array([2, 1, 1, 1])
(Pdb) p np.bincount(y[train])
array([2, 3, 3, 2])
(Pdb) p np.bincount(y)
array([4, 4, 4, 3])

 x: array([ 0.2,  0.3,  0.3,  0.2])
 y: array([ 0.4,  0.2,  0.2,  0.2])
>>  raise AssertionError('\nArrays are not almost equal to 1 decimals\n\n(mismatch 25.0%)\n x: array([ 0.2,  0.3,  0.3,  0.2])\n y: array([ 0.4,  0.2,  0.2,  0.2])')

The distribution between training and test part are not the same to one digit.
But that's impossible in this setup! I checked and a training set of [2, 3, 3, 2] is actually the true mode. So we do find the most likely configuration of the training set. As we have n_train + n_test = n_samples in this example, we have no choice in how to create the test set.
I have no idea how to fix the test, or why it was passing before. Probably because of one of the bugs.

ping @GaelVaroquaux who wrote the tests ;)

So indeed, the test that is failing passes on master with this configuration:

ipdb> p np.bincount(y)
array([4, 4, 4, 3])
ipdb> p np.bincount(y[train])
array([3, 3, 3, 2])
ipdb> p np.bincount(y[test])
array([1, 1, 1, 1])

So it violated the n_train / n_test sizes and put one more in the training set than it should have, to balance the classes.
@GaelVaroquaux was that expected behavior?
I mean it kind of makes sense, but it is non-obvious to me.

I can't think clearly enough on how to attain a balance between.

a] maintaining the train_size and test_size
b] maintaining the proportionality of the class labels between the train and test size :(

It is also not clear to me, how this block of the previous code preserves the class proportionality in the previous code?. Was it simply that it was not tested enough?

        if len(train) + len(test) < n_train + n_test:
            # We complete by affecting randomly the missing indexes
            missing_indices = np.where(bincount(train + test,
                                                minlength=len(y)) == 0)[0]
            missing_indices = rng.permutation(missing_indices)
            n_missing_train = n_train - len(train)
            n_missing_test = n_test - len(test)

            if n_missing_train > 0:
                train.extend(missing_indices[:n_missing_train])
            if n_missing_test > 0:
                test.extend(missing_indices[-n_missing_test:])

@MechCoder it doesn't. It just samples randomly. There are very strict tests that passed.

My intuition of the problem was: n_train and n_test are specified directly by the user, so these are hard limits. The stratification is a "best effort" kind of thing. The best possible is to pick a mode of the multivariate hypergeometric, but doing that exactly would require quite a bit of code. So I do an approximation that might be off by one per class (I have not proven this bound).

This should be good now. Reviews please?

CIs are not happy

Next try. Actually computing the mode (approximately) now, not sampling, to satisfy the proportion tests. But now breaking ties at random to satisfy the IID test.

CI error is server error in curl

Weird the appveyor error report has happened on @agramfort's account instead of sklearn-ci. Anyway this is not a big deal and can be ignored.

There are still some inline comments to tackle in the diff view but other than that +1 as well.

@ogrisel I added (hopefully helpful) doctests.

LGTM, let me squash-merge that. Thanks for the fix @amueller.

thanks for the reviews and merge :)