I can add a few more test cases to this. With the following data

import sklearn.cross_validation
from sklearn.datasets import make_blobs
X, y = make_blobs(n_samples=1000, n_features=2, random_state=42, cluster_std=5.0)

All manner of crazy things happen

split = sklearn.cross_validation.StratifiedShuffleSplit(y, test_size=0.09, train_size=0.1)
train_idx, test_idx = next(iter(split))
assert_equal(train_idx.shape[0], split.n_train) # pass
assert_equal(test_idx.shape[0], split.n_test) # 901 != 90

AssertionError:
Items are not equal:
ACTUAL: 901
DESIRED: 90

split = sklearn.cross_validation.StratifiedShuffleSplit(y, test_size=0.11, train_size=0.11)
train_idx, test_idx = next(iter(split))
assert_equal(train_idx.shape[0], split.n_train) # 900 != 110
assert_equal(test_idx.shape[0], split.n_test) # 111 != 110

AssertionError:
Items are not equal:
ACTUAL: 900
DESIRED: 110

AssertionError:
Items are not equal:
ACTUAL: 111
DESIRED: 110

I'm not that bothered about getting 1 extra instance in the test or training set, but getting 10x the amount of data that was requested is not optimal.

Right, so the problem is this line

test.extend(missing_idx[-(split.n_test - len(test)):])

https://github.com/scikit-learn/scikit-learn/blob/51a765a/sklearn/cross_validation.py#L1005

If split.n_test - len(test) is 0, meaning that we already have to correct number of test indices, we end up adding all of the missing indices to the test set, including those that went to train. The simple fix is pretty ugly, so not sure if you're still wanting to rewrite the whole thing.

This seems to have already been fixed on master though.

This seems to have already been fixed on master though.

Sorry, I may have missed something here. Have you found a bug that has not already been fixed in master or that was not the one for which the issue was created?

@lesteve no, after adding the comments I noticed that on the upstream master the issue has been fixed, so this should probably be closed.

The link above is to the 0.17 sources (linked to from the API docs). Sorry for the confusion.