{"action":"created","issue":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/8755","repository_url":"https://api.github.com/repos/scikit-learn/scikit-learn","labels_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/8755/labels{/name}","comments_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/8755/comments","events_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/8755/events","html_url":"https://github.com/scikit-learn/scikit-learn/issues/8755","id":222256371,"number":8755,"title":"train_test_split fails for too many values (32bit only)","user":{"login":"scottmsul","id":5205428,"avatar_url":"https://avatars1.githubusercontent.com/u/5205428?v=4","gravatar_id":"","url":"https://api.github.com/users/scottmsul","html_url":"https://github.com/scottmsul","followers_url":"https://api.github.com/users/scottmsul/followers","following_url":"https://api.github.com/users/scottmsul/following{/other_user}","gists_url":"https://api.github.com/users/scottmsul/gists{/gist_id}","starred_url":"https://api.github.com/users/scottmsul/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/scottmsul/subscriptions","organizations_url":"https://api.github.com/users/scottmsul/orgs","repos_url":"https://api.github.com/users/scottmsul/repos","events_url":"https://api.github.com/users/scottmsul/events{/privacy}","received_events_url":"https://api.github.com/users/scottmsul/received_events","type":"User","site_admin":false},"labels":[{"id":12995755,"url":"https://api.github.com/repos/scikit-learn/scikit-learn/labels/Bug","name":"Bug","color":"e10c02","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":9,"created_at":"2017-04-17T23:10:11Z","updated_at":"2018-05-21T22:42:03Z","closed_at":null,"author_association":"NONE","body":"Consider the following code:\r\n\r\n```py\r\nimport numpy as np\r\nfrom sklearn.model_selection import train_test_split\r\n\r\nn = 10000\r\ny = np.random.randint(0, 2, size=n)\r\n\r\ny_train, y_test = train_test_split(y, train_size=int(n/2),\r\n                                   test_size=int(n/2), stratify=y, random_state=123)\r\n\r\nprint('num train: {}'.format(len(y_train)))\r\nprint('train mean: {}'.format(y_train.mean()))\r\nprint('num test: {}'.format(len(y_test)))\r\nprint('test mean: {}'.format(y_test.mean()))\r\n```\r\n\r\nWhen n=10,000, I correctly obtain:\r\n```py\r\nnum train: 5000\r\ntrain mean: 0.4958\r\nnum test: 5000\r\ntest mean: 0.4958\r\n```\r\n\r\nBut for larger n, such as n=100,000, I get the following error:\r\n```pytb\r\n---------------------------------------------------------------------------\r\nIndexError                                Traceback (most recent call last)\r\n/home/scott/Development/scratch/sklearn/stratify.py in <module>()\r\n      6 y = np.random.randint(0, 2, size=n)\r\n      7 \r\n----> 8 y_train, y_test = train_test_split(y, train_size=n/2, test_size=n/2, stratify=y, random_state=123)\r\n      9 \r\n     10 print 'num train: {}'.format(len(y_train))\r\n\r\n/home/scott/anaconda/lib/python2.7/site-packages/sklearn/model_selection/_split.pyc in train_test_split(*arrays, **options)\r\n   1700     train, test = next(cv.split(X=arrays[0], y=stratify))\r\n   1701     return list(chain.from_iterable((safe_indexing(a, train),\r\n-> 1702                                      safe_indexing(a, test)) for a in arrays))\r\n   1703 \r\n   1704 \r\n\r\n/home/scott/anaconda/lib/python2.7/site-packages/sklearn/model_selection/_split.pyc in <genexpr>((a,))\r\n   1700     train, test = next(cv.split(X=arrays[0], y=stratify))\r\n   1701     return list(chain.from_iterable((safe_indexing(a, train),\r\n-> 1702                                      safe_indexing(a, test)) for a in arrays))\r\n   1703 \r\n   1704 \r\n\r\n/home/scott/anaconda/lib/python2.7/site-packages/sklearn/utils/__init__.pyc in safe_indexing(X, indices)\r\n    110             return X.take(indices, axis=0)\r\n    111         else:\r\n--> 112             return X[indices]\r\n    113     else:\r\n    114         return [X[idx] for idx in indices]\r\n\r\nIndexError: arrays used as indices must be of integer (or boolean) type\r\n```\r\n\r\nAnd for n=1,000,000, I don't get an exception, instead the strange results:\r\n```py\r\nnum train: 1785\r\ntrain mean: 0.414565826331\r\nnum test: 894\r\ntest mean: 0.414988814318\r\n```\r\n\r\nWhy is this? Is this a bug? Does train_test_split fail with too many values?"},"comment":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/comments/390804663","html_url":"https://github.com/scikit-learn/scikit-learn/issues/8755#issuecomment-390804663","issue_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/8755","id":390804663,"user":{"login":"amueller","id":449558,"avatar_url":"https://avatars3.githubusercontent.com/u/449558?v=4","gravatar_id":"","url":"https://api.github.com/users/amueller","html_url":"https://github.com/amueller","followers_url":"https://api.github.com/users/amueller/followers","following_url":"https://api.github.com/users/amueller/following{/other_user}","gists_url":"https://api.github.com/users/amueller/gists{/gist_id}","starred_url":"https://api.github.com/users/amueller/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/amueller/subscriptions","organizations_url":"https://api.github.com/users/amueller/orgs","repos_url":"https://api.github.com/users/amueller/repos","events_url":"https://api.github.com/users/amueller/events{/privacy}","received_events_url":"https://api.github.com/users/amueller/received_events","type":"User","site_admin":false},"created_at":"2018-05-21T22:42:03Z","updated_at":"2018-05-21T22:42:03Z","author_association":"MEMBER","body":"Could it be an issue in\r\n```python\r\n        class_indices = np.split(np.argsort(y_indices, kind='mergesort'),\r\n                                 np.cumsum(class_counts)[:-1])\r\n```\r\nmaybe converting class_counts to float will fix it?"}}
{"actor":{"display_login":"amueller"}}