{"action":"created","issue":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/4619","repository_url":"https://api.github.com/repos/scikit-learn/scikit-learn","labels_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/4619/labels{/name}","comments_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/4619/comments","events_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/4619/events","html_url":"https://github.com/scikit-learn/scikit-learn/issues/4619","id":69396541,"number":4619,"title":"Python crashes when calculating large t-SNE","user":{"login":"michaelbrundage","id":552566,"avatar_url":"https://avatars.githubusercontent.com/u/552566?v=3","gravatar_id":"","url":"https://api.github.com/users/michaelbrundage","html_url":"https://github.com/michaelbrundage","followers_url":"https://api.github.com/users/michaelbrundage/followers","following_url":"https://api.github.com/users/michaelbrundage/following{/other_user}","gists_url":"https://api.github.com/users/michaelbrundage/gists{/gist_id}","starred_url":"https://api.github.com/users/michaelbrundage/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/michaelbrundage/subscriptions","organizations_url":"https://api.github.com/users/michaelbrundage/orgs","repos_url":"https://api.github.com/users/michaelbrundage/repos","events_url":"https://api.github.com/users/michaelbrundage/events{/privacy}","received_events_url":"https://api.github.com/users/michaelbrundage/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/labels/Enhancement","name":"Enhancement","color":"d7e102"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":10,"created_at":"2015-04-19T07:37:55Z","updated_at":"2016-03-11T02:12:48Z","closed_at":null,"body":"Python 2.7.9, scikit-learn 0.16.1\r\nMacBook Pro, 16GB RAM, MacOS 10.10.3\r\n\r\nWhen running the following code in an iPython notebook, it runs for a long time producing no output, and then the iPython kernel crashes and has to restart.\r\n\r\n```python\r\nimport numpy as np\r\nfrom sklearn.manifold import TSNE\r\n\r\nX = np.random.random((100000, 100))\r\ntsne = TSNE(n_components=2, perplexity=40, learning_rate=100, verbose=2).fit_transform(X)\r\n```\r\n\r\nRunning the same code from the command line produces one line of output\r\n\r\n`[t-SNE] Computing pairwise distances...`\r\n\r\nand then it dies.\r\n\r\nThe error is pretty much the same as reported in [this older issue](https://github.com/scikit-learn/scikit-learn/issues/3088). I'll attach a crash log next, but I think it's out of memory because if I reduce the rows of X to 10000, then it runs to completion without error.\r\n\r\nSome suggested improvements:\r\n\r\n* Estimate the memory required to carry out this operation and warn or error out earlier.\r\n* Try to recover from OOM errors in libBLAS more gracefully.\r\n* If possible, reduce the memory requirements of sklearn.manifold.TSNE or implement out-of-core\r\n* Document the memory requirements in sklearn.manifold (and sklearn.decomposition), and perhaps warn about potential OOM errors and what they'll look like to users. You've already nicely documented the runtime complexities of the algorithms but have not documented their space complexities.\r\n"},"comment":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/comments/195145072","html_url":"https://github.com/scikit-learn/scikit-learn/issues/4619#issuecomment-195145072","issue_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/4619","id":195145072,"user":{"login":"philipperemy","id":4516927,"avatar_url":"https://avatars.githubusercontent.com/u/4516927?v=3","gravatar_id":"","url":"https://api.github.com/users/philipperemy","html_url":"https://github.com/philipperemy","followers_url":"https://api.github.com/users/philipperemy/followers","following_url":"https://api.github.com/users/philipperemy/following{/other_user}","gists_url":"https://api.github.com/users/philipperemy/gists{/gist_id}","starred_url":"https://api.github.com/users/philipperemy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/philipperemy/subscriptions","organizations_url":"https://api.github.com/users/philipperemy/orgs","repos_url":"https://api.github.com/users/philipperemy/repos","events_url":"https://api.github.com/users/philipperemy/events{/privacy}","received_events_url":"https://api.github.com/users/philipperemy/received_events","type":"User","site_admin":false},"created_at":"2016-03-11T02:12:48Z","updated_at":"2016-03-11T02:12:48Z","body":"@michaelbrundage Thanks for your prompt answer! Just to let you know, you can use the python wrapper of the C++ implementation of the Bhtsne available on Laurent's GitHub. It tested it with (300k, 50) and it works good! Maybe not as tunable as the one in Sklearn but it does the job pretty well."}}