{"action":"created","issue":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/6943","repository_url":"https://api.github.com/repos/scikit-learn/scikit-learn","labels_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/6943/labels{/name}","comments_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/6943/comments","events_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/6943/events","html_url":"https://github.com/scikit-learn/scikit-learn/issues/6943","id":162452493,"number":6943,"title":"Some processes not working under clustering.MeanShift","user":{"login":"martinosorb","id":11293950,"avatar_url":"https://avatars.githubusercontent.com/u/11293950?v=3","gravatar_id":"","url":"https://api.github.com/users/martinosorb","html_url":"https://github.com/martinosorb","followers_url":"https://api.github.com/users/martinosorb/followers","following_url":"https://api.github.com/users/martinosorb/following{/other_user}","gists_url":"https://api.github.com/users/martinosorb/gists{/gist_id}","starred_url":"https://api.github.com/users/martinosorb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/martinosorb/subscriptions","organizations_url":"https://api.github.com/users/martinosorb/orgs","repos_url":"https://api.github.com/users/martinosorb/repos","events_url":"https://api.github.com/users/martinosorb/events{/privacy}","received_events_url":"https://api.github.com/users/martinosorb/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2016-06-27T13:31:31Z","updated_at":"2016-06-28T09:49:29Z","closed_at":null,"body":"Hi,\r\nI'm using the parallel version of clustering.MeanShift (which I had written, interestingly). I've now noticed that most of the processes are actually \"sleeping\", and only a few actually work. Even more oddly, this doesn't always happen:\r\n - the problem is worse on some machine than on others\r\n - the problem doesn't seem to appear when working with 2 dimensions instead of 4 (see code below).\r\n - changing the code to use `multiprocessing` instead of `joblib` makes it work\r\n\r\n\r\nI have no idea where to start...\r\n\r\n#### Reproduce\r\nWhen running the code\r\n\r\n```python\r\nfrom sklearn.cluster import MeanShift\r\nimport numpy as np\r\n\r\nndim = 4\r\npoints = np.random.random([100000, ndim])\r\n\r\nMS = MeanShift(n_jobs=20, bandwidth=0.1)\r\nprint(\"Starting.\")\r\nMS.fit(points)\r\n```\r\n\r\na call to `htop` shows:\r\n\r\n![screenshot from 2016-06-27 14-11-33](https://cloud.githubusercontent.com/assets/11293950/16381147/feccc14a-3c72-11e6-8de6-e3d570d7ee2b.png)\r\n\r\n\r\n#### Versions\r\n\r\nLinux-2.6.32-573.3.1.el6.x86_64-x86_64-with-redhat-6.6-Carbon\r\nPython 3.4.2 (default, Feb  4 2015, 08:24:27) \r\n[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)]\r\nNumPy 1.11.1\r\nSciPy 0.17.1\r\nScikit-Learn 0.17.1\r\n\r\n\r\n\r\n<!-- Thanks for contributing! -->\r\n"},"comment":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/comments/229003483","html_url":"https://github.com/scikit-learn/scikit-learn/issues/6943#issuecomment-229003483","issue_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/6943","id":229003483,"user":{"login":"lesteve","id":1680079,"avatar_url":"https://avatars.githubusercontent.com/u/1680079?v=3","gravatar_id":"","url":"https://api.github.com/users/lesteve","html_url":"https://github.com/lesteve","followers_url":"https://api.github.com/users/lesteve/followers","following_url":"https://api.github.com/users/lesteve/following{/other_user}","gists_url":"https://api.github.com/users/lesteve/gists{/gist_id}","starred_url":"https://api.github.com/users/lesteve/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lesteve/subscriptions","organizations_url":"https://api.github.com/users/lesteve/orgs","repos_url":"https://api.github.com/users/lesteve/repos","events_url":"https://api.github.com/users/lesteve/events{/privacy}","received_events_url":"https://api.github.com/users/lesteve/received_events","type":"User","site_admin":false},"created_at":"2016-06-28T09:49:29Z","updated_at":"2016-06-28T09:49:29Z","body":"So it seems like the automatic batching of tasks is not well suited to some machines. I am not exactly sure why yet.\r\n\r\nA work-around that works for me is to set joblib.parallel.MIN_IDEAL_BATCH_DURATION to a higher value. If you can test whether this snippet works for you, that'd be great:\r\n\r\n```py\r\n\r\nimport numpy as np\r\n\r\nfrom sklearn.cluster import MeanShift\r\nfrom sklearn.externals.joblib import parallel\r\n\r\nparallel.MIN_IDEAL_BATCH_DURATION = 1.\r\nparallel.MAX_IDEAL_BATCH_DURATION = parallel.MIN_IDEAL_BATCH_DURATION * 10\r\n\r\nndim = 4\r\npoints = np.random.random([100000, ndim])\r\n\r\nMS = MeanShift(n_jobs=20, bandwidth=0.1)\r\nprint(\"Starting.\")\r\nMS.fit(points)\r\n```"}}
{"actor":{"display_login":"lesteve"}}