{"action":"created","issue":{"url":"https://api.github.com/repos/dask/distributed/issues/32","repository_url":"https://api.github.com/repos/dask/distributed","labels_url":"https://api.github.com/repos/dask/distributed/issues/32/labels{/name}","comments_url":"https://api.github.com/repos/dask/distributed/issues/32/comments","events_url":"https://api.github.com/repos/dask/distributed/issues/32/events","html_url":"https://github.com/dask/distributed/issues/32","id":119544315,"number":32,"title":"Slow inter-node communication","user":{"login":"broxtronix","id":63793,"avatar_url":"https://avatars.githubusercontent.com/u/63793?v=3","gravatar_id":"","url":"https://api.github.com/users/broxtronix","html_url":"https://github.com/broxtronix","followers_url":"https://api.github.com/users/broxtronix/followers","following_url":"https://api.github.com/users/broxtronix/following{/other_user}","gists_url":"https://api.github.com/users/broxtronix/gists{/gist_id}","starred_url":"https://api.github.com/users/broxtronix/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/broxtronix/subscriptions","organizations_url":"https://api.github.com/users/broxtronix/orgs","repos_url":"https://api.github.com/users/broxtronix/repos","events_url":"https://api.github.com/users/broxtronix/events{/privacy}","received_events_url":"https://api.github.com/users/broxtronix/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":13,"created_at":"2015-11-30T17:49:27Z","updated_at":"2016-04-07T03:03:36Z","closed_at":null,"body":"[ This issue began as a private email discussion between myself and @mrocklin, but we are moving it here so that we can better keep track of it. cc also @shoyer, @freeman-lab, @mrocklin  ]\r\n\r\n@mrocklin recently suggested that I try dask.imperative ([docs](http://dask.pydata.org/en/latest/imperative.html), [example](http://nbviewer.ipython.org/github/blaze/dask-examples/blob/master/do-and-profiler.ipynb)) for parallelizing complex computations that do not fit in the working memory of my cluster.  Distributed should be able to process a dask graph as follows:\r\n```\r\nmy_dask_value.compute(get=my_executor.get)\r\n```\r\nI tried this and it works great, but I am noticing some surprisingly slow performance when moderately sized results (100 x 2MB image arrays) are collect back from the workers to the executor.  The computation itself seems to run very quickly, but this \"gather\" operation takes longer than expected.  I created a notebook [here](https://gist.github.com/broxtronix/62f13db4d0869b7b3ad6) that illustrates this problem.\r\n\r\n@mrocklin tried my notebook and got similarly slow performance when using the `dask.multiprocessing` scheduler, but got much better results when using `distributed`.  He set up his cluster as follows:\r\n```\r\n$ dcenter\r\n$ dworker 127.0.0.1:8787 --ncores 1  # run this in four separate terminals\r\n```\r\nHis results are [here](https://gist.github.com/mrocklin/fd5d3c160a573d7d72cf).  However, when I tried setting up my cluster in this way, my benchmark for distributed is still clocking in at 1 min 22 seconds, while the threaded result runs in 10 seconds on my laptop.  \r\n\r\nThere must be something else afoot here since @mrocklin is getting much better performance than I am.  FWIW, I'm running the distributed master branch just after you merged in your recent scheduler changes (git hash: fad4ce342d9e935b87cbc99e5c2740299b40e17b) on an Apple laptop running OSX, and Anaconda Python 3.5.0. "},"comment":{"url":"https://api.github.com/repos/dask/distributed/issues/comments/206673429","html_url":"https://github.com/dask/distributed/issues/32#issuecomment-206673429","issue_url":"https://api.github.com/repos/dask/distributed/issues/32","id":206673429,"user":{"login":"mrocklin","id":306380,"avatar_url":"https://avatars.githubusercontent.com/u/306380?v=3","gravatar_id":"","url":"https://api.github.com/users/mrocklin","html_url":"https://github.com/mrocklin","followers_url":"https://api.github.com/users/mrocklin/followers","following_url":"https://api.github.com/users/mrocklin/following{/other_user}","gists_url":"https://api.github.com/users/mrocklin/gists{/gist_id}","starred_url":"https://api.github.com/users/mrocklin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mrocklin/subscriptions","organizations_url":"https://api.github.com/users/mrocklin/orgs","repos_url":"https://api.github.com/users/mrocklin/repos","events_url":"https://api.github.com/users/mrocklin/events{/privacy}","received_events_url":"https://api.github.com/users/mrocklin/received_events","type":"User","site_admin":false},"created_at":"2016-04-07T03:03:36Z","updated_at":"2016-04-07T03:03:36Z","body":"I ran into this.  Tracked it down (I think) to a Tornado issue: https://github.com/tornadoweb/tornado/issues/1685"}}