{"action":"created","issue":{"url":"https://api.github.com/repos/dask/dask/issues/1411","repository_url":"https://api.github.com/repos/dask/dask","labels_url":"https://api.github.com/repos/dask/dask/issues/1411/labels{/name}","comments_url":"https://api.github.com/repos/dask/dask/issues/1411/comments","events_url":"https://api.github.com/repos/dask/dask/issues/1411/events","html_url":"https://github.com/dask/dask/issues/1411","id":167425270,"number":1411,"title":"leaking file descriptors","user":{"login":"akcheung","id":2018684,"avatar_url":"https://avatars.githubusercontent.com/u/2018684?v=3","gravatar_id":"","url":"https://api.github.com/users/akcheung","html_url":"https://github.com/akcheung","followers_url":"https://api.github.com/users/akcheung/followers","following_url":"https://api.github.com/users/akcheung/following{/other_user}","gists_url":"https://api.github.com/users/akcheung/gists{/gist_id}","starred_url":"https://api.github.com/users/akcheung/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/akcheung/subscriptions","organizations_url":"https://api.github.com/users/akcheung/orgs","repos_url":"https://api.github.com/users/akcheung/repos","events_url":"https://api.github.com/users/akcheung/events{/privacy}","received_events_url":"https://api.github.com/users/akcheung/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":23,"created_at":"2016-07-25T17:48:20Z","updated_at":"2016-07-27T19:04:17Z","closed_at":"2016-07-26T14:46:25Z","body":"I believe either the scheduler or the workers are not closing file descriptors appropriately after calling `scatter`. Sample code:\r\n\r\n```\r\nimport psutil\r\nprint(psutil.Process().num_fds())\r\n\r\nfrom distributed import Executor\r\nexecutor = Executor() \r\n\r\nimport numpy as np\r\nfrom dask import delayed, compute\r\n\r\nfor i in range(0,10):\r\n  #data = np.random.randint(0, 1000, size=(145, 174, 145, 288))\r\n  data = np.random.randint(0, 1000, size=(145, 174, 145, 2))\r\n  print(psutil.Process().num_fds())\r\n\r\n  remote_slices = executor.scatter([data[..., i] for i in range(data.shape[-1])])\r\n  print(psutil.Process().num_fds())\r\n\r\n  jobs = [delayed(len)(slice) for slice in remote_slices]\r\n  print(psutil.Process().num_fds())\r\n\r\n  j = compute(*jobs, get=executor.get)\r\n  print(psutil.Process().num_fds())\r\n```\r\n\r\nThis prints\r\n```\r\n4\r\n751\r\n803\r\n803\r\n805\r\n805\r\n822\r\n822\r\n824\r\n824\r\n831\r\n831\r\n833\r\n833\r\n852\r\n852\r\n854\r\n854\r\n871\r\n871\r\n873\r\n873\r\n875\r\n875\r\n877\r\n877\r\n879\r\n879\r\n881\r\n881\r\n883\r\n883\r\n885\r\n885\r\n887\r\n887\r\n889\r\n889\r\n891\r\n891\r\n893\r\n```\r\n\r\nAnd if you use the commented out line to generate data then the program crashes with `Too many open files` on my machine."},"comment":{"url":"https://api.github.com/repos/dask/dask/issues/comments/235686966","html_url":"https://github.com/dask/dask/issues/1411#issuecomment-235686966","issue_url":"https://api.github.com/repos/dask/dask/issues/1411","id":235686966,"user":{"login":"akcheung","id":2018684,"avatar_url":"https://avatars.githubusercontent.com/u/2018684?v=3","gravatar_id":"","url":"https://api.github.com/users/akcheung","html_url":"https://github.com/akcheung","followers_url":"https://api.github.com/users/akcheung/followers","following_url":"https://api.github.com/users/akcheung/following{/other_user}","gists_url":"https://api.github.com/users/akcheung/gists{/gist_id}","starred_url":"https://api.github.com/users/akcheung/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/akcheung/subscriptions","organizations_url":"https://api.github.com/users/akcheung/orgs","repos_url":"https://api.github.com/users/akcheung/repos","events_url":"https://api.github.com/users/akcheung/events{/privacy}","received_events_url":"https://api.github.com/users/akcheung/received_events","type":"User","site_admin":false},"created_at":"2016-07-27T19:04:17Z","updated_at":"2016-07-27T19:04:17Z","body":"Here's the full trace:\r\n```\r\nTraceback (most recent call last):\r\n  File \"test.py\", line 17, in <module>\r\n    remote_slices = executor.scatter([data[..., i] for i in range(data.shape[-1])])\r\n  File \"/usr/local/lib/python3.4/dist-packages/distributed/executor.py\", line 889, in scatter\r\n    broadcast=broadcast)\r\n  File \"/usr/local/lib/python3.4/dist-packages/distributed/utils.py\", line 116, in sync\r\n    six.reraise(type(error[0]), error[0], traceback[0])\r\n  File \"/usr/local/lib/python3.4/dist-packages/six.py\", line 686, in reraise\r\n    raise value\r\n  File \"/usr/local/lib/python3.4/dist-packages/distributed/utils.py\", line 102, in f\r\n    result[0] = yield gen.maybe_future(func(*args, **kwargs))\r\n  File \"/usr/local/lib/python3.4/dist-packages/tornado/gen.py\", line 1015, in run\r\n    value = future.result()\r\n  File \"/usr/local/lib/python3.4/dist-packages/tornado/concurrent.py\", line 237, in result\r\n    raise_exc_info(self._exc_info)\r\n  File \"<string>\", line 3, in raise_exc_info\r\n  File \"/usr/local/lib/python3.4/dist-packages/tornado/gen.py\", line 1021, in run\r\n    yielded = self.gen.throw(*exc_info)\r\n  File \"/usr/local/lib/python3.4/dist-packages/distributed/executor.py\", line 787, in _scatter\r\n    client=self.id, broadcast=broadcast)\r\n  File \"/usr/local/lib/python3.4/dist-packages/tornado/gen.py\", line 1015, in run\r\n    value = future.result()\r\n  File \"/usr/local/lib/python3.4/dist-packages/tornado/concurrent.py\", line 237, in result\r\n    raise_exc_info(self._exc_info)\r\n  File \"<string>\", line 3, in raise_exc_info\r\n  File \"/usr/local/lib/python3.4/dist-packages/tornado/gen.py\", line 1021, in run\r\n    yielded = self.gen.throw(*exc_info)\r\n  File \"/usr/local/lib/python3.4/dist-packages/distributed/core.py\", line 438, in send_recv_from_rpc\r\n    result = yield send_recv(stream=stream, op=key, **kwargs)\r\n  File \"/usr/local/lib/python3.4/dist-packages/tornado/gen.py\", line 1015, in run\r\n    value = future.result()\r\n  File \"/usr/local/lib/python3.4/dist-packages/tornado/concurrent.py\", line 237, in result\r\n    raise_exc_info(self._exc_info)\r\n  File \"<string>\", line 3, in raise_exc_info\r\n  File \"/usr/local/lib/python3.4/dist-packages/tornado/gen.py\", line 1021, in run\r\n    yielded = self.gen.throw(*exc_info)\r\n  File \"/usr/local/lib/python3.4/dist-packages/distributed/core.py\", line 335, in send_recv\r\n    yield write(stream, msg)\r\n  File \"/usr/local/lib/python3.4/dist-packages/tornado/gen.py\", line 1015, in run\r\n    value = future.result()\r\n  File \"/usr/local/lib/python3.4/dist-packages/tornado/concurrent.py\", line 237, in result\r\n    raise_exc_info(self._exc_info)\r\n  File \"<string>\", line 3, in raise_exc_info\r\n  File \"/usr/local/lib/python3.4/dist-packages/tornado/gen.py\", line 285, in wrapper\r\n    yielded = next(result)\r\n  File \"/usr/local/lib/python3.4/dist-packages/distributed/core.py\", line 269, in write\r\n    frames = protocol.dumps(msg)\r\n  File \"/usr/local/lib/python3.4/dist-packages/distributed/protocol.py\", line 81, in dumps\r\n    frames = dumps_msgpack(small)\r\n  File \"/usr/local/lib/python3.4/dist-packages/distributed/protocol.py\", line 155, in dumps_msgpack\r\n    payload = msgpack.dumps(msg, use_bin_type=True)\r\n  File \"/usr/local/lib/python3.4/dist-packages/msgpack/__init__.py\", line 47, in packb\r\n    return Packer(**kwargs).pack(o)\r\nSystemError: error return without exception set\r\n```"}}
{"actor":{"display_login":"akcheung"}}