{"action":"created","issue":{"url":"https://api.github.com/repos/ray-project/ray/issues/1565","repository_url":"https://api.github.com/repos/ray-project/ray","labels_url":"https://api.github.com/repos/ray-project/ray/issues/1565/labels{/name}","comments_url":"https://api.github.com/repos/ray-project/ray/issues/1565/comments","events_url":"https://api.github.com/repos/ray-project/ray/issues/1565/events","html_url":"https://github.com/ray-project/ray/issues/1565","id":298400374,"node_id":"MDU6SXNzdWUyOTg0MDAzNzQ=","number":1565,"title":"ArrowIOError during training","user":{"login":"ericl","id":14922,"node_id":"MDQ6VXNlcjE0OTIy","avatar_url":"https://avatars3.githubusercontent.com/u/14922?v=4","gravatar_id":"","url":"https://api.github.com/users/ericl","html_url":"https://github.com/ericl","followers_url":"https://api.github.com/users/ericl/followers","following_url":"https://api.github.com/users/ericl/following{/other_user}","gists_url":"https://api.github.com/users/ericl/gists{/gist_id}","starred_url":"https://api.github.com/users/ericl/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ericl/subscriptions","organizations_url":"https://api.github.com/users/ericl/orgs","repos_url":"https://api.github.com/users/ericl/repos","events_url":"https://api.github.com/users/ericl/events{/privacy}","received_events_url":"https://api.github.com/users/ericl/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2018-02-19T21:03:08Z","updated_at":"2018-09-03T00:36:08Z","closed_at":"2018-07-19T12:37:46Z","author_association":"CONTRIBUTOR","body":"<!--\r\nGeneral questions should be asked on the mailing list ray-dev@googlegroups.com.\r\n\r\nBefore submitting an issue, please fill out the following form.\r\n-->\r\n\r\n### System information\r\n- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Ubuntu 16.04\r\n- **Ray installed from (source or binary)**: source\r\n- **Ray version**: master\r\n- **Python version**: 2.7\r\n- **Exact command to reproduce**: start apex training with 4 concurrent experiments; wait ~10 hours. Note that this doesn't seem to happen with 1 concurrent experiment.\r\n\r\n<!--\r\nYou can obtain the Ray version with\r\n\r\npython -c \"import ray; print(ray.__version__)\"\r\n-->\r\n\r\n### Describe the problem\r\nI've seen this happen a few times after several hours at high load (near 10Gbps network usage by plasma manager)\r\n\r\n### Source code / logs\r\n```\r\nTraceback (most recent call last):\r\n  File \"./train.py\", line 83, in <module>\r\n    run_experiments(experiments, scheduler=_make_scheduler(args))\r\n  File \"/home/ubuntu/ray/python/ray/tune/tune.py\", line 54, in run_experiments\r\n    runner.step()\r\n  File \"/home/ubuntu/ray/python/ray/tune/trial_runner.py\", line 91, in step\r\n    self._process_events()\r\n  File \"/home/ubuntu/ray/python/ray/tune/trial_runner.py\", line 213, in _process_events\r\n    [result_id], _ = ray.wait(list(self._running))\r\n  File \"/home/ubuntu/ray/python/ray/worker.py\", line 2337, in wait\r\n    num_returns)\r\n  File \"plasma.pyx\", line 595, in pyarrow.plasma.PlasmaClient.wait\r\n  File \"error.pxi\", line 79, in pyarrow.lib.check_status\r\npyarrow.lib.ArrowIOError: Encountered unexpected EOF\r\nRemote function train failed with:\r\n\r\nTraceback (most recent call last):\r\n  File \"/home/ubuntu/ray/python/ray/worker.py\", line 762, in _process_task\r\n    *arguments)\r\n  File \"/home/ubuntu/ray/python/ray/actor.py\", line 209, in actor_method_executor\r\n    method_returns = method(actor, *args)\r\n  File \"/home/ubuntu/ray/python/ray/tune/trainable.py\", line 107, in train\r\n    result = self._train()\r\n  File \"/home/ubuntu/ray/python/ray/rllib/dqn/dqn.py\", line 205, in _train\r\n    self.global_timestep += self.optimizer.step()\r\n  File \"/home/ubuntu/ray/python/ray/rllib/optimizers/apex_optimizer.py\", line 224, in step\r\n    sample_timesteps, train_timesteps = self._step()\r\n  File \"/home/ubuntu/ray/python/ray/rllib/optimizers/apex_optimizer.py\", line 244, in _step\r\n    for ev, sample_batch in self.sample_tasks.completed():\r\n  File \"/home/ubuntu/ray/python/ray/rllib/optimizers/apex_optimizer.py\", line 36, in completed\r\n    ready, _ = ray.wait(pending, num_returns=len(pending), timeout=10)\r\n  File \"/home/ubuntu/ray/python/ray/worker.py\", line 2337, in wait\r\n    num_returns)\r\n  File \"plasma.pyx\", line 595, in pyarrow.plasma.PlasmaClient.wait\r\n  File \"error.pxi\", line 79, in pyarrow.lib.check_status\r\nArrowIOError: Broken pipe\r\n```"},"comment":{"url":"https://api.github.com/repos/ray-project/ray/issues/comments/417971820","html_url":"https://github.com/ray-project/ray/issues/1565#issuecomment-417971820","issue_url":"https://api.github.com/repos/ray-project/ray/issues/1565","id":417971820,"node_id":"MDEyOklzc3VlQ29tbWVudDQxNzk3MTgyMA==","user":{"login":"whikwon","id":30768596,"node_id":"MDQ6VXNlcjMwNzY4NTk2","avatar_url":"https://avatars1.githubusercontent.com/u/30768596?v=4","gravatar_id":"","url":"https://api.github.com/users/whikwon","html_url":"https://github.com/whikwon","followers_url":"https://api.github.com/users/whikwon/followers","following_url":"https://api.github.com/users/whikwon/following{/other_user}","gists_url":"https://api.github.com/users/whikwon/gists{/gist_id}","starred_url":"https://api.github.com/users/whikwon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/whikwon/subscriptions","organizations_url":"https://api.github.com/users/whikwon/orgs","repos_url":"https://api.github.com/users/whikwon/repos","events_url":"https://api.github.com/users/whikwon/events{/privacy}","received_events_url":"https://api.github.com/users/whikwon/received_events","type":"User","site_admin":false},"created_at":"2018-09-03T00:36:08Z","updated_at":"2018-09-03T00:36:08Z","author_association":"NONE","body":"I got the same issue during training. \r\n![pyarrow_error](https://user-images.githubusercontent.com/30768596/44962506-9b2bef80-af5c-11e8-95a9-a989409c25a7.png)\r\n"}}
{"actor":{"display_login":"whikwon"}}