{"action":"closed","issue":{"url":"https://api.github.com/repos/ray-project/ray/issues/3520","repository_url":"https://api.github.com/repos/ray-project/ray","labels_url":"https://api.github.com/repos/ray-project/ray/issues/3520/labels{/name}","comments_url":"https://api.github.com/repos/ray-project/ray/issues/3520/comments","events_url":"https://api.github.com/repos/ray-project/ray/issues/3520/events","html_url":"https://github.com/ray-project/ray/issues/3520","id":390149190,"node_id":"MDU6SXNzdWUzOTAxNDkxOTA=","number":3520,"title":"Segmentation fault with ray 0.6.0 and PyTorch 0.4.1 and 1.0.0","user":{"login":"TinkeringCode","id":41548354,"node_id":"MDQ6VXNlcjQxNTQ4MzU0","avatar_url":"https://avatars0.githubusercontent.com/u/41548354?v=4","gravatar_id":"","url":"https://api.github.com/users/TinkeringCode","html_url":"https://github.com/TinkeringCode","followers_url":"https://api.github.com/users/TinkeringCode/followers","following_url":"https://api.github.com/users/TinkeringCode/following{/other_user}","gists_url":"https://api.github.com/users/TinkeringCode/gists{/gist_id}","starred_url":"https://api.github.com/users/TinkeringCode/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/TinkeringCode/subscriptions","organizations_url":"https://api.github.com/users/TinkeringCode/orgs","repos_url":"https://api.github.com/users/TinkeringCode/repos","events_url":"https://api.github.com/users/TinkeringCode/events{/privacy}","received_events_url":"https://api.github.com/users/TinkeringCode/received_events","type":"User","site_admin":false},"labels":[{"id":1119780425,"node_id":"MDU6TGFiZWwxMTE5NzgwNDI1","url":"https://api.github.com/repos/ray-project/ray/labels/stability-blocker","name":"stability-blocker","color":"fbca04","default":false}],"state":"closed","locked":false,"assignee":{"login":"pcmoritz","id":113316,"node_id":"MDQ6VXNlcjExMzMxNg==","avatar_url":"https://avatars1.githubusercontent.com/u/113316?v=4","gravatar_id":"","url":"https://api.github.com/users/pcmoritz","html_url":"https://github.com/pcmoritz","followers_url":"https://api.github.com/users/pcmoritz/followers","following_url":"https://api.github.com/users/pcmoritz/following{/other_user}","gists_url":"https://api.github.com/users/pcmoritz/gists{/gist_id}","starred_url":"https://api.github.com/users/pcmoritz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pcmoritz/subscriptions","organizations_url":"https://api.github.com/users/pcmoritz/orgs","repos_url":"https://api.github.com/users/pcmoritz/repos","events_url":"https://api.github.com/users/pcmoritz/events{/privacy}","received_events_url":"https://api.github.com/users/pcmoritz/received_events","type":"User","site_admin":false},"assignees":[{"login":"pcmoritz","id":113316,"node_id":"MDQ6VXNlcjExMzMxNg==","avatar_url":"https://avatars1.githubusercontent.com/u/113316?v=4","gravatar_id":"","url":"https://api.github.com/users/pcmoritz","html_url":"https://github.com/pcmoritz","followers_url":"https://api.github.com/users/pcmoritz/followers","following_url":"https://api.github.com/users/pcmoritz/following{/other_user}","gists_url":"https://api.github.com/users/pcmoritz/gists{/gist_id}","starred_url":"https://api.github.com/users/pcmoritz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pcmoritz/subscriptions","organizations_url":"https://api.github.com/users/pcmoritz/orgs","repos_url":"https://api.github.com/users/pcmoritz/repos","events_url":"https://api.github.com/users/pcmoritz/events{/privacy}","received_events_url":"https://api.github.com/users/pcmoritz/received_events","type":"User","site_admin":false}],"milestone":null,"comments":17,"created_at":"2018-12-12T10:01:40Z","updated_at":"2018-12-24T00:30:28Z","closed_at":"2018-12-24T00:30:27Z","author_association":"NONE","body":"### System information\r\n- **OS Platform and Distribution**: Ubuntu 16.04, Ubuntu 18.04, Amazon EC2 optimized Linux\r\n- **Ray installed from (source or binary)**: pip\r\n- **Ray version**: 0.6.0\r\n- **Python version**: 3.7 & 3.6\r\n\r\n### A minimal example  to reproduce:\r\n\r\n```\r\nimport ray\r\nimport torch\r\n\r\n\r\nclass NeuralNet(torch.nn.Module):\r\n    def __init__(self):\r\n        super().__init__()\r\n        self.l = torch.nn.Linear(2, 2)\r\n\r\n    def forward(self, x):\r\n        return self.l(x)\r\n\r\n\r\n@ray.remote\r\nclass TestActor:\r\n    def __init__(self):\r\n        self.net = NeuralNet()\r\n        self.crit = torch.nn.MSELoss()\r\n\r\n    def train(self):\r\n        p = self.net(torch.tensor([[1.0, 2.0]]))\r\n        loss = self.crit(p, torch.tensor([[3.0, 4.0]]))\r\n        self.net.zero_grad()\r\n        loss.backward()\r\n        return loss.item()\r\n\r\n\r\nif __name__ == '__main__':\r\n    ray.init()\r\n    ac = TestActor.remote()\r\n    print(ray.get(ac.train.remote()))\r\n```\r\n\r\n\r\n### Problem Description\r\nI clean installed PyTorch with conda on different OS and don't get Seg faults when not using ray. This may be a misunderstanding on my part as this seems like an issue that others would have met minutes after it was introduced... Isn't this how ray is supposed to be used when running locally?\r\n\r\n### Console Output\r\n```\r\nProcess STDOUT and STDERR is being redirected to /tmp/ray/session_2018-12-12_10-51-32_500/logs.\r\nWaiting for redis server at 127.0.0.1:31236 to respond...\r\nWaiting for redis server at 127.0.0.1:56879 to respond...\r\nStarting the Plasma object store with 13.470806835000001 GB memory using /dev/shm.\r\nFailed to start the UI, you may need to run 'pip install jupyter'.\r\nFatal Python error: Segmentation fault\r\n\r\nStack (most recent call first):\r\n  File \"/home/eric/miniconda3/envs/TorchRay/lib/python3.7/site-packages/torch/autograd/__init__.py\", line 90 in backward\r\n  File \"/home/eric/miniconda3/envs/TorchRay/lib/python3.7/site-packages/torch/tensor.py\", line 93 in backward\r\n  File \"/home/eric/PycharmProjects/rayTorchTest/test.py\", line 25 in train\r\n  File \"/home/eric/miniconda3/envs/TorchRay/lib/python3.7/site-packages/ray/function_manager.py\", line 481 in actor_method_executor\r\n  File \"/home/eric/miniconda3/envs/TorchRay/lib/python3.7/site-packages/ray/worker.py\", line 856 in _process_task\r\n  File \"/home/eric/miniconda3/envs/TorchRay/lib/python3.7/site-packages/ray/worker.py\", line 967 in _wait_for_and_process_task\r\n  File \"/home/eric/miniconda3/envs/TorchRay/lib/python3.7/site-packages/ray/worker.py\", line 1010 in main_loop\r\n  File \"/home/eric/miniconda3/envs/TorchRay/lib/python3.7/site-packages/ray/workers/default_worker.py\", line 99 in <module>\r\nA worker died or was killed while executing task 00000000d024403f9ddae404df35ac4a32625560.\r\nTraceback (most recent call last):\r\n  File \"/home/eric/PycharmProjects/rayTorchTest/test.py\", line 32, in <module>\r\n    print(ray.get(ac.train.remote()))\r\n  File \"/home/eric/miniconda3/envs/TorchRay/lib/python3.7/site-packages/ray/worker.py\", line 2366, in get\r\n    raise RayGetError(object_ids, value)\r\nray.worker.RayGetError: Could not get objectid ObjectID(01000000d024403f9ddae404df35ac4a32625560). It was created by remote function <unknown> which failed with:\r\n\r\nRemote function <unknown> failed with:\r\n\r\nInvalid return value: likely worker died or was killed while executing the task; check previous logs or dmesg for errors.\r\n\r\nProcess finished with exit code 1\r\n```\r\n\r\n### Edit 1:\r\nThe same script runs successfully with the following dependencies:\r\n```\r\npython==3.6.0\r\ntorch==0.4.1\r\nray==0.4.0\r\nredis==2.10.6\r\n```\r\nUpgrading to ray 0.5.3 throws another error that looks very similar (worker died) although it doesn't state that a segfault occurred. Upgrading to ray 0.6.0 causes the above-demonstrated segfault."}}
{"actor":{"display_login":"robertnishihara"}}