{"action":"created","issue":{"url":"https://api.github.com/repos/ray-project/ray/issues/3200","repository_url":"https://api.github.com/repos/ray-project/ray","labels_url":"https://api.github.com/repos/ray-project/ray/issues/3200/labels{/name}","comments_url":"https://api.github.com/repos/ray-project/ray/issues/3200/comments","events_url":"https://api.github.com/repos/ray-project/ray/issues/3200/events","html_url":"https://github.com/ray-project/ray/issues/3200","id":376705587,"node_id":"MDU6SXNzdWUzNzY3MDU1ODc=","number":3200,"title":"A simple job will hang with raylet","user":{"login":"adoda","id":35217964,"node_id":"MDQ6VXNlcjM1MjE3OTY0","avatar_url":"https://avatars3.githubusercontent.com/u/35217964?v=4","gravatar_id":"","url":"https://api.github.com/users/adoda","html_url":"https://github.com/adoda","followers_url":"https://api.github.com/users/adoda/followers","following_url":"https://api.github.com/users/adoda/following{/other_user}","gists_url":"https://api.github.com/users/adoda/gists{/gist_id}","starred_url":"https://api.github.com/users/adoda/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/adoda/subscriptions","organizations_url":"https://api.github.com/users/adoda/orgs","repos_url":"https://api.github.com/users/adoda/repos","events_url":"https://api.github.com/users/adoda/events{/privacy}","received_events_url":"https://api.github.com/users/adoda/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2018-11-02T08:01:18Z","updated_at":"2018-11-14T01:53:40Z","closed_at":"2018-11-10T23:16:43Z","author_association":"CONTRIBUTOR","body":"<!--\r\nGeneral questions should be asked on the mailing list ray-dev@googlegroups.com.\r\n\r\nBefore submitting an issue, please fill out the following form.\r\n-->\r\n\r\n### System information\r\n- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Linux Red Hat 4.8.5-4\r\n- **Ray installed from (source or binary)**: source\r\n- **Ray version**: master with commit: e49839c73f16489ed2baef55861e9b4e6e20814d\r\n- **Python version**: python2.7\r\n- **Exact command to reproduce**: execute the simple job in ray cluster\r\n\r\n<!--\r\nYou can obtain the Ray version with\r\n\r\npython -c \"import ray; print(ray.__version__)\"\r\n-->\r\n\r\n### Describe the problem\r\n<!-- Describe the problem clearly here. -->\r\n\r\nWe execute a simple job in ray cluster with 3 nodes, each node with resource like:\r\n```\r\ncpu: 10 cores\r\nmemory: 30GB\r\n```\r\n\r\nThe job like:\r\n```\r\nimport time\r\nimport ray\r\nimport os\r\n\r\nenv_dist = os.environ\r\nredis_address = env_dist.get('REDIS_ADDRESS')\r\n\r\nray.init(redis_address=redis_address)\r\n\r\n@ray.remote\r\ndef f():\r\n  time.sleep(0.01)\r\n  return ray.services.get_node_ip_address()\r\n\r\n# Get a list of the IP addresses of the nodes that have joined the cluster.\r\nip_list = set(ray.get([f.remote() for _ in range(100000)]))\r\nprint(\"ip_list\", ip_list)\r\n```\r\n\r\nThe job will be hang forever， and will print the info after tens minutes: \r\n```\r\nThe node with client ID 7addcc81bb1ef8d170875e27635197c9b3d680cd has been marked dead because the monitor has missed too many heartbeats from it.\r\n```\r\n\r\nWe can found some log in raylet_0.err like:\r\n```\r\nW1102 15:20:24.683997   402 node_manager.cc:232] Last heartbeat was sent 562 ms ago\r\nW1102 15:20:36.525331   402 node_manager.cc:232] Last heartbeat was sent 579 ms ago\r\nW1102 15:20:48.447289   402 node_manager.cc:232] Last heartbeat was sent 556 ms ago\r\nF1102 15:21:11.764168   402 node_manager.cc:342]  Check failed: client_id != gcs_client_->client_table().GetLocalClientId() Exiting because this node manager has mistakenly been marked dead by the monitor.\r\n*** Check failure stack trace: ***\r\n    @           0x558b78  google::LogMessage::Fail()\r\n    @           0x558ad4  google::LogMessage::SendToLog()\r\n    @           0x558458  google::LogMessage::Flush()\r\n    @           0x558267  google::LogMessage::~LogMessage()\r\n    @           0x4ca428  ray::RayLog::~RayLog()\r\n    @           0x4f77a2  ray::raylet::NodeManager::ClientRemoved()\r\n    @           0x4a0a71  ray::gcs::ClientTable::HandleNotification()\r\n    @           0x4a131b  _ZNSt17_Function_handlerIFvPN3ray3gcs14AsyncGcsClientERKNS0_8UniqueIDERKSt6vectorI16ClientTableDataTSaIS8_EEEZZNS1_11ClientTable7ConnectERKS8_ENKUlS3_S6_SG_E_clES3_S6_SG_EUlS3_S6_SC_E_E9_M_invokeERKSt9_Any_dataS3_S6_SC_\r\n    @           0x4af27c  _ZZN3ray3gcs3LogINS_8UniqueIDE15ClientTableDataE9SubscribeERKS2_S6_RKSt8functionIFvPNS0_14AsyncGcsClientES6_RKSt6vectorI16ClientTableDataTSaISB_EEEERKS7_IFvS9_EEENKUlRKSsE_clESP_\r\n    @           0x4c5ff9  (anonymous namespace)::ProcessCallback()\r\n    @           0x4c6f18  ray::gcs::SubscribeRedisCallback()\r\n    @           0x510c56  redisProcessCallbacks\r\n    @           0x4c9eb3  RedisAsioClient::handle_read()\r\n    @           0x4ca341  boost::asio::detail::reactive_null_buffers_op<>::do_complete()\r\n    @           0x48b7a1  boost::asio::detail::task_io_service::run()\r\n    @           0x483714  main\r\n    @     0x7f53a10f3b15  __libc_start_main\r\n    @           0x487d09  (unknown)\r\n    @              (nil)  (unknown)\r\n```\r\n\r\nThis simple job will run ok with the version of ray some weeks ago (without raylet).\r\n\r\nAnd if we decrease the times from 100,000 to 10,000, the job can finished.\r\n### Source code / logs\r\n<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->\r\n"},"comment":{"url":"https://api.github.com/repos/ray-project/ray/issues/comments/438508258","html_url":"https://github.com/ray-project/ray/issues/3200#issuecomment-438508258","issue_url":"https://api.github.com/repos/ray-project/ray/issues/3200","id":438508258,"node_id":"MDEyOklzc3VlQ29tbWVudDQzODUwODI1OA==","user":{"login":"robertnishihara","id":249517,"node_id":"MDQ6VXNlcjI0OTUxNw==","avatar_url":"https://avatars3.githubusercontent.com/u/249517?v=4","gravatar_id":"","url":"https://api.github.com/users/robertnishihara","html_url":"https://github.com/robertnishihara","followers_url":"https://api.github.com/users/robertnishihara/followers","following_url":"https://api.github.com/users/robertnishihara/following{/other_user}","gists_url":"https://api.github.com/users/robertnishihara/gists{/gist_id}","starred_url":"https://api.github.com/users/robertnishihara/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/robertnishihara/subscriptions","organizations_url":"https://api.github.com/users/robertnishihara/orgs","repos_url":"https://api.github.com/users/robertnishihara/repos","events_url":"https://api.github.com/users/robertnishihara/events{/privacy}","received_events_url":"https://api.github.com/users/robertnishihara/received_events","type":"User","site_admin":false},"created_at":"2018-11-14T01:53:40Z","updated_at":"2018-11-14T01:53:40Z","author_association":"CONTRIBUTOR","body":"@adoda, did a Redis server die? Are there any interesting logs for the Redis servers?\r\n\r\nDid any processes die? The logs you posted for the plasma store looks fine. For the raylet it looks like something is taking too long, but it doesn't look like anything failed."}}
{"actor":{"display_login":"robertnishihara"}}