{"action":"created","issue":{"url":"https://api.github.com/repos/ray-project/ray/issues/2931","repository_url":"https://api.github.com/repos/ray-project/ray","labels_url":"https://api.github.com/repos/ray-project/ray/issues/2931/labels{/name}","comments_url":"https://api.github.com/repos/ray-project/ray/issues/2931/comments","events_url":"https://api.github.com/repos/ray-project/ray/issues/2931/events","html_url":"https://github.com/ray-project/ray/issues/2931","id":362892042,"node_id":"MDU6SXNzdWUzNjI4OTIwNDI=","number":2931,"title":"Connection Issue - Reset by Peer","user":{"login":"BrianMiner","id":8807554,"node_id":"MDQ6VXNlcjg4MDc1NTQ=","avatar_url":"https://avatars3.githubusercontent.com/u/8807554?v=4","gravatar_id":"","url":"https://api.github.com/users/BrianMiner","html_url":"https://github.com/BrianMiner","followers_url":"https://api.github.com/users/BrianMiner/followers","following_url":"https://api.github.com/users/BrianMiner/following{/other_user}","gists_url":"https://api.github.com/users/BrianMiner/gists{/gist_id}","starred_url":"https://api.github.com/users/BrianMiner/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/BrianMiner/subscriptions","organizations_url":"https://api.github.com/users/BrianMiner/orgs","repos_url":"https://api.github.com/users/BrianMiner/repos","events_url":"https://api.github.com/users/BrianMiner/events{/privacy}","received_events_url":"https://api.github.com/users/BrianMiner/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-09-22T21:25:20Z","updated_at":"2018-10-21T17:40:03Z","closed_at":null,"author_association":"NONE","body":"The same code on the same Ec2 instance now throw the error below. It appears to come from the line in my code: register_trainable(\"exp\", obj_function)\r\n\r\nI have redis '2.10.6' installed.\r\n\r\nAny hints on the cause?\r\n\r\n```\r\nProcess STDOUT and STDERR is being redirected to /tmp/raylogs/.\r\nWaiting for redis server at 127.0.0.1:60723 to respond...\r\nWaiting for redis server at 127.0.0.1:48080 to respond...\r\nStarting local scheduler with the following resources: {'CPU': 96, 'GPU': 0}.\r\n\r\n======================================================================\r\nView the web UI at http://localhost:8889/notebooks/ray_ui33174.ipynb?token=74b2c0738a07e667d60580a4b2f24d7317954eaf830ab0ef\r\n======================================================================\r\n\r\n---------------------------------------------------------------------------\r\nConnectionResetError                      Traceback (most recent call last)\r\n~/anaconda3/lib/python3.6/site-packages/redis/connection.py in send_packed_command(self, command)\r\n    589             for item in command:\r\n--> 590                 self._sock.sendall(item)\r\n    591         except socket.timeout:\r\n\r\nConnectionResetError: [Errno 104] Connection reset by peer\r\n\r\nDuring handling of the above exception, another exception occurred:\r\n\r\nConnectionError                           Traceback (most recent call last)\r\n~/anaconda3/lib/python3.6/site-packages/redis/client.py in execute_command(self, *args, **options)\r\n    666         try:\r\n--> 667             connection.send_command(*args)\r\n    668             return self.parse_response(connection, command_name, **options)\r\n\r\n~/anaconda3/lib/python3.6/site-packages/redis/connection.py in send_command(self, *args)\r\n    609         \"Pack and send a command to the Redis server\"\r\n--> 610         self.send_packed_command(self.pack_command(*args))\r\n    611 \r\n\r\n~/anaconda3/lib/python3.6/site-packages/redis/connection.py in send_packed_command(self, command)\r\n    602             raise ConnectionError(\"Error %s while writing to socket. %s.\" %\r\n--> 603                                   (errno, errmsg))\r\n    604         except:\r\n\r\nConnectionError: Error 104 while writing to socket. Connection reset by peer.\r\n\r\nDuring handling of the above exception, another exception occurred:\r\n\r\nConnectionResetError                      Traceback (most recent call last)\r\n~/anaconda3/lib/python3.6/site-packages/redis/connection.py in send_packed_command(self, command)\r\n    589             for item in command:\r\n--> 590                 self._sock.sendall(item)\r\n    591         except socket.timeout:\r\n\r\nConnectionResetError: [Errno 104] Connection reset by peer\r\n\r\nDuring handling of the above exception, another exception occurred:\r\n\r\nConnectionError                           Traceback (most recent call last)\r\n<ipython-input-6-48c88d974e24> in <module>()\r\n     75         ray.init()\r\n     76 \r\n---> 77     register_trainable(\"exp\", obj_function) #registers the above config and the objective function\r\n     78 \r\n     79     hpo=HyperOptSearch(space, max_concurrent=4, reward_attr=\"neg_mean_loss\") #smaller is better for log loss\r\n\r\n~/anaconda3/lib/python3.6/site-packages/ray/tune/registry.py in register_trainable(name, trainable)\r\n     36         raise TypeError(\"Second argument must be convertable to Trainable\",\r\n     37                         trainable)\r\n---> 38     _global_registry.register(TRAINABLE_CLASS, name, trainable)\r\n     39 \r\n     40 \r\n\r\n~/anaconda3/lib/python3.6/site-packages/ray/tune/registry.py in register(self, category, key, value)\r\n     77         self._to_flush[(category, key)] = pickle.dumps(value)\r\n     78         if _internal_kv_initialized():\r\n---> 79             self.flush_values()\r\n     80 \r\n     81     def contains(self, category, key):\r\n\r\n~/anaconda3/lib/python3.6/site-packages/ray/tune/registry.py in flush_values(self)\r\n     99     def flush_values(self):\r\n    100         for (category, key), value in self._to_flush.items():\r\n--> 101             _internal_kv_put(_make_key(category, key), value, overwrite=True)\r\n    102         self._to_flush.clear()\r\n    103 \r\n\r\n~/anaconda3/lib/python3.6/site-packages/ray/experimental/internal_kv.py in _internal_kv_put(key, value, overwrite)\r\n     29     worker = ray.worker.get_global_worker()\r\n     30     if overwrite:\r\n---> 31         updated = worker.redis_client.hset(key, \"value\", value)\r\n     32     else:\r\n     33         updated = worker.redis_client.hsetnx(key, \"value\", value)\r\n\r\n~/anaconda3/lib/python3.6/site-packages/ray/utils.py in _wrapper(*args, **kwargs)\r\n    322                 def _wrapper(*args, **kwargs):\r\n    323                     with self.lock:\r\n--> 324                         return orig_attr(*args, **kwargs)\r\n    325 \r\n    326                 self._wrapper_cache[attr] = _wrapper\r\n\r\n~/anaconda3/lib/python3.6/site-packages/redis/client.py in hset(self, name, key, value)\r\n   1990         Returns 1 if HSET created a new field, otherwise 0\r\n   1991         \"\"\"\r\n-> 1992         return self.execute_command('HSET', name, key, value)\r\n   1993 \r\n   1994     def hsetnx(self, name, key, value):\r\n\r\n~/anaconda3/lib/python3.6/site-packages/redis/client.py in execute_command(self, *args, **options)\r\n    671             if not connection.retry_on_timeout and isinstance(e, TimeoutError):\r\n    672                 raise\r\n--> 673             connection.send_command(*args)\r\n    674             return self.parse_response(connection, command_name, **options)\r\n    675         finally:\r\n\r\n~/anaconda3/lib/python3.6/site-packages/redis/connection.py in send_command(self, *args)\r\n    608     def send_command(self, *args):\r\n    609         \"Pack and send a command to the Redis server\"\r\n--> 610         self.send_packed_command(self.pack_command(*args))\r\n    611 \r\n    612     def can_read(self, timeout=0):\r\n\r\n~/anaconda3/lib/python3.6/site-packages/redis/connection.py in send_packed_command(self, command)\r\n    601                 errmsg = e.args[1]\r\n    602             raise ConnectionError(\"Error %s while writing to socket. %s.\" %\r\n--> 603                                   (errno, errmsg))\r\n    604         except:\r\n    605             self.disconnect()\r\n\r\nConnectionError: Error 104 while writing to socket. Connection reset by peer.\r\n\r\n```"},"comment":{"url":"https://api.github.com/repos/ray-project/ray/issues/comments/431688398","html_url":"https://github.com/ray-project/ray/issues/2931#issuecomment-431688398","issue_url":"https://api.github.com/repos/ray-project/ray/issues/2931","id":431688398,"node_id":"MDEyOklzc3VlQ29tbWVudDQzMTY4ODM5OA==","user":{"login":"stvreumi","id":15280812,"node_id":"MDQ6VXNlcjE1MjgwODEy","avatar_url":"https://avatars2.githubusercontent.com/u/15280812?v=4","gravatar_id":"","url":"https://api.github.com/users/stvreumi","html_url":"https://github.com/stvreumi","followers_url":"https://api.github.com/users/stvreumi/followers","following_url":"https://api.github.com/users/stvreumi/following{/other_user}","gists_url":"https://api.github.com/users/stvreumi/gists{/gist_id}","starred_url":"https://api.github.com/users/stvreumi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stvreumi/subscriptions","organizations_url":"https://api.github.com/users/stvreumi/orgs","repos_url":"https://api.github.com/users/stvreumi/repos","events_url":"https://api.github.com/users/stvreumi/events{/privacy}","received_events_url":"https://api.github.com/users/stvreumi/received_events","type":"User","site_admin":false},"created_at":"2018-10-21T17:40:03Z","updated_at":"2018-10-21T17:40:03Z","author_association":"NONE","body":"Hello @robertnishihara, I have same issue here.\r\nWhen I using my own tensorflow code with `ray.tune`, it give me the error message:\r\n```\r\nredis.exceptions.ConnectionError: Error 104 while writing to socket. Connection reset by peer.\r\n```\r\nBut when I running `tune_mnist_ray.py` in my computer, it works well and complete in the end.\r\nin the `tmp/raylogs` of the `redis-*` file, it say something like this:\r\n```\r\n36799:M 21 Oct 17:23:46.804 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.\r\n36799:M 21 Oct 17:23:46.804 # Server started, Redis version 3.9.102\r\n36799:M 21 Oct 17:23:46.804 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.\r\n36799:M 21 Oct 17:23:46.804 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.\r\n```\r\nIs there any other info I can give you so that you can help us fix this bug?"}}
{"actor":{"display_login":"stvreumi"}}