{"action":"created","issue":{"url":"https://api.github.com/repos/dask/distributed/issues/1772","repository_url":"https://api.github.com/repos/dask/distributed","labels_url":"https://api.github.com/repos/dask/distributed/issues/1772/labels{/name}","comments_url":"https://api.github.com/repos/dask/distributed/issues/1772/comments","events_url":"https://api.github.com/repos/dask/distributed/issues/1772/events","html_url":"https://github.com/dask/distributed/issues/1772","id":298339153,"number":1772,"title":"Failed to start workers on 2nd platform after upgrading to latest GIT status","user":{"login":"BnY","id":157254,"avatar_url":"https://avatars1.githubusercontent.com/u/157254?v=4","gravatar_id":"","url":"https://api.github.com/users/BnY","html_url":"https://github.com/BnY","followers_url":"https://api.github.com/users/BnY/followers","following_url":"https://api.github.com/users/BnY/following{/other_user}","gists_url":"https://api.github.com/users/BnY/gists{/gist_id}","starred_url":"https://api.github.com/users/BnY/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/BnY/subscriptions","organizations_url":"https://api.github.com/users/BnY/orgs","repos_url":"https://api.github.com/users/BnY/repos","events_url":"https://api.github.com/users/BnY/events{/privacy}","received_events_url":"https://api.github.com/users/BnY/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-02-19T16:40:31Z","updated_at":"2018-02-20T09:27:39Z","closed_at":null,"author_association":"NONE","body":"After upgrading all our nodes running dask to the latest GIT status (needed for retry mechanism) we encountered the problem shown by the log below when starting up the workers on the 2nd host... (First host is fine!)\r\n\r\nUnfortunately I cannot help a lot more, because I'm not the administrator of the cluster. After IT rolled back to 1.20.2 the workers on all nodes startup fine...\r\n\r\n```\r\ndask-worker tcp://<IP_ADDRESS>:8786 --memory-limit 6e9 --local-directory /u01/USERNAME/dask_worker_dir --nprocs 8 --nthreads 1\r\n\r\ndistributed.nanny - INFO -         Start Nanny at: 'tcp://<IP_ADDRESS>:24534'\r\ndistributed.nanny - INFO -         Start Nanny at: 'tcp://<IP_ADDRESS>:17988'\r\ndistributed.nanny - INFO -         Start Nanny at: 'tcp://<IP_ADDRESS>:24278'\r\ndistributed.diskutils - WARNING - Found stale lock file and directory u'/u01/USERNAME/dask_worker_dir/worker-HyyuR7', purging\r\ndistributed.diskutils - WARNING - Found stale lock file and directory u'/u01/USERNAME/dask_worker_dir/worker-0B5Jok', purging\r\ndistributed.diskutils - WARNING - Found stale lock file and directory u'/u01/USERNAME/dask_worker_dir/worker-NmuI4k', purging\r\ndistributed.nanny - INFO -         Start Nanny at: 'tcp://<IP_ADDRESS>:18293'\r\ndistributed.worker - INFO -       Start worker at:  tcp://<IP_ADDRESS>:37222\r\ndistributed.worker - INFO -          Listening to:  tcp://<IP_ADDRESS>:37222\r\ndistributed.worker - INFO -              nanny at:        <IP_ADDRESS>:24534\r\ndistributed.worker - INFO - Waiting to connect to:   tcp://<IP_ADDRESS>:8786\r\ndistributed.worker - INFO - -------------------------------------------------\r\ndistributed.worker - INFO -               Threads:                          1\r\ndistributed.worker - INFO -                Memory:                    6.00 GB\r\ndistributed.worker - INFO -       Local Directory: /u01/USERNAME/dask_worker_dir/worker-XH_Qpm\r\ndistributed.worker - INFO - -------------------------------------------------\r\ndistributed.nanny - INFO -         Start Nanny at: 'tcp://<IP_ADDRESS>:12528'\r\ndistributed.worker - INFO -       Start worker at:  tcp://<IP_ADDRESS>:16643\r\ndistributed.worker - INFO -          Listening to:  tcp://<IP_ADDRESS>:16643\r\ndistributed.worker - INFO -              nanny at:        <IP_ADDRESS>:17988\r\ndistributed.worker - INFO - Waiting to connect to:   tcp://<IP_ADDRESS>:8786\r\ndistributed.worker - INFO - -------------------------------------------------\r\ndistributed.worker - INFO -               Threads:                          1\r\ndistributed.worker - INFO -                Memory:                    6.00 GB\r\ndistributed.worker - INFO -       Local Directory: /u01/USERNAME/dask_worker_dir/worker-eUNonW\r\ndistributed.worker - INFO - -------------------------------------------------\r\ndistributed.nanny - INFO -         Start Nanny at: 'tcp://<IP_ADDRESS>:35961'\r\ndistributed.nanny - ERROR - Failed to start worker\r\nTraceback (most recent call last):\r\n  File \"/opt/cloudera/parcels/Anaconda-4.1.1/lib/python2.7/site-packages/distributed/nanny.py\", line 522, in run\r\n    yield worker._start(*worker_start_args)\r\n  File \"/opt/cloudera/parcels/Anaconda-4.1.1/lib/python2.7/site-packages/tornado/gen.py\", line 1055, in run\r\n    value = future.result()\r\n  File \"/opt/cloudera/parcels/Anaconda-4.1.1/lib/python2.7/site-packages/tornado/concurrent.py\", line 238, in result\r\n    raise_exc_info(self._exc_info)\r\n  File \"/opt/cloudera/parcels/Anaconda-4.1.1/lib/python2.7/site-packages/tornado/gen.py\", line 1063, in run\r\n    yielded = self.gen.throw(*exc_info)\r\n  File \"/opt/cloudera/parcels/Anaconda-4.1.1/lib/python2.7/site-packages/distributed/worker.py\", line 372, in _start\r\n    yield self._register_with_scheduler()\r\n  File \"/opt/cloudera/parcels/Anaconda-4.1.1/lib/python2.7/site-packages/tornado/gen.py\", line 1055, in run\r\n    value = future.result()\r\n  File \"/opt/cloudera/parcels/Anaconda-4.1.1/lib/python2.7/site-packages/tornado/concurrent.py\", line 238, in result\r\n    raise_exc_info(self._exc_info)\r\n  File \"/opt/cloudera/parcels/Anaconda-4.1.1/lib/python2.7/site-packages/tornado/gen.py\", line 1069, in run\r\n    yielded = self.gen.send(value)\r\n  File \"/opt/cloudera/parcels/Anaconda-4.1.1/lib/python2.7/site-packages/distributed/worker.py\", line 295, in _register_with_scheduler\r\n    (response,))\r\nValueError: Unexpected response from register: {'status': 'error', 'message': u'name taken, -0', 'time': 1519054893.084648}\r\ndistributed.worker - INFO -       Start worker at:  tcp://<IP_ADDRESS>:36269\r\ndistributed.worker - INFO -          Listening to:  tcp://<IP_ADDRESS>:36269\r\ndistributed.worker - INFO -              nanny at:        <IP_ADDRESS>:24278\r\ndistributed.worker - INFO - Waiting to connect to:   tcp://<IP_ADDRESS>:8786\r\ndistributed.worker - INFO - -------------------------------------------------\r\ndistributed.worker - INFO -               Threads:                          1\r\ndistributed.worker - INFO -                Memory:                    6.00 GB\r\ndistributed.worker - INFO -       Local Directory: /u01/USERNAME/dask_worker_dir/worker-3hke9z\r\ndistributed.worker - INFO - -------------------------------------------------\r\ndistributed.nanny - ERROR - Failed to start worker\r\n\r\n\r\n```"},"comment":{"url":"https://api.github.com/repos/dask/distributed/issues/comments/366918298","html_url":"https://github.com/dask/distributed/issues/1772#issuecomment-366918298","issue_url":"https://api.github.com/repos/dask/distributed/issues/1772","id":366918298,"user":{"login":"BnY","id":157254,"avatar_url":"https://avatars1.githubusercontent.com/u/157254?v=4","gravatar_id":"","url":"https://api.github.com/users/BnY","html_url":"https://github.com/BnY","followers_url":"https://api.github.com/users/BnY/followers","following_url":"https://api.github.com/users/BnY/following{/other_user}","gists_url":"https://api.github.com/users/BnY/gists{/gist_id}","starred_url":"https://api.github.com/users/BnY/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/BnY/subscriptions","organizations_url":"https://api.github.com/users/BnY/orgs","repos_url":"https://api.github.com/users/BnY/repos","events_url":"https://api.github.com/users/BnY/events{/privacy}","received_events_url":"https://api.github.com/users/BnY/received_events","type":"User","site_admin":false},"created_at":"2018-02-20T09:27:39Z","updated_at":"2018-02-20T09:27:39Z","author_association":"NONE","body":"Thanks for the option.\r\n\r\nI did try \"--nprocs = 1\" yesterday on the 2nd node. However I saw strange numbers in \"worker count\" and available GB coming from the client.restart(). I didn't trust the environment anymore so I had IT roll back the install. Drawback of working for large company...\r\n"}}
{"actor":{"display_login":"BnY"}}