{"action":"created","issue":{"url":"https://api.github.com/repos/dask/distributed/issues/1914","repository_url":"https://api.github.com/repos/dask/distributed","labels_url":"https://api.github.com/repos/dask/distributed/issues/1914/labels{/name}","comments_url":"https://api.github.com/repos/dask/distributed/issues/1914/comments","events_url":"https://api.github.com/repos/dask/distributed/issues/1914/events","html_url":"https://github.com/dask/distributed/issues/1914","id":313323998,"number":1914,"title":"Losing all workers on cluster ","user":{"login":"BrendanMartin","id":8090350,"avatar_url":"https://avatars3.githubusercontent.com/u/8090350?v=4","gravatar_id":"","url":"https://api.github.com/users/BrendanMartin","html_url":"https://github.com/BrendanMartin","followers_url":"https://api.github.com/users/BrendanMartin/followers","following_url":"https://api.github.com/users/BrendanMartin/following{/other_user}","gists_url":"https://api.github.com/users/BrendanMartin/gists{/gist_id}","starred_url":"https://api.github.com/users/BrendanMartin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/BrendanMartin/subscriptions","organizations_url":"https://api.github.com/users/BrendanMartin/orgs","repos_url":"https://api.github.com/users/BrendanMartin/repos","events_url":"https://api.github.com/users/BrendanMartin/events{/privacy}","received_events_url":"https://api.github.com/users/BrendanMartin/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":11,"created_at":"2018-04-11T13:28:19Z","updated_at":"2018-04-26T12:56:35Z","closed_at":null,"author_association":"NONE","body":"I'm not sure how much detail to put here, or if this is the right place, but I am attempting to run `dask_searchcv` on a Kubernetes cluster and whenever I run the grid search all of my workers get killed immediately.\r\n\r\nFrom the logs on the cluster I have:\r\n```\r\ndistributed.scheduler - INFO - Register tcp://10.16.1.9:41940\r\n\r\ndistributed.scheduler - INFO - Starting worker compute stream, tcp://10.16.1.9:41940\r\n\r\ndistributed.scheduler - INFO - Register tcp://10.16.0.9:46410\r\n\r\ndistributed.scheduler - INFO - Starting worker compute stream, tcp://10.16.0.9:46410\r\n\r\ndistributed.scheduler - INFO - Register tcp://10.16.2.9:38646\r\n\r\ndistributed.scheduler - INFO - Starting worker compute stream, tcp://10.16.2.9:38646\r\n\r\ndistributed.scheduler - INFO - Remove worker tcp://10.16.2.9:38646\r\n\r\ndistributed.scheduler - INFO - Remove worker tcp://10.16.1.9:41940\r\n\r\ndistributed.scheduler - INFO - Remove worker tcp://10.16.0.9:46410\r\n\r\ndistributed.scheduler - INFO - Lost all workers\r\n```\r\n\r\nI have a cluster of 3 CPUs and 12G RAM, and so 3 workers are spawned by default. \r\n\r\nOne thing I noticed, not sure if it's related, but when I run the grid search and look at the workers, there's always one worker that's at 90%+ utilization while the others are only around 5%."},"comment":{"url":"https://api.github.com/repos/dask/distributed/issues/comments/384631251","html_url":"https://github.com/dask/distributed/issues/1914#issuecomment-384631251","issue_url":"https://api.github.com/repos/dask/distributed/issues/1914","id":384631251,"user":{"login":"mrocklin","id":306380,"avatar_url":"https://avatars3.githubusercontent.com/u/306380?v=4","gravatar_id":"","url":"https://api.github.com/users/mrocklin","html_url":"https://github.com/mrocklin","followers_url":"https://api.github.com/users/mrocklin/followers","following_url":"https://api.github.com/users/mrocklin/following{/other_user}","gists_url":"https://api.github.com/users/mrocklin/gists{/gist_id}","starred_url":"https://api.github.com/users/mrocklin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mrocklin/subscriptions","organizations_url":"https://api.github.com/users/mrocklin/orgs","repos_url":"https://api.github.com/users/mrocklin/repos","events_url":"https://api.github.com/users/mrocklin/events{/privacy}","received_events_url":"https://api.github.com/users/mrocklin/received_events","type":"User","site_admin":false},"created_at":"2018-04-26T12:56:35Z","updated_at":"2018-04-26T12:56:35Z","author_association":"OWNER","body":"> Do those always reflect the latest version? Because when I use helm install stable/dask, and after updating all of my client packages, this is the check versions now:\r\n\r\nNo, they reflect the version that is in the helm chart.  You have two options:\r\n\r\n1.  Build and push your own docker image.  Presumably small changes to https://github.com/dask/dask-docker can get you where you want to be.  You'll likely end up needing to do this at some point anyway\r\n2.  Set your local client version of Dask to be the same as what is in your image."}}
{"actor":{"display_login":"mrocklin"}}