{"action":"created","issue":{"url":"https://api.github.com/repos/dask/dask-jobqueue/issues/151","repository_url":"https://api.github.com/repos/dask/dask-jobqueue","labels_url":"https://api.github.com/repos/dask/dask-jobqueue/issues/151/labels{/name}","comments_url":"https://api.github.com/repos/dask/dask-jobqueue/issues/151/comments","events_url":"https://api.github.com/repos/dask/dask-jobqueue/issues/151/events","html_url":"https://github.com/dask/dask-jobqueue/issues/151","id":360577827,"node_id":"MDU6SXNzdWUzNjA1Nzc4Mjc=","number":151,"title":"Dask client connects to PBS workers, then rapidly loses them","user":{"login":"vprzybylo","id":32338353,"node_id":"MDQ6VXNlcjMyMzM4MzUz","avatar_url":"https://avatars0.githubusercontent.com/u/32338353?v=4","gravatar_id":"","url":"https://api.github.com/users/vprzybylo","html_url":"https://github.com/vprzybylo","followers_url":"https://api.github.com/users/vprzybylo/followers","following_url":"https://api.github.com/users/vprzybylo/following{/other_user}","gists_url":"https://api.github.com/users/vprzybylo/gists{/gist_id}","starred_url":"https://api.github.com/users/vprzybylo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vprzybylo/subscriptions","organizations_url":"https://api.github.com/users/vprzybylo/orgs","repos_url":"https://api.github.com/users/vprzybylo/repos","events_url":"https://api.github.com/users/vprzybylo/events{/privacy}","received_events_url":"https://api.github.com/users/vprzybylo/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":7,"created_at":"2018-09-15T21:33:45Z","updated_at":"2018-09-20T15:19:17Z","closed_at":null,"author_association":"NONE","body":"Similar to #30, I have a bash script created to run a python job on Cheyenne using a scheduler file and a 'start-workers.py' script but after the workers come online and the main script is run, the workers die off rapidly. \r\n\r\nI have tried using qsub to start an interactive job for jupyter lab in prior attempts (see pangeo issue #352) but if the connection is dropped, the job disappears from the queue and I am unable to ssh back in to receive the output data.  For this reason, I believe it is better to submit a job via a scheduler file.  \r\n\r\nHere is an outline of the steps I have taken to submit the job via bash script.  Note I have changed the number of threads, processes, memory, etc with no luck in keeping the workers online.  I simply copied the arguments from the jupyter lab cell that was working fine.\r\n\r\nThis is the main bash script that calls the other scheduler/worker scripts:\r\n#!/bin/bash\r\nset -e\r\n\r\necho \"Launching dask scheduler\"\r\ns=$(qsub launch-dask-scheduler.sh)\r\nsjob=${s%.*}\r\necho ${sjob}\r\n\r\nblock until the scheduler starts\r\nwhile true; do\r\n    status=`qstat ${sjob} | tail -n 1`\r\n    echo ${status}\r\n    if [[ ${status} =~ \" R \" ]]; then\r\n        break\r\n    fi\r\n    sleep 1\r\ndone\r\n\r\ncommented out until done\r\nworkers=${1:-10}\r\necho \"Launching dask workers (${workers})\"\r\nfor i in $(seq 1 ${workers}); do\r\n    echo 'launching dask worker'\r\n    wjob=$(qsub launch-dask-worker.sh)\r\ndone\r\n\r\nsource activate pangeo\r\npython start_workers.py\r\n\r\nstatus=`qstat ${wjob} | tail -n 1`\r\nblock until the workers start\r\nwhile true; do\r\n    status=`qstat ${wjob} | tail -n 1`\r\n   \r\n    echo ${status}\r\n    if [[ ${status} =~ \" R \" ]]; then\r\n        break\r\n    fi\r\n    sleep 1\r\ndone\r\necho 'running IPAS script'\r\nsource activate pangeo\r\npython ./Agg_Main_phiarray.py\r\n\r\n################################\r\n\r\nlaunch-dask-scheduler.sh script:\r\n#!/bin/bash\r\n#PBS -N dask-scheduler\r\n#PBS -q economy\r\n#PBS -A UALB0017\r\n#PBS -l select=1:ncpus=36:mpiprocs=2:ompthreads=4:mem=109G\r\n#PBS -l walltime=05:00:00\r\n#PBS -j oe\r\n\r\nSetup Environment\r\nmodule purge\r\nsource activate pangeo\r\n\r\nSCHEDULER=/glade/scratch/$USER/scheduler.json\r\nrm -f $SCHEDULER\r\nmpirun --np 9 dask-mpi --nthreads 4 \\\r\n    --memory-limit 22e9 \\\r\n    --interface ib0 \\\r\n    --local-directory /glade/scratch/$USER \\\r\n    --scheduler-file=$SCHEDULER\r\n\r\n#######################################\r\nstart_workers.py script:\r\nfrom dask_jobqueue import PBSCluster\r\nfrom dask.distributed import Client\r\ncluster = PBSCluster(processes=18,\r\n                     cores=8,\r\n                     memory=\"6GB\",\r\n                     project='UALB0017',\r\n                     queue='economy',\r\n                     resource_spec='select=1:ncpus=36:ompthreads=4:mem=109G',\r\n                     walltime='05:00:00')\r\n\r\nc = cluster.start_workers(10)\r\nclient = Client(cluster)\r\n\r\n#######################################\r\nlaunch-dask-worker.sh (not using this currently, instead using the python start_workers.py script)\r\n#!/bin/bash\r\n#PBS -N dask-workers\r\n#PBS -q economy\r\n#PBS -A UALB0017\r\n#PBS -l select=1:ncpus=36:mpiprocs=2:ompthreads=4\r\n#PBS -l walltime=05:00:00\r\n#PBS -j oe\r\n\r\nSetup Environment\r\nmodule purge\r\nsource activate pangeo\r\n\r\nSCHEDULER=/glade/scratch/$USER/scheduler.json\r\nmpirun --np 9 dask-mpi --nthreads 4 \\\r\n    --memory-limit 22e9 \\\r\n    --interface ib0 \\\r\n    --no-scheduler --local-directory /glade/scratch/$USER \\\r\n    --scheduler-file=$SCHEDULER\r\n\r\n########################################\r\n\r\nHere is a log file for a dask worker:\r\ndistributed.nanny - INFO -         Start Nanny at: 'tcp://10.148.9.94:46539'\r\ndistributed.nanny - INFO -         Start Nanny at: 'tcp://10.148.9.94:59753'\r\ndistributed.nanny - INFO -         Start Nanny at: 'tcp://10.148.9.94:58225'\r\ndistributed.nanny - INFO -         Start Nanny at: 'tcp://10.148.9.94:55667'\r\ndistributed.diskutils - INFO - Found stale lock file and directory '/gpfs/u/home/przybylo/IPAS_array/dask-worker-space/wor\r\nker-c0kfm5ja', purging\r\ndistributed.worker - INFO -       Start worker at:    tcp://10.148.9.94:52521\r\ndistributed.worker - INFO -          Listening to:    tcp://10.148.9.94:52521\r\ndistributed.worker - INFO -              nanny at:          10.148.9.94:32798\r\ndistributed.worker - INFO -              bokeh at:           10.148.9.94:8789\r\ndistributed.worker - INFO - Waiting to connect to:     tcp://10.148.0.7:44783\r\ndistributed.worker - INFO - -------------------------------------------------\r\ndistributed.worker - INFO -       Start worker at:    tcp://10.148.9.94:43057\r\ndistributed.worker - INFO -          Listening to:    tcp://10.148.9.94:43057\r\ndistributed.worker - INFO -              nanny at:          10.148.9.94:43720\r\ndistributed.worker - INFO -              bokeh at:          10.148.9.94:36811\r\ndistributed.worker - INFO - Waiting to connect to:     tcp://10.148.0.7:44783\r\ndistributed.worker - INFO - -------------------------------------------------\r\ndistributed.worker - INFO -               Threads:                          4\r\ndistributed.worker - INFO -       Start worker at:    tcp://10.148.9.94:58792\r\ndistributed.worker - INFO -                Memory:                  333.33 MB\r\ndistributed.worker - INFO -       Local Directory: /gpfs/u/home/przybylo/IPAS_array/dask-worker-space/worker-qg0ew52i\r\ndistributed.worker - INFO -          Listening to:    tcp://10.148.9.94:58792\r\ndistributed.worker - INFO -              nanny at:          10.148.9.94:59953\r\ndistributed.worker - INFO -       Start worker at:    tcp://10.148.9.94:41574\r\ndistributed.worker - INFO - -------------------------------------------------\r\ndistributed.worker - INFO -              bokeh at:          10.148.9.94:50134\r\ndistributed.worker - INFO -          Listening to:    tcp://10.148.9.94:41574\r\ndistributed.worker - INFO - Waiting to connect to:     tcp://10.148.0.7:44783\r\ndistributed.worker - INFO -               Threads:                          4\r\ndistributed.worker - INFO -              nanny at:          10.148.9.94:37153\r\ndistributed.worker - INFO - -------------------------------------------------\r\ndistributed.worker - INFO -                Memory:                  333.33 MB\r\ndistributed.worker - INFO -              bokeh at:          10.148.9.94:33988\r\ndistributed.worker - INFO -       Local Directory: /gpfs/u/home/przybylo/IPAS_array/dask-worker-space/worker-pa63sigd\r\ndistributed.worker - INFO - Waiting to connect to:     tcp://10.148.0.7:44783\r\ndistributed.worker - INFO - -------------------------------------------------\r\ndistributed.worker - INFO - -------------------------------------------------\r\ndistributed.worker - INFO -               Threads:                          4\r\ndistributed.worker - INFO -       Start worker at:    tcp://10.148.9.94:41721\r\ndistributed.worker - INFO -                Memory:                  333.33 MB\r\ndistributed.worker - INFO -          Listening to:    tcp://10.148.9.94:41721\r\ndistributed.worker - INFO -       Local Directory: /gpfs/u/home/przybylo/IPAS_array/dask-worker-space/worker-hdjn52na\r\ndistributed.worker - INFO -               Threads:                          4\r\ndistributed.worker - INFO -              nanny at:          10.148.9.94:50370\r\ndistributed.worker - INFO - -------------------------------------------------\r\ndistributed.worker - INFO -              bokeh at:          10.148.9.94:56406\r\ndistributed.worker - INFO -                Memory:                  333.33 MB\r\ndistributed.worker - INFO - Waiting to connect to:     tcp://10.148.0.7:44783\r\ndistributed.worker - INFO -       Local Directory: /gpfs/u/home/przybylo/IPAS_array/dask-worker-space/worker-viv52842\r\ndistributed.worker - INFO - -------------------------------------------------\r\ndistributed.worker - INFO - -------------------------------------------------\r\ndistributed.diskutils - INFO - Found stale lock file and directory '/gpfs/u/home/przybylo/IPAS_array/dask-worker-space/wor\r\nker-vo2pubtu', purging\r\ndistributed.worker - INFO -               Threads:                          4\r\ndistributed.worker - INFO -                Memory:                  333.33 MB\r\ndistributed.worker - INFO -       Local Directory: /gpfs/u/home/przybylo/IPAS_array/dask-worker-space/worker-j8_v1p2h\r\ndistributed.worker - INFO - -------------------------------------------------\r\ndistributed.worker - INFO -       Start worker at:    tcp://10.148.9.94:41047\r\ndistributed.worker - INFO -          Listening to:    tcp://10.148.9.94:41047\r\ndistributed.worker - INFO -              nanny at:          10.148.9.94:46539\r\ndistributed.worker - INFO -              bokeh at:          10.148.9.94:53889\r\ndistributed.worker - INFO - Waiting to connect to:     tcp://10.148.0.7:44783\r\ndistributed.worker - INFO - -------------------------------------------------\r\ndistributed.worker - INFO -               Threads:                          4\r\ndistributed.worker - INFO -                Memory:                  333.33 MB\r\ndistributed.worker - INFO -       Local Directory: /gpfs/u/home/przybylo/IPAS_array/dask-worker-space/worker-y27xg4wq\r\ndistributed.worker - INFO - -------------------------------------------------\r\ndistributed.worker - INFO -       Start worker at:    tcp://10.148.9.94:49268\r\ndistributed.worker - INFO -          Listening to:    tcp://10.148.9.94:49268\r\ndistributed.worker - INFO -              nanny at:          10.148.9.94:33282\r\ndistributed.worker - INFO -              bokeh at:          10.148.9.94:38760\r\ndistributed.worker - INFO - Waiting to connect to:     tcp://10.148.0.7:44783\r\ndistributed.worker - INFO - -------------------------------------------------\r\ndistributed.worker - INFO -               Threads:                          4\r\ndistributed.worker - INFO -                Memory:                  333.33 MB\r\ndistributed.worker - INFO -       Local Directory: /gpfs/u/home/przybylo/IPAS_array/dask-worker-space/worker-6rath3fb\r\ndistributed.worker - INFO - -------------------------------------------------\r\ndistributed.worker - INFO -       Start worker at:    tcp://10.148.9.94:36633\r\ndistributed.worker - INFO -          Listening to:    tcp://10.148.9.94:36633\r\ndistributed.worker - INFO -              nanny at:          10.148.9.94:39102\r\ndistributed.worker - INFO -              bokeh at:          10.148.9.94:45682\r\ndistributed.worker - INFO - Waiting to connect to:     tcp://10.148.0.7:44783\r\ndistributed.worker - INFO - -------------------------------------------------\r\ndistributed.worker - INFO -               Threads:                          4\r\ndistributed.worker - INFO -                Memory:                  333.33 MB\r\ndistributed.worker - INFO -       Local Directory: /gpfs/u/home/przybylo/IPAS_array/dask-worker-space/worker-s1fr0lv6\r\ndistributed.worker - INFO - -------------------------------------------------\r\ndistributed.worker - INFO -       Start worker at:    tcp://10.148.9.94:42357\r\ndistributed.worker - INFO -       Start worker at:    tcp://10.148.9.94:54327\r\ndistributed.worker - INFO -          Listening to:    tcp://10.148.9.94:42357\r\ndistributed.worker - INFO -          Listening to:    tcp://10.148.9.94:54327\r\ndistributed.worker - INFO -              nanny at:          10.148.9.94:58225\r\ndistributed.worker - INFO -              nanny at:          10.148.9.94:58002\r\ndistributed.worker - INFO -              bokeh at:          10.148.9.94:45961\r\ndistributed.worker - INFO -              bokeh at:          10.148.9.94:52568\r\ndistributed.worker - INFO - Waiting to connect to:     tcp://10.148.0.7:44783\r\ndistributed.worker - INFO - Waiting to connect to:     tcp://10.148.0.7:44783\r\ndistributed.worker - INFO - -------------------------------------------------\r\ndistributed.worker - INFO - -------------------------------------------------\r\ndistributed.worker - INFO -               Threads:                          4\r\ndistributed.worker - INFO -               Threads:                          4\r\ndistributed.worker - INFO -                Memory:                  333.33 MB\r\ndistributed.worker - INFO -                Memory:                  333.33 MB\r\ndistributed.worker - INFO -       Local Directory: /gpfs/u/home/przybylo/IPAS_array/dask-worker-space/worker-imugcrhu\r\ndistributed.worker - INFO -       Local Directory: /gpfs/u/home/przybylo/IPAS_array/dask-worker-space/worker-31s8103f\r\ndistributed.worker - INFO - -------------------------------------------------\r\ndistributed.worker - INFO - -------------------------------------------------\r\ndistributed.worker - INFO -       Start worker at:    tcp://10.148.9.94:47381\r\ndistributed.worker - INFO -          Listening to:    tcp://10.148.9.94:47381\r\ndistributed.worker - INFO -              nanny at:          10.148.9.94:59753\r\ndistributed.worker - INFO -              bokeh at:          10.148.9.94:53258\r\ndistributed.worker - INFO - Waiting to connect to:     tcp://10.148.0.7:44783\r\ndistributed.worker - INFO - -------------------------------------------------\r\ndistributed.worker - INFO -               Threads:                          4\r\ndistributed.worker - INFO -                Memory:                  333.33 MB\r\ndistributed.worker - INFO -       Local Directory: /gpfs/u/home/przybylo/IPAS_array/dask-worker-space/worker-i7fldc3s\r\ndistributed.worker - INFO - -------------------------------------------------\r\ndistributed.worker - INFO -       Start worker at:    tcp://10.148.9.94:49428\r\ndistributed.worker - INFO -       Start worker at:    tcp://10.148.9.94:46224\r\ndistributed.worker - INFO -          Listening to:    tcp://10.148.9.94:49428\r\ndistributed.worker - INFO -          Listening to:    tcp://10.148.9.94:46224\r\ndistributed.worker - INFO -              nanny at:          10.148.9.94:37400\r\ndistributed.worker - INFO -              nanny at:          10.148.9.94:55009\r\ndistributed.worker - INFO -              bokeh at:          10.148.9.94:36557\r\ndistributed.worker - INFO -              bokeh at:          10.148.9.94:45115\r\ndistributed.worker - INFO - Waiting to connect to:     tcp://10.148.0.7:44783\r\ndistributed.worker - INFO - -------------------------------------------------\r\ndistributed.worker - INFO - Waiting to connect to:     tcp://10.148.0.7:44783\r\ndistributed.worker - INFO - -------------------------------------------------\r\ndistributed.worker - INFO -       Start worker at:    tcp://10.148.9.94:39940\r\ndistributed.worker - INFO -          Listening to:    tcp://10.148.9.94:39940\r\ndistributed.worker - INFO -       Start worker at:    tcp://10.148.9.94:58470\r\ndistributed.worker - INFO -              nanny at:          10.148.9.94:49486\r\ndistributed.worker - INFO -              bokeh at:          10.148.9.94:46379\r\ndistributed.worker - INFO -          Listening to:    tcp://10.148.9.94:58470\r\ndistributed.worker - INFO - Waiting to connect to:     tcp://10.148.0.7:44783\r\ndistributed.worker - INFO -              nanny at:          10.148.9.94:46733\r\ndistributed.worker - INFO - -------------------------------------------------\r\ndistributed.worker - INFO -              bokeh at:          10.148.9.94:55313\r\ndistributed.worker - INFO - Waiting to connect to:     tcp://10.148.0.7:44783\r\ndistributed.worker - INFO -               Threads:                          4\r\ndistributed.worker - INFO - -------------------------------------------------\r\ndistributed.worker - INFO -               Threads:                          4\r\ndistributed.worker - INFO -       Start worker at:    tcp://10.148.9.94:38657\r\ndistributed.worker - INFO -                Memory:                  333.33 MB\r\ndistributed.worker - INFO -       Local Directory: /gpfs/u/home/przybylo/IPAS_array/dask-worker-space/worker-mbj0bzhp\r\ndistributed.worker - INFO - -------------------------------------------------\r\ndistributed.worker - INFO -       Start worker at:    tcp://10.148.9.94:48927\r\ndistributed.worker - INFO -          Listening to:    tcp://10.148.9.94:48927\r\ndistributed.worker - INFO -              nanny at:          10.148.9.94:55667\r\ndistributed.worker - INFO -              bokeh at:          10.148.9.94:36750\r\ndistributed.worker - INFO - Waiting to connect to:     tcp://10.148.0.7:44783\r\ndistributed.worker - INFO - -------------------------------------------------\r\ndistributed.worker - INFO -               Threads:                          4\r\ndistributed.worker - INFO -                Memory:                  333.33 MB\r\ndistributed.worker - INFO -       Local Directory: /gpfs/u/home/przybylo/IPAS_array/dask-worker-space/worker-ldadnvth\r\ndistributed.worker - INFO - -------------------------------------------------\r\ndistributed.worker - INFO - Waiting to connect to:     tcp://10.148.0.7:44783\r\ndistributed.worker - INFO - Waiting to connect to:     tcp://10.148.0.7:44783\r\ndistributed.worker - INFO - Waiting to connect to:     tcp://10.148.0.7:44783\r\ndistributed.worker - INFO - Waiting to connect to:     tcp://10.148.0.7:44783\r\ndistributed.worker - INFO - Waiting to connect to:     tcp://10.148.0.7:44783\r\ndistributed.worker - INFO - Waiting to connect to:     tcp://10.148.0.7:44783\r\ndistributed.worker - INFO - Waiting to connect to:     tcp://10.148.0.7:44783\r\ndistributed.worker - INFO - Waiting to connect to:     tcp://10.148.0.7:44783\r\ndistributed.worker - INFO - Waiting to connect to:     tcp://10.148.0.7:44783\r\ndistributed.worker - INFO - Waiting to connect to:     tcp://10.148.0.7:44783\r\ndistributed.nanny - INFO - Closing Nanny at 'tcp://10.148.9.94:46539'\r\ndistributed.worker - INFO - Stopping worker at tcp://10.148.9.94:41047\r\ndistributed.nanny - INFO - Closing Nanny at 'tcp://10.148.9.94:49486'`\r\n\r\n\r\n\r\n###########################################\r\nAnd a log file for the scheduler:\r\n`distributed.nanny - INFO -         Start Nanny at: 'tcp://10.148.9.31:41209'\r\ndistributed.nanny - INFO -         Start Nanny at: 'tcp://10.148.9.31:53305'\r\ndistributed.nanny - INFO -         Start Nanny at: 'tcp://10.148.9.31:50455'\r\n[cli_6]: write_line error; fd=29 buf=:cmd=init pmi_version=1 pmi_subversion=1\r\n:\r\nsystem msg for write_line failure : Bad file descriptor\r\n[cli_6]: Unable to write to PMI_fd\r\n[cli_6]: write_line error; fd=29 buf=:cmd=get_appnum\r\n:\r\nsystem msg for write_line failure : Bad file descriptor\r\nFatal error in PMPI_Init_thread: Other MPI error, error stack:\r\nMPIR_Init_thread(474): \r\nMPID_Init(152).......: channel initialization failed\r\nMPID_Init(426).......: PMI_Get_appnum returned -1\r\n[cli_6]: write_line error; fd=29 buf=:cmd=abort exitcode=1094159\r\n:\r\nsystem msg for write_line failure : Bad file descriptor\r\ndistributed.nanny - WARNING - Worker process 21557 was killed by unknown signal\r\n[cli_1]: Error reading initack on 8\r\nError on readline:: Bad file descriptor\r\n[cli_1]: readline failed\r\n[cli_1]: readline failed\r\n[cli_1]: readline failed\r\n[cli_1]: readline failed\r\nFatal error in PMPI_Init_thread: Other MPI error, error stack:\r\nMPIR_Init_thread(474): \r\nMPID_Init(152).......: channel initialization failed\r\nMPID_Init(426).......: PMI_Get_appnum returned -1\r\n[cli_1]: readline failed\r\ndistributed.nanny - WARNING - Worker process 21559 exited with status 8388357179234872675\r\n[cli_4]: write_line error; fd=21 buf=:cmd=init pmi_version=1 pmi_subversion=1\r\n:\r\nsystem msg for write_line failure : Bad file descriptor\r\n[cli_4]: Unable to write to PMI_fd\r\n[cli_4]: write_line error; fd=21 buf=:cmd=get_appnum\r\n:\r\nsystem msg for write_line failure : Bad file descriptor\r\nFatal error in PMPI_Init_thread: Other MPI error, error stack:\r\nMPIR_Init_thread(474): \r\nMPID_Init(152).......: channel initialization failed\r\nMPID_Init(426).......: PMI_Get_appnum returned -1\r\n[cli_4]: write_line error; fd=21 buf=:cmd=abort exitcode=1094159\r\n:\r\nsystem msg for write_line failure : Bad file descriptor\r\ndistributed.nanny - WARNING - Restarting worker\r\ndistributed.nanny - WARNING - Worker process 21561 was killed by unknown signal\r\n[cli_5]: write_line error; fd=25 buf=:cmd=init pmi_version=1 pmi_subversion=1\r\n:\r\nsystem msg for write_line failure : Bad file descriptor\r\n[cli_5]: Unable to write to PMI_fd\r\n[cli_5]: write_line error; fd=25 buf=:cmd=get_appnum\r\n:\r\nsystem msg for write_line failure : Bad file descriptor\r\nFatal error in PMPI_Init_thread: Other MPI error, error stack:\r\nMPIR_Init_thread(474): \r\nMPID_Init(152).......: channel initialization failed\r\nMPID_Init(426).......: PMI_Get_appnum returned -1\r\n[cli_5]: write_line error; fd=25 buf=:cmd=abort exitcode=1094159\r\n:\r\nsystem msg for write_line failure : Bad file descriptor\r\ndistributed.nanny - WARNING - Restarting worker\r\n[cli_2]: write_line error; fd=13 buf=:cmd=init pmi_version=1 pmi_subversion=1\r\n##########################################\r\n\r\nAny advise on why the worker won't stay running in the queue?  Thanks for taking the time to go through all of this!\r\n\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/dask/dask-jobqueue/issues/comments/423223507","html_url":"https://github.com/dask/dask-jobqueue/issues/151#issuecomment-423223507","issue_url":"https://api.github.com/repos/dask/dask-jobqueue/issues/151","id":423223507,"node_id":"MDEyOklzc3VlQ29tbWVudDQyMzIyMzUwNw==","user":{"login":"guillaumeeb","id":17138587,"node_id":"MDQ6VXNlcjE3MTM4NTg3","avatar_url":"https://avatars0.githubusercontent.com/u/17138587?v=4","gravatar_id":"","url":"https://api.github.com/users/guillaumeeb","html_url":"https://github.com/guillaumeeb","followers_url":"https://api.github.com/users/guillaumeeb/followers","following_url":"https://api.github.com/users/guillaumeeb/following{/other_user}","gists_url":"https://api.github.com/users/guillaumeeb/gists{/gist_id}","starred_url":"https://api.github.com/users/guillaumeeb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/guillaumeeb/subscriptions","organizations_url":"https://api.github.com/users/guillaumeeb/orgs","repos_url":"https://api.github.com/users/guillaumeeb/repos","events_url":"https://api.github.com/users/guillaumeeb/events{/privacy}","received_events_url":"https://api.github.com/users/guillaumeeb/received_events","type":"User","site_admin":false},"created_at":"2018-09-20T15:19:17Z","updated_at":"2018-09-20T15:19:17Z","author_association":"MEMBER","body":"> I am using Pythons multiprocessing for parallelization\r\n\r\nThat sounds wrong to me, but maybe I'm mistaken. You should be using only Dask API exept if you have a very specific use case.\r\n\r\nCould you show a simplified version of things your doing after the main() function definition?"}}
{"actor":{"display_login":"guillaumeeb"}}