<p>I have deployed <code>dask_gateway</code> 0.8.0 (with <code>dask==2.25.0</code> and <code>distributed==2.25.0</code>) in a Kubernetes cluster.
When I create a new cluster with:</p>
<pre><code>cluster = gateway.new_cluster(public_address = gateway._public_address)
</code></pre>
<p>I get this error:</p>
<pre><code>Task exception was never retrieved
future: &lt;Task finished coro=&lt;connect.&lt;locals&gt;._() done, defined at /home/jovyan/.local/lib/python3.6/site-packages/distributed/comm/core.py:288&gt; exception=CommClosedError()&gt;
Traceback (most recent call last):
  File &quot;/home/jovyan/.local/lib/python3.6/site-packages/distributed/comm/core.py&quot;, line 297, in _
    handshake = await asyncio.wait_for(comm.read(), 1)
  File &quot;/cvmfs/sft.cern.ch/lcg/releases/Python/3.6.5-f74f0/x86_64-centos7-gcc8-opt/lib/python3.6/asyncio/tasks.py&quot;, line 351, in wait_for
    yield from waiter
concurrent.futures._base.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File &quot;/home/jovyan/.local/lib/python3.6/site-packages/distributed/comm/core.py&quot;, line 304, in _
    raise CommClosedError() from e
distributed.comm.core.CommClosedError
</code></pre>
<p>However, if I check the pods, the cluster has actually been created, and I can scale it up, and everything seems fine in the dashboard (I can even see the workers).</p>
<p>However, I cannot get the client:</p>
<pre><code>&gt; client = cluster.get_client()
Task exception was never retrieved
future: &lt;Task finished coro=&lt;connect.&lt;locals&gt;._() done, defined at /home/jovyan/.local/lib/python3.6/site-packages/distributed/comm/core.py:288&gt; exception=CommClosedError()&gt;
Traceback (most recent call last):
  File &quot;/home/jovyan/.local/lib/python3.6/site-packages/distributed/comm/core.py&quot;, line 297, in _
    handshake = await asyncio.wait_for(comm.read(), 1)
  File &quot;/cvmfs/sft.cern.ch/lcg/releases/Python/3.6.5-f74f0/x86_64-centos7-gcc8-opt/lib/python3.6/asyncio/tasks.py&quot;, line 351, in wait_for
    yield from waiter
concurrent.futures._base.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File &quot;/home/jovyan/.local/lib/python3.6/site-packages/distributed/comm/core.py&quot;, line 304, in _
    raise CommClosedError() from e
distributed.comm.core.CommClosedError
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
~/.local/lib/python3.6/site-packages/distributed/comm/core.py in connect(addr, timeout, deserialize, handshake_overrides, **connection_args)
    321             if not comm:
--&gt; 322                 _raise(error)
    323         except FatalCommClosedError:

~/.local/lib/python3.6/site-packages/distributed/comm/core.py in _raise(error)
    274         )
--&gt; 275         raise IOError(msg)
    276 

OSError: Timed out trying to connect to 'gateway://traefik-dask-gateway:80/jhub.0373ea68815d47fca6a6c489c8f7263a' after 100 s: connect() didn't finish in time

During handling of the above exception, another exception occurred:

OSError                                   Traceback (most recent call last)
&lt;ipython-input-19-affca45186d3&gt; in &lt;module&gt;
----&gt; 1 client = cluster.get_client()

~/.local/lib/python3.6/site-packages/dask_gateway/client.py in get_client(self, set_as_default)
   1066             set_as_default=set_as_default,
   1067             asynchronous=self.asynchronous,
-&gt; 1068             loop=self.loop,
   1069         )
   1070         if not self.asynchronous:

~/.local/lib/python3.6/site-packages/distributed/client.py in __init__(self, address, loop, timeout, set_as_default, scheduler_file, security, asynchronous, name, heartbeat_interval, serializers, deserializers, extensions, direct_to_workers, connection_limit, **kwargs)
    743             ext(self)
    744 
--&gt; 745         self.start(timeout=timeout)
    746         Client._instances.add(self)
    747 

~/.local/lib/python3.6/site-packages/distributed/client.py in start(self, **kwargs)
    948             self._started = asyncio.ensure_future(self._start(**kwargs))
    949         else:
--&gt; 950             sync(self.loop, self._start, **kwargs)
    951 
    952     def __await__(self):

~/.local/lib/python3.6/site-packages/distributed/utils.py in sync(loop, func, callback_timeout, *args, **kwargs)
    337     if error[0]:
    338         typ, exc, tb = error[0]
--&gt; 339         raise exc.with_traceback(tb)
    340     else:
    341         return result[0]

~/.local/lib/python3.6/site-packages/distributed/utils.py in f()
    321             if callback_timeout is not None:
    322                 future = asyncio.wait_for(future, callback_timeout)
--&gt; 323             result[0] = yield future
    324         except Exception as exc:
    325             error[0] = sys.exc_info()

/cvmfs/sft.cern.ch/lcg/views/LCG_96python3/x86_64-centos7-gcc8-opt/lib/python3.6/site-packages/tornado/gen.py in run(self)
   1131 
   1132                     try:
-&gt; 1133                         value = future.result()
   1134                     except Exception:
   1135                         self.had_exception = True

~/.local/lib/python3.6/site-packages/distributed/client.py in _start(self, timeout, **kwargs)
   1045 
   1046         try:
-&gt; 1047             await self._ensure_connected(timeout=timeout)
   1048         except (OSError, ImportError):
   1049             await self._close()

~/.local/lib/python3.6/site-packages/distributed/client.py in _ensure_connected(self, timeout)
   1103         try:
   1104             comm = await connect(
-&gt; 1105                 self.scheduler.address, timeout=timeout, **self.connection_args
   1106             )
   1107             comm.name = &quot;Client-&gt;Scheduler&quot;

~/.local/lib/python3.6/site-packages/distributed/comm/core.py in connect(addr, timeout, deserialize, handshake_overrides, **connection_args)
    332                 backoff = min(backoff, 1)  # wait at most one second
    333             else:
--&gt; 334                 _raise(error)
    335         else:
    336             break

~/.local/lib/python3.6/site-packages/distributed/comm/core.py in _raise(error)
    273             error,
    274         )
--&gt; 275         raise IOError(msg)
    276 
    277     backoff = 0.01

OSError: Timed out trying to connect to 'gateway://traefik-dask-gateway:80/jhub.0373ea68815d47fca6a6c489c8f7263a' after 100 s: Timed out trying to connect to 'gateway://traefik-dask-gateway:80/jhub.0373ea68815d47fca6a6c489c8f7263a' after 100 s: connect() didn't finish in time
</code></pre>
<p>How do I debug this? Any pointer would be greatly appreciated.</p>
<p>I already tried increasing all the timeouts, but nothing changed:</p>
<pre><code>os.environ[&quot;DASK_DISTRIBUTED__COMM__TIMEOUTS__CONNECT&quot;]=&quot;100s&quot;
os.environ[&quot;DASK_DISTRIBUTED__COMM__TIMEOUTS__TCP&quot;]=&quot;600s&quot;
os.environ[&quot;DASK_DISTRIBUTED__COMM__RETRY__DELAY__MIN&quot;]=&quot;1s&quot;
os.environ[&quot;DASK_DISTRIBUTED__COMM__RETRY__DELAY__MAX&quot;]=&quot;60s&quot;
</code></pre>
<p>I wrote a tutorial about the steps I took to deploy dask gateway, see <a href="https://zonca.dev/2020/08/dask-gateway-jupyterhub.html" rel="nofollow noreferrer">https://zonca.dev/2020/08/dask-gateway-jupyterhub.html</a>.
I am quite sure this was working fine a few weeks ago, but I cannot identify what changed...</p>

<p>You need to use compatible versions of <code>dask</code> and <code>dask-distributed</code> everywhere.</p>
<p>I believe this is an error related to an upgrade in the communications protocol for <code>distributed</code>. See <a href="https://github.com/dask/dask-gateway/issues/316#issuecomment-702947730" rel="nofollow noreferrer">https://github.com/dask/dask-gateway/issues/316#issuecomment-702947730</a></p>
<p>These are the pinned versions of the dependencies for the docker images as of Nov 10, 2020 (in <code>conda</code> <code>environment.yml</code> compatible format):</p>
<pre><code>  - python=3.7.7
  - dask=2.21.0
  - distributed=2.21.0
  - cloudpickle=1.5.0
  - toolz=0.10.0
</code></pre>
