<p>I'm trying to implement in Kubernetes Dask distributed with one scheduler and three workers. I have one pod (<code>frontend.yaml</code>) for the scheduler and three other pods (<code>replicas: 3 in worker.yaml</code>) for the workers. The problem is that the workers are trying to connect to the scheduler and get a timeout as they don't know the IP address of the scheduler. Since these IP addresses are defined dynamically by Kubernetes I need to understand how this should be configured.</p>
<p>The IP addresses of the scheduler and the workers are determined dynamically:</p>
<pre><code>sofer@abc:~/aks$ kubectl get pods -o wide
NAME                          READY   STATUS             RESTARTS   AGE     IP          
img-python-6ff4d88876-n97g8   2/2     Running            0          4m42s   10.1.0.34  
worker-5f6d948874-fn67p       1/1     Running            4          4m22s   10.1.0.18  
worker-5f6d948874-hhjs7       0/1     Running            3          4m22s   10.1.0.38  
worker-5f6d948874-jbh4k       1/1     Running            4          4m22s   10.1.0.22
</code></pre>
<p>frontend.yaml</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: img-python
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: cont-python
  template:
    metadata:
      labels:
        app: cont-python
    spec:
      containers:
        - name: cont-scheduler
          image: xxx.azurecr.io/img-python-001:latest
          command: [&quot;dask-scheduler&quot;]
          ports:
            - containerPort: 8786
            - containerPort: 8787
</code></pre>
<p>worker.yaml</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: worker
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: cont-worker
  template:
    metadata:
      labels:
        app: cont-worker
    spec:
      containers:
        - name: cont-worker
          image: sofacr.azurecr.io/img-python-001:latest
          workingDir: /code
          command: [&quot;dask-worker&quot;]
          args: [&quot;scheduler:8786&quot;]
          env:
            - name: PYTHONPATH
              value: &quot;/code&quot;
            - name: SCHEDULER_ADDRESS
              value: &quot;scheduler&quot;
            - name: SCHEDULER_PORT
              value: &quot;8786&quot;
</code></pre>

<p>Thank you <a href="https://stackoverflow.com/users/3888719/michael-delgado">Michael Delgado</a> . Posting your suggestion as an answer to help community members.</p>
<p>You can check the <a href="https://github.com/dask/helm-chart/tree/main/dask" rel="nofollow noreferrer">dask</a> and <a href="https://github.com/dask/helm-chart/tree/main/daskhub" rel="nofollow noreferrer">daskhub</a> helm charts. These set up full deployments with <a href="https://kubernetes.dask.org/en/latest/" rel="nofollow noreferrer">dask_kubernetes</a> or <a href="https://gateway.dask.org/" rel="nofollow noreferrer">dask-gateway</a>. Rather than manually provisioning workers and specifying the IP address, these deployments create the scheduler and workers dynamically</p>
<p>As suggested by <a href="https://stackoverflow.com/users/1362485/ps0604">ps0604</a>, to deploy dask using the Dask Helm chart, you can refer to <a href="https://docs.dask.org/en/latest/setup/kubernetes-helm.html#kubernetes-helm-single" rel="nofollow noreferrer">Helm Install Dask for a Single User</a></p>
