<p>I have a Spark Cluster with one Master and 4 Workers running in a 3 Node Kubernetes Cluster.
The Spark UI and Master are exposed through Ingress/NodePort and is hence accessible from outside the Kubernetes cluster.</p>

<p>But, the worker ports are not exposed.
<strong>Due to this, the Worker logs (stdout and stderr) are not accessible through the UI.</strong>
The logs URL is getting redirected to <code>&lt;Worker1_Pod_IP:8080&gt;</code> , <code>&lt;Worker2_Pod_IP:8080&gt;</code> and so on.</p>

<p><strong>My setup is such that there are two worker pods running on the same machine. So even if I NodePort expose the workers, there will be a conflict in ports as the same port will be assigned for the two workers on the machine.</strong>
Spark History server only provides Event Logs and not Worker logs.</p>

<p>How can this be solved? Is there a way the NodePort value can be dynamically assigned for the workers</p>

<p>I believe you are talking about the <code>SPARK_WORKER_WEBUI_PORT</code> and not the <code>SPARK_WORKER_PORT</code> as described below since that its assigned a random port.</p>

<p><a href="https://i.stack.imgur.com/W9nHa.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/W9nHa.png" alt="Spark"></a></p>

<p>This is a bit tricky because you can only expose a single port per node. If you have two spark workers per node you could create two deployments for your nodes, one that exposes the <code>SPARK_WORKER_WEBUI_PORT</code> on <code>8081</code> and another on <code>8082</code> and also make sure that only one pod is scheduled per node.</p>

<p>You can pass these values in the <a href="https://kubernetes.io/docs/tasks/inject-data-application/define-environment-variable-container/" rel="nofollow noreferrer">container environment varialbles</a></p>
