<p>Hell everyone, I have a problem with Apache Spark (version 3.3.1) on <code>k8s</code>.</p>
<p>In short:
When I run the statement</p>
<pre class="lang-py prettyprint-override"><code>print(sc.uiWebUrl)
</code></pre>
<p>within a pod, I would get a <code>URL</code> that is accessible from outside the <code>k8s</code> cluster.
Something like:</p>
<pre><code>http://{{my-ingress-host}}
</code></pre>
<p>Long story:
I want to create a workspace for Apache Spark on <code>k8s</code>, where the driver's pod, is the <strong>workspace</strong> that I work on. I want to let the client run Apache Spark either with <code>pyspark-shell</code> or with the <code>pyspark</code> python library.</p>
<p>In either way, I want that the UI's web url would be a one that is accessible from the outside world (outside the <code>k8s</code> cluster).
Why? Because of <code>UX</code>, I want to make my client's life easier.</p>
<p>Because I run on <code>k8s</code>, part of the configuration of my Apache Spark program is:</p>
<pre><code>spark.driver.host={{driver-service}}.{{drivers-namespace}}.svc.cluster.local
spark.driver.bindAddress=0.0.0.0
</code></pre>
<p>Because of that, the output of this code:</p>
<pre class="lang-py prettyprint-override"><code>print(sc.webUiUrl)
</code></pre>
<p>Would be:</p>
<pre><code>http://{{driver-service}}.{{drivers-namespace}}.svc.cluster.local:4040
</code></pre>
<p>Also in the pyspark-shell, the same address would be displayed.</p>
<p>So my question is, is there a way to change the ui web url's host to a host that I have defined in my <code>ingress</code> to make my client's life easier?
So the new output would be:</p>
<pre><code>http://{{my-defined-host}}
</code></pre>
<p>Other points I want to make sure to adjust the solution as much as possible:</p>
<ul>
<li>I don't have a <code>nginx</code> ingress in my k8s cluster. Maybe I have a <code>HAPROXY</code> ingress. But I would want to be coupled to my ingress implementation as <strong>least</strong> as possiable.</li>
<li>I would prefer that the client would need to configure Apache Spark as <strong>least</strong> as possible.</li>
<li>I would prefer that the ui web url of the Spark's context would be set when creating the context, meaning before the <code>pyspark-shell</code> displays the welcome screen.</li>
<li>I have tried messing with the <code>ui.proxy</code> configurations, and it haven't helped. And sometimes made things worst.</li>
</ul>
<p>Thanks ahead for everyone, any help would be appreciated.</p>

Avihai Shalom
<p>You can change your web UI's host to a host that you want by setting the <a href="https://spark.apache.org/docs/3.4.1/configuration.html#environment-variables" rel="nofollow noreferrer"><code>SPARK_PUBLIC_DNS</code></a> environment variable. This needs to be done on the driver, since the web UI runs on the driver.</p>
<p>To set the port for the web UI, you can do that using the <a href="https://spark.apache.org/docs/3.4.1/configuration.html#spark-ui" rel="nofollow noreferrer"><code>spark.ui.port</code></a> config parameter.</p>
<p>So putting both together using <code>spark-submit</code> for example, makes something like the following:</p>
<pre><code>bin/spark-submit \
   --class ... \
   --master k8s://... \
   ....
   ....
   ....
   --conf spark.kubernetes.driverEnv.SPARK_PUBLIC_DNS=YOUR_VALUE_HERE
   --conf spark.ui.port=YOUR_WANTED_PORT_HERE
   ...
</code></pre>
