<p>I'm working to setup Jupyter notebook servers on Kubernetes that are able to launch <code>pyspark</code>. Each user is able to have a multiple servers running at once, and would access each by navigating to the appropriate host combined with a path to the server's fully-qualified name. For example: <code>http://&lt;hostname&gt;/&lt;username&gt;/&lt;notebook server name&gt;</code>.</p>

<p>I have a top-level function defined that allows a user create a <code>SparkSession</code> that points to the Kubernetes master URL and sets their pod to be the Spark driver.</p>

<p>This is all well and good, but I would like to enable end users to access the URL for the Spark Web UI so that they can track their jobs. The <a href="https://spark.apache.org/docs/latest/running-on-kubernetes.html#accessing-driver-ui" rel="nofollow noreferrer">Spark on Kubernetes documentation</a> has port forwarding as their recommended scheme for achieving this. It seems to be that for any security-minded organization, allowing any random user to setup port forwarding in this way would be unacceptable.</p>

<p>I would like to use an <code>Ingress</code> Kubernetes definition to allow external access to the driver's Spark Web UI. I've setup something like the following:</p>

<pre><code># Service
apiVersion: v1
kind: Service
metadata:
  namespace: &lt;notebook namespae&gt;
  name: &lt;username&gt;-&lt;notebook server name&gt;-svc
spec:
  type: ClusterIP
  sessionAffinity: None
  selector:
    app: &lt;username&gt;-&lt;notebook server name&gt;-notebook
  ports:
  - name: app-svc-port
    protocol: TCP
    port: 8888
    targetPort: 8888
  - name: spark-ui-port
    protocol: TCP
    port: 4040
    targetPort: 4040

# Ingress
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  namespace: workspace
  name: &lt;username&gt;-&lt;notebook server name&gt;-ing
  annotations:
    kubernetes.io/ingress.class: traefik
spec:
  rules:
  - host: &lt;hostname&gt;
    http:
      paths:
      - path: /&lt;username&gt;/&lt;notebook server name&gt;
        backend:
          serviceName: &lt;username&gt;-&lt;notebook server name&gt;-svc
          servicePort: app-svc-port
      - path: /&lt;username&gt;/&lt;notebook server name&gt;/spark-ui
        backend:
          serviceName: &lt;username&gt;-&lt;notebook server name&gt;-svc
          servicePort: spark-ui-port
</code></pre>

<p>However, under this setup, when I navigate to <code>http://&lt;hostname&gt;/&lt;username&gt;/&lt;notebook server name&gt;/spark-ui/</code>, I'm redirected to <code>http://&lt;hostname&gt;/jobs</code>. This is because <code>/jobs</code> is the default entry point to Spark's Web UI. However, I don't have an ingress rule for that path, and can't set such a rule since every user's Web UI would collide with each other in the load balancer (unless I have a  misunderstanding, which is totally possible).</p>

<p>Under the <a href="https://spark.apache.org/docs/latest/configuration.html#spark-ui" rel="nofollow noreferrer">Spark UI configuration</a> settings, there doesn't seem to be a way to set a root path for the Spark session. You can change the port on which it runs, but what I'd like to do make the UI serve at something like: <code>http://&lt;hostname&gt;/&lt;username&gt;/&lt;notebook server name&gt;/spark-ui/&lt;jobs, stages, etc&gt;</code>. Is there really no way of changing what comes after the hostname of the URL and before the last part?</p>

<p>1: set your spark config</p>
<pre><code>spark.ui.proxyBase: /foo
</code></pre>
<p>2: Set the nginx annotations in Ingress</p>
<pre><code>annotations:
  nginx.ingress.kubernetes.io/proxy-redirect-from: http://$host/
  nginx.ingress.kubernetes.io/proxy-redirect-to: http://$host/foo/
</code></pre>
<p>3:Annotation to rewrite target:</p>
<pre><code>annotations:
  nginx.ingress.kubernetes.io/rewrite-target: /$2

spec:
  rules:
  - host: &lt;host&gt;
    http:
      paths:
      - backend:
          serviceName: &lt;service&gt;
          servicePort: &lt;port&gt;
        path: /foo(/|$)(.*)
</code></pre>
