<p>With spark-submit I launch application on a Kubernetes cluster. And I can see Spark-UI only when I go to the <a href="http://driver-pod:port" rel="noreferrer">http://driver-pod:port</a>.</p>

<p>How can I start Spark-UI History Server on a cluster?
How to make, that all running spark jobs are registered on the Spark-UI History Server. </p>

<p>Is this possible?</p>

JDev
<p>Yes it is possible. Briefly you will need to ensure following:</p>

<ul>
<li>Make sure all your applications store event logs in a specific location (<code>filesystem</code>, <code>s3</code>, <code>hdfs</code> etc).</li>
<li>Deploy the history server in your cluster with access to above event logs location.</li>
</ul>

<p>Now spark (by default) only read from the <code>filesystem</code> path so I will elaborate this case in details with <a href="https://github.com/GoogleCloudPlatform/spark-on-k8s-operator" rel="noreferrer">spark operator</a>:</p>

<ul>
<li>Create a <code>PVC</code> with a volume type that supports <a href="https://kubernetes.io/docs/concepts/storage/persistent-volumes/#volume-mode" rel="noreferrer">ReadWriteMany</a> mode. For example <code>NFS</code> volume. The following snippet assumes you have storage class for <code>NFS</code> (<code>nfs-volume</code>) already configured:</li>
</ul>

<pre><code>apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: spark-pvc
  namespace: spark-apps
spec:
  accessModes:
    - ReadWriteMany
  volumeMode: Filesystem
  resources:
    requests:
      storage: 5Gi
  storageClassName: nfs-volume
</code></pre>

<ul>
<li>Make sure all your spark applications have event logging enabled and at the correct path:</li>
</ul>

<pre><code>  sparkConf:
    "spark.eventLog.enabled": "true"
    "spark.eventLog.dir": "file:/mnt"
</code></pre>

<ul>
<li>With event logs volume mounted to each application (you can also use operator mutating web hook to centralize it ) pod. An example manifest with mentioned config is show below:</li>
</ul>

<pre><code>---
apiVersion: "sparkoperator.k8s.io/v1beta2"
kind: SparkApplication
metadata:
  name: spark-java-pi
  namespace: spark-apps

spec:
  type: Java
  mode: cluster

  image: gcr.io/spark-operator/spark:v2.4.4
  mainClass: org.apache.spark.examples.SparkPi
  mainApplicationFile: "local:///opt/spark/examples/jars/spark-examples_2.11-2.4.4.jar"

  imagePullPolicy: Always
  sparkVersion: 2.4.4
  sparkConf:
    "spark.eventLog.enabled": "true"
    "spark.eventLog.dir": "file:/mnt"
  restartPolicy:
    type: Never
  volumes:
    - name: spark-data
      persistentVolumeClaim:
        claimName: spark-pvc
  driver:
    cores: 1
    coreLimit: "1200m"
    memory: "512m"
    labels:
      version: 2.4.4
    serviceAccount: spark
    volumeMounts:
      - name: spark-data
        mountPath: /mnt
  executor:
    cores: 1
    instances: 1
    memory: "512m"
    labels:
      version: 2.4.4
    volumeMounts:
      - name: spark-data
        mountPath: /mnt

</code></pre>

<ul>
<li>Install spark history server mounting the shared volume. Then you will have access events in history server UI:</li>
</ul>

<pre><code>apiVersion: apps/v1beta1
kind: Deployment

metadata:
  name: spark-history-server
  namespace: spark-apps

spec:
  replicas: 1

  template:
    metadata:
      name: spark-history-server
      labels:
        app: spark-history-server

    spec:
      containers:
        - name: spark-history-server
          image: gcr.io/spark-operator/spark:v2.4.0

          resources:
            requests:
              memory: "512Mi"
              cpu: "100m"

          command:
            -  /sbin/tini
            - -s
            - --
            - /opt/spark/bin/spark-class
            - -Dspark.history.fs.logDirectory=/data/
            - org.apache.spark.deploy.history.HistoryServer

          ports:
            - name: http
              protocol: TCP
              containerPort: 18080

          readinessProbe:
            timeoutSeconds: 4
            httpGet:
              path: /
              port: http

          livenessProbe:
            timeoutSeconds: 4
            httpGet:
              path: /
              port: http

          volumeMounts:
            - name: data
              mountPath: /data
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: spark-pvc
          readOnly: true

</code></pre>

<p>Feel free to configure <code>Ingress</code>, <code>Service</code> for accessing the <code>UI</code>.
<a href="https://i.stack.imgur.com/C7BXT.png" rel="noreferrer"><img src="https://i.stack.imgur.com/C7BXT.png" alt="enter image description here"></a></p>

<p>Also you can use Google Cloud Storage, Azrue Blob Storage or AWS S3 as event log location. For this you will need to install some extra <code>jars</code> so I would recommend having a look at lightbend spark history server <a href="https://github.com/lightbend/spark-history-server-docker" rel="noreferrer">image</a> and <a href="https://github.com/helm/charts/tree/master/stable/spark-history-server" rel="noreferrer">charts</a>.</p>
