<p>I am trying to use <code>spark</code> on <code>Kubernetes</code>. Idea is to using <code>spark-submit</code> to <code>k8s</code> cluster which is running <code>prometheus operator</code>. Now I know that <code>prometheus operator</code> can respond to <code>ServiceMonitor</code> yaml but I am confused how to provide some of the things required in the <code>YAML</code> using <code>spark-submit</code></p>

<p>Here is the <code>YAML</code>:</p>

<pre><code>apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: sparkloads-metrics
  namespace: runspark
spec:
  selector:
    matchLabels:
      app: runspark
  namespaceSelector:
    matchNames:
    - runspark
  endpoints:
  - port: 8192 ---&gt; How to provide the name to port using `spark-submit`
    interval: 30s
    scheme: http
</code></pre>

devnull
<p>You cannot provide additional ports and their names to the Service created by <code>SparkSubmit</code> yet (Spark v2.4.4). Things can change in the later versions.</p>

<p>What you can do is create additional Kubernetes <code>Service</code> (Spark Monitoring Service, eg. of type Cluster IP) per Spark job after the Job submission with <code>SparkSubmit</code>, for instance running <code>spark-submit ... &amp;&amp; kubectl apply ...</code> . Or use any of the available Kubernetes clients with the language of your choice.</p>

<p>Note that you can use <a href="https://kubernetes.io/docs/concepts/workloads/controllers/garbage-collection/#owners-and-dependents" rel="nofollow noreferrer">Kubernetes OwnerReference</a> to configure automatic <code>Service</code> deletion/GC on <code>Spark Driver Pod</code> deletion.</p>

<p>Then you can supply the <code>ServiceMonitor</code>'s via the Prometheus Operator Helm values:</p>

<pre><code>prometheus:
  additionalServiceMonitors:
     - name: spark-metrics           # &lt;- Spark Monitoring Service name
      selector:
        matchLabels:
          k8s-app: spark-metrics     # &lt;- Spark Monitoring Service label
      namespaceSelector:
        any: true
      endpoints:
      - interval: 10s
        port: metrics                # &lt;- Spark Monitoring Service port name
</code></pre>

<p>Be aware of the fact that Spark doesn't provide a way to customize <code>Spark Pod</code>s yet, so your Pod ports which should expose metrics are not exposed on a Pod level and won't be accessible via <code>Service</code>. To overcome it you can add additional <code>EXPOSE ... 8088</code> statement in the <code>Dockerfile</code> and rebuild Spark image.</p>

<p>This guide should help you to setup Spark monitoring with PULL strategy using for example <a href="https://github.com/prometheus/jmx_exporter" rel="nofollow noreferrer">Jmx Exporter</a>.</p>

<p>There is an alternative (though it is recommended only for short-running Spark jobs, but you can try it in your environment if you do not run huge workloads):</p>

<ul>
<li>Deploy <a href="https://github.com/helm/charts/tree/master/stable/prometheus-pushgateway" rel="nofollow noreferrer">Prometheus Pushgateway</a> and integrate it with your Prometheus Operator</li>
<li>Configure <a href="https://github.com/banzaicloud/spark-metrics/blob/master/PrometheusSink.md" rel="nofollow noreferrer">Spark Prometheus Sink</a></li>
</ul>

<p>By doing that your Spark Pods will PUSH metrics to the Gateway and Prometheus will PULL them from the Gateway in order.</p>

<p>You can refer the <a href="https://github.com/jahstreet/spark-on-kubernetes-helm/tree/master/charts/spark-monitoring" rel="nofollow noreferrer">Spark Monitoring Helm chart</a> example with the Prometheus Operator and Prometheus Pushgateway combined.</p>

<p>Hope it helps.</p>
