<p>I am running a standalone Apache Spark cluster in a Kubernetes environment. </p>

<p>There is a need to export metrics to Prometheus and then finally display them in Grafana. </p>

<p>I found that installing a Graphite exporter was the simplest solution to do this since I am experiencing some trouble with getting all Spark metrics while only using the JMX exporter.</p>

<p>The thing I am having trouble with is to create mappings from graphite output to output that is parsable by prometheus templating.</p>

<p>For instance, I want to be able to parse</p>

<p><code>app_20200120105608_0736_0_executor_threadpool_completeTasks</code></p>

<p>so that it matches to something similar to this:</p>

<pre><code>- match: '*.*.threadpool.*.*'
  name: app_data
  labels:
    application: $1 // app_20200120105608_0736
    executor_id: $2 // 0
    type: $3 // threadpool
    qty: $4 // completeTasks
</code></pre>

<p>I am not convinced that this will be the best solution overall so any other suggestions are welcome (For instance how this could be done in a proper way with using a JMX exporter only while also getting Spark app data.)</p>

toerq
<p>If I understand correctly you try to build something like <code>Spark -&gt; Graphite -&gt; Prometheus -&gt; Grafana</code>. Avoid to do that since Graphite adds overhead to your monitoring system.</p>

<p>You have several options available:</p>

<ul>
<li>Query Graphite directly from Grafana with <a href="https://grafana.com/docs/grafana/latest/features/datasources/graphite/" rel="nofollow noreferrer">Graphite data source</a></li>
<li>Setup Jmx Exporter properly. You can refer the <a href="https://stackoverflow.com/questions/49488956/monitoring-apache-spark-with-prometheus">discussion</a> to get the idea on how to do it with jmx-exporter. Also I can help you to deal with the errors you have if you share the problems you have with it.</li>
<li>Setup <a href="https://github.com/helm/charts/tree/master/stable/prometheus-pushgateway" rel="nofollow noreferrer">Prometheus Push Gateway</a> and the corresponding <a href="https://github.com/banzaicloud/spark-metrics/blob/master/PrometheusSink.md" rel="nofollow noreferrer">Spark Prometheus sink</a>. Note that this solution is advised for short running jobs. If you have long running jobs the Jmx Exporter is preferable.</li>
</ul>

<p>Hope it helps.</p>
