<p>How can I use Prometheus (Which uses a 'pull' approach) to monitor my Spark clusters when I spin up a new cluster every time I need, which means I don't have one steady monitoring endpoint.</p>

idan ahal
<p>It depends a bit on how / where you spin up your clusters. When using Kubernetes, you can use <a href="https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config" rel="nofollow noreferrer">Kubernetes SD configurations</a> to discover your targets or use the <a href="https://github.com/prometheus-operator/prometheus-operator" rel="nofollow noreferrer">Prometheus Operator</a> (see also <a href="https://dzlab.github.io/bigdata/2020/07/03/spark3-monitoring-2/" rel="nofollow noreferrer">here</a>).</p>
<p>If running on EMR you could use bootstrap actions to configure the <a href="https://github.com/prometheus/node_exporter" rel="nofollow noreferrer">node_exporter</a> and <a href="https://github.com/prometheus/jmx_exporter" rel="nofollow noreferrer">jmx_exporter</a>. See <a href="https://github.com/vinicelms/emr-monitoring-prometheus-grafana" rel="nofollow noreferrer">this</a> for an example how to do so.</p>
<p>And alternatively, you could use the <a href="https://prometheus.io/docs/practices/pushing/" rel="nofollow noreferrer">Prometheus Pushgateway</a> together with a custom Spark sink (<a href="https://github.com/banzaicloud/spark-metrics/blob/master/PrometheusSink.md" rel="nofollow noreferrer">example</a>) for this use case.</p>
