<p>I am trying to scrape metrics for spark driver and executor using javaagent with below options. I have Prometheus in kubernetes cluster and I am running this spark application outside the kubernetes cluster.</p>
<pre><code>spark.executor.extraJavaOptions=-javaagent:/opt/clkd/prometheus/jmx_prometheus_javaagent-0.3.1.jar=53700:executor_pattern.yaml
</code></pre>
<p>but I got below exception since both executors running on the same machine</p>
<pre><code>Caused by: java.net.BindException: Address already in use .... 
</code></pre>
<p>I see many have posted the same question but I couldn't find the answer. Please let me know how can I resolve this issue.</p>

<p>I think that you need to switch from the pull-based monitoring, to push-based monitoring.  For things, such as Spark jobs it makes more sense, as they aren't running all the time.  For that you have some alternatives:</p>
<ul>
<li>Spark Prometheus Sink from Banzai Cloud as outlined in their <a href="https://banzaicloud.com/blog/spark-prometheus-sink/" rel="nofollow noreferrer">blog post</a></li>
<li>Setup GraphiteSink as described in the <a href="https://spark.apache.org/docs/2.4.6/monitoring.html" rel="nofollow noreferrer">Spark documentation</a>, and point it to the <a href="https://github.com/prometheus/graphite_exporter" rel="nofollow noreferrer">https://github.com/prometheus/graphite_exporter</a>, and then <strong>scrape</strong> metrics from that exporter</li>
</ul>
<hr />
<p>Initial answer:</p>
<p>You can't have 2 processes listening on the same port, so just bind Prometheus from different jobs onto the different ports. Port is the number after the <code>jmx_prometheus_javaagent-0.3.1.jar=</code>, and before <code>:</code> character - in your case it's <code>53700</code>. So you can use one port for one task, and another port (maybe <code>53701</code>) for 2nd task...</p>
