<p>I am using the Spark operator to run Spark on Kubernetes. (<a href="https://github.com/GoogleCloudPlatform/spark-on-k8s-operator" rel="nofollow noreferrer">https://github.com/GoogleCloudPlatform/spark-on-k8s-operator</a>)</p>

<p>I am trying to run a Java agent in Spark driver and executor pods and send the metrics through a Kubernetes service to Prometheus operator.</p>

<p>I am using this example 
<a href="https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/examples/spark-pi-prometheus.yaml" rel="nofollow noreferrer">https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/examples/spark-pi-prometheus.yaml</a></p>

<p>Java agent is exposing the metrics on port 8090 for a short time (I can validate that with port-forwarding kubctl port-forward &lt; spark-driver-pod-name > 8090:8090 ), also the service is also exposing the metrics for few mins ( can validate that with port-forwarding kubctl port-forward svc/&lt; spark-service-name > 8090:8090 ). </p>

<p>Promethues is able to register these pod's URL in the prometheus, but when it is trying to scrape the metrics(runs for every 30 seconds), the pod's URL is down. </p>

<p>How can i make the Java agent JMX exporter to run long, until the driver and executors completed the job. could you please guide or help me here, who have come across this scenario before?</p>

<p>Either Prometheus needs to scrape the metrics of every 5 seconds (chances are that metrics may not be accurate), or you need to use pushgateway, like mentioned in this blog(<a href="https://banzaicloud.com/blog/spark-monitoring/" rel="nofollow noreferrer">https://banzaicloud.com/blog/spark-monitoring/</a>) to push the metrics to Prometheus</p>
<p><strong>Pushing</strong> the metrics <strong>to</strong> <strong>Prometheus</strong>, is a best practice for <strong>batch jobs</strong>.
<strong>Pulling</strong> the metrics <strong>from</strong> <strong>Prometheus</strong> is a best approach for <strong>long running services</strong> (ex:REST Services)</p>
