title: How to send JVM metrics of Spark to Prometheus in Kubernetes
body: <p>I am using the Spark operator to run Spark on Kubernetes. (<a href="https://github.com/GoogleCloudPlatform/spark-on-k8s-operator" rel="nofollow noreferrer">https://github.com/GoogleCloudPlatform/spark-on-k8s-operator</a>)</p>
  
  <p>I am trying to run a Java agent in Spark driver and executor pods and send the metrics through a Kubernetes service to Prometheus operator.</p>
  
  <p>I am using this example 
  <a href="https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/examples/spark-pi-prometheus.yaml" rel="nofollow noreferrer">https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/examples/spark-pi-prometheus.yaml</a></p>
  
  <p>Java agent is exposing the metrics on port 8090 for a short time (I can validate that with port-forwarding kubctl port-forward &lt; spark-driver-pod-name > 8090:8090 ), also the service is also exposing the metrics for few mins ( can validate that with port-forwarding kubctl port-forward svc/&lt; spark-service-name > 8090:8090 ). </p>
  
  <p>Promethues is able to register these pod's URL in the prometheus, but when it is trying to scrape the metrics(runs for every 30 seconds), the pod's URL is down. </p>
  
  <p>How can i make the Java agent JMX exporter to run long, until the driver and executors completed the job. could you please guide or help me here, who have come across this scenario before?</p>