title: Monitoring Spark 3 applications with Prometheus
body: <p>Have some very basic questions around the pull mechanism with metrics and how Spark 3 applications can be monitored using Prometheus:</p>
  <ol>
  <li>Does the PrometheusServlet sink supported with Spark-3 contain all the metrics since the application start time? Are these metrics un-aggregated?</li>
  <li>Where/How is the metric information in the sink stored and what really happens when Prometheus scrapes the end point? If the end point displays all the metric information since the application start time, wouldn't the memory spent to store these metrics be a concern for long running spark applications?</li>
  <li>Does Prometheus fetch all the metrics (since the application start time) on every re-scrape? If not, how does it know the metric last scraped?</li>
  </ol>
  <p>Thanks.</p>