<p>I'm working with prometheus to scrape k8s service metrics.</p>
<p>I created a service monitor for my service as below :</p>
<pre><code>apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: scmpoll-service-monitor-{{ .Release.Name }}
  namespace: {{ .Release.Namespace }}
  labels:
    app: scmpoll-{{ template &quot;jenkins-exporter.name&quot; . }}
    chart: {{ template &quot;jenkins-exporter.chart&quot; . }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
spec:
  namespaceSelector:
    matchNames:
    - {{ .Release.Namespace }}
  selector:
    matchLabels:
      app: scmpoll-{{ template &quot;jenkins-exporter.name&quot; . }}
      chart: {{ template &quot;jenkins-exporter.chart&quot; . }}
      release: {{ .Release.Name }}
  endpoints:
  - interval: 1440m
    targetPort: 9759
    path: /metrics
    port: http
</code></pre>
<p>I set interval: 1440m because I want prometheus to scrape data once a day.</p>
<p>After deploying the chart, the service monitor was added to prometheus targets but with status unknown and scrape duration 0s. The screenshot below:</p>
<p><a href="https://i.stack.imgur.com/JBXzy.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/JBXzy.png" alt="enter image description here" /></a></p>
<p>Also I have the service monitor added to prometheus config:</p>
<pre><code>- job_name: monitoring/scmpoll-service-monitor-my-release/0
  honor_timestamps: true
  scrape_interval: 1d
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
</code></pre>
<p>I can see that scrape-intervel is set to 1d (24h) but the state is unknown in the screenshot above. Do I have to wait for the next 24h and check or does this mean the configuration is wrong?</p>
<p>Test : i made test with scrape 20minutes and it was the same thing : status unknow with no metrics until the 20 minutes passed and status became UP and metrics were scraped.</p>
<p>I'm not working on prometheus-operator chart, it's an independent chart.</p>

<p>As mentioned, in general it's not advisable to use scrape intervals of more than 2 minutes in Prometheus (e.g. see <a href="https://stackoverflow.com/a/40233721/4747193">here</a>). This is due to the default <a href="https://prometheus.io/docs/prometheus/latest/querying/basics/#staleness" rel="nofollow noreferrer">staleness</a> period of 5 minutes, which means that a scrape interval of 2 minutes allows for one failed scrape without the metrics being treated as stale.</p>
<p>There's nothing wrong with scraping a target more often. So, you can leave the scrape interval at e.g. 60 seconds, even if the metrics don't change often.</p>
<p>If you can't scrape the target so often for some reason, you can use a <a href="https://github.com/prometheus/pushgateway" rel="nofollow noreferrer">Pushgateway</a>. The target pushes its metrics to the Pushgateway at its own pace (e.g. once per day) and Prometheus scrapes the metrics from the Pushgateway in its own interval (e.g. every 60 seconds).</p>
