<p>I am currently running metrics server inside my pod. The data is being sent inside pod at localhost:9090. I am able to get the data inside pod via curl. The deployment.yaml is annotated to scrape the data, but I don't see any new metrics in pod. What am I doing wrong?</p>
<p>metrics I see inside pod :</p>
<pre><code>cpu_usage{process=&quot;COMMAND&quot;, pid=&quot;PID&quot;} %CPU
cpu_usage{process=&quot;/bin/sh&quot;, pid=&quot;1&quot;} 0.0
cpu_usage{process=&quot;sh&quot;, pid=&quot;8&quot;} 0.0
cpu_usage{process=&quot;/usr/share/filebeat/bin/filebeat-god&quot;, pid=&quot;49&quot;} 0.0
cpu_usage{process=&quot;/usr/share/filebeat/bin/filebeat&quot;, pid=&quot;52&quot;} 0.0
cpu_usage{process=&quot;php-fpm:&quot;, pid=&quot;66&quot;} 0.0
cpu_usage{process=&quot;php-fpm:&quot;, pid=&quot;67&quot;} 0.0
cpu_usage{process=&quot;php-fpm:&quot;, pid=&quot;68&quot;} 0.0
cpu_usage{process=&quot;nginx:&quot;, pid=&quot;69&quot;} 0.0
cpu_usage{process=&quot;nginx:&quot;, pid=&quot;70&quot;} 0.0
cpu_usage{process=&quot;nginx:&quot;, pid=&quot;71&quot;} 0.0
cpu_usage{process=&quot;/bin/sh&quot;, pid=&quot;541&quot;} 0.0
cpu_usage{process=&quot;bash&quot;, pid=&quot;556&quot;} 0.0
cpu_usage{process=&quot;/bin/sh&quot;, pid=&quot;1992&quot;} 0.0
cpu_usage{process=&quot;ps&quot;, pid=&quot;1993&quot;} 0.0
cpu_usage{process=&quot;/bin/sh&quot;, pid=&quot;1994&quot;} 0.0
</code></pre>
<p>deployment.yaml</p>
<pre><code>  template: 
    metadata:
      labels: 
        app: supplier-service
      annotations:
        prometheus.io/path: /
        prometheus.io/scrape: 'true'
        prometheus.io/port: '9090'



          ports: 
            - containerPort: 80
            - containerPort: 443
            - containerPort: 9090
</code></pre>
<p>prometheus.yml</p>
<pre><code>global:
  scrape_interval: 15s # By default, scrape targets every 15seconds. # Attach these labels to any time series or alerts when #communicating with external systems (federation, re$
  external_labels:
    monitor: 'codelab-monitor'
# Scraping Prometheus itself
scrape_configs:
- job_name: 'prometheus'
  scrape_interval: 5s
  static_configs:
  - targets: ['localhost:9090']
- job_name: 'kubernetes-service-endpoints'
  scrape_interval: 5s
  kubernetes_sd_configs:
  - role: endpoints
  relabel_configs:
  - action: labelmap
    regex: __meta_kubernetes_service_label_(.+)
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
    action: keep
    regex: true
  - source_labels: [__address__]
    action: replace
    regex: ([^:]+)(?::\d+)?
    replacement: $1:9090
    target_label: __address__
  - source_labels: [__meta_kubernetes_service_name]
    action: replace
    target_label: kubernetes_name
</code></pre>
<p>Port numbers are correct. What am I doing wrong?</p>

We are Borg
<p>Your <code>kubernetes_sd_configs</code> is configured to look for <code>endpoints</code>, which are created by services. Do you have an endpoint created for your service? You could check with <code>kubectl get endpoints</code> in your namespace. If you don't want to create a service, I suppose you could configure Prometheus to scrape pod targets too, check <a href="https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config" rel="nofollow noreferrer">the docs</a> for more info.</p>
<p>Also the <a href="https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels" rel="nofollow noreferrer">documentation for metrics and labels</a> say the metric name must match the regular expression <code>[a-zA-Z_:][a-zA-Z0-9_:]*</code>, so the dash (<code>-</code>) in your metric name might be an issue too.</p>
