<p>I have prometheus config that works on my cluster deployed by terraform. Now, I would like to setup Prometheus using the same prometheus.yml locally (outside terraform). I created a new project to set up Promethese using docker-compose and use the same prometheus.yml file but when I go to prometheus site, it seems like the metrics for kubernetes is not available, such as these metrics about kubernetes containers:
container_cpu_usage_seconds_total
container_cpu_load_average_10s
container_memory_usage_bytes
container_memory_rss</p>
<p>Could you please let me know what I am missing in my project to make this work?</p>
<p>This is <code>prometheus.yml</code></p>
<pre><code>global:
  scrape_interval: 15s
scrape_configs:
- job_name: 'prometheus'
  static_configs:
  - targets: ['localhost:9090']
- job_name: 'kube-state-metrics'
  static_configs:
    - targets: ['10.36.1.10']
    - targets: ['10.36.2.6']
    - targets: ['10.36.1.12']
- job_name: 'kubernetes-pods'
  kubernetes_sd_configs:
  - api_server: https://10.36.1.10:6443
    role: node
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    tls_config:
      insecure_skip_verify: true
  - api_server: https://10.36.2.6:6443
    role: node
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    tls_config:
      insecure_skip_verify: true
  - api_server: https://10.36.1.12:6443
    role: node
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    tls_config:
      insecure_skip_verify: true
  - role: pod
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
    action: keep
    regex: true
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
    action: replace
    regex: ([^:]+)(?::\d+)?;(\d+)
    replacement: $1:$2
    target_label: __address__
  - action: labelmap
    regex: __meta_kubernetes_pod_label_(.+)
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_pod_label_component]
    action: replace
    target_label: job
  - source_labels: [__meta_kubernetes_pod_name]
    action: replace
    target_label: kubernetes_pod_name

</code></pre>
<p>And, this is <code>docker-compose.yml</code></p>
<pre><code>version: '3'

services:
  prometheus:
    image: prom/prometheus:v2.21.0
    ports:
      - 9000:9090
    volumes:
      - ./prometheus:/etc/prometheus
      - prometheus-data:/prometheus
    command: --web.enable-lifecycle  --config.file=/etc/prometheus/prometheus.yml

  grafana:
    image: grafana/grafana:$GRAFANA_VERSION
    environment:
      GF_SECURITY_ADMIN_USER: $GRAFANA_ADMIN_USER
      GF_SECURITY_ADMIN_PASSWORD: $GRAFANA_ADMIN_PASSWORD
    ports:
      - 3000:3000
    volumes:
      - grafana-storage:/var/lib/grafana
    depends_on:
      - prometheus
    networks:
      - internal

networks:
  internal:

volumes:
  prometheus-data:
  grafana-storage:
</code></pre>
<p><a href="https://i.stack.imgur.com/eA6VZ.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/eA6VZ.png" alt="ipK8s" /></a></p>
<p><a href="https://i.stack.imgur.com/YxuiX.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/YxuiX.png" alt="node_exporter" /></a></p>

<p>You are running the <strong>Prometheus</strong> at the local and server level both.</p>
<p>server one working fine and getting the metrics of Kubernetes containers as it's on kubernetes.</p>
<p>while docker-compose one not working due to you are running it locally on docker not on kubernetes cluster.</p>
<p>it's the issue of the <strong>Target</strong>  your Prometheus is not getting the metrics of your <strong>Kubernetes</strong> cluster.</p>
<p>for example, you are running the <strong>Prometheus</strong> locally but want to monitor the external <strong>Kubernetes</strong> cluster you have to expose your <strong>Kube-state-metrics</strong> service using <strong>IP</strong>.</p>
<p>in that case your Prometheus config will have job like</p>
<pre><code>scrape_configs:
  - job_name: 'kube-state-metrics'
    static_configs:
      - targets: ['address'] //address of the k8s service IP
</code></pre>
<p>in your case you have to do something like</p>
<pre><code>kubernetes_sd_configs:
- api_server: https://&lt;ip&gt;:6443
  role: node
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  tls_config:
    insecure_skip_verify: true
</code></pre>
<p>this will fetch the metrics of kubernetes cluster and you can see data locally.</p>
<p>you can read this nice gist : <a href="https://gist.github.com/sacreman/b61266d2ec52cf3a1af7c278d9d93450" rel="nofollow noreferrer">https://gist.github.com/sacreman/b61266d2ec52cf3a1af7c278d9d93450</a></p>
