title: prometheus node-exporter on kubernetes
body: <p>I have deployed prometheus on kubernetes cluster (EKS). I was able to successfully scrape <code>prometheus</code> and <code>traefik</code> with following </p>
  
  <pre><code>scrape_configs:
    # A scrape configuration containing exactly one endpoint to scrape:
  
    # The job name is added as a label `job=&lt;job_name&gt;` to any timeseries scraped from this config.
    - job_name: 'prometheus'
      # Override the global default and scrape targets from this job every 5 seconds.
      scrape_interval: 5s
      static_configs:
        - targets: ['prometheus.kube-monitoring.svc.cluster.local:9090']
  
    - job_name: 'traefik'
      static_configs:
        - targets: ['traefik.kube-system.svc.cluster.local:8080']
  </code></pre>
  
  <p>But node-exporter deployed as <code>DaemonSet</code> with following definition is not exposing the node metrics. </p>
  
  <pre><code>apiVersion: apps/v1
  kind: DaemonSet
  metadata:
    name: node-exporter
    namespace: kube-monitoring
  spec:
    selector:
      matchLabels:
        app: node-exporter
    template:
      metadata:
        name: node-exporter
        labels:
          app: node-exporter
      spec:
        hostNetwork: true
        hostPID: true
        containers:
        - name: node-exporter
          image: prom/node-exporter:v0.18.1
          args:
          - "--path.procfs=/host/proc"
          - "--path.sysfs=/host/sys"
          ports:
          - containerPort: 9100
            hostPort: 9100
            name: scrape
          resources:
            requests:
              memory: 30Mi
              cpu: 100m
            limits:
              memory: 50Mi
              cpu: 200m
          volumeMounts:
          - name: proc
            readOnly:  true
            mountPath: /host/proc
          - name: sys
            readOnly: true
            mountPath: /host/sys
        tolerations:
          - effect: NoSchedule
            operator: Exists
        volumes:
        - name: proc
          hostPath:
            path: /proc
        - name: sys
          hostPath:
            path: /sys
  </code></pre>
  
  <p>and following scrape_configs in prometheus</p>
  
  <pre><code>scrape_configs:
    - job_name: 'kubernetes-nodes'
      scheme: http
      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - target_label: __address__
        replacement: kubernetes.kube-monitoring.svc.cluster.local:9100
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        target_label: __metrics_path__
        replacement: /api/v1/nodes/${1}/proxy/metrics 
  </code></pre>
  
  <p>I also tried to <code>curl http://localhost:9100/metrics</code> from one of the container, but got <code>curl: (7) Failed to connect to localhost port 9100: Connection refused</code></p>
  
  <p>What I am missing here with the configuration ?</p>
  
  <p>After suggestion to install Prometheus by helm, I didn't install it on test cluster and tried to compare my original configuration with helm installed Prometheus. </p>
  
  <p>Following pods were running :</p>
  
  <pre><code>NAME                                                     READY   STATUS    RESTARTS   AGE
  alertmanager-prometheus-prometheus-oper-alertmanager-0   2/2     Running   0          4m33s
  prometheus-grafana-66c7bcbf4b-mh42x                      2/2     Running   0          4m38s
  prometheus-kube-state-metrics-7fbb4697c-kcskq            1/1     Running   0          4m38s
  prometheus-prometheus-node-exporter-6bf9f                1/1     Running   0          4m38s
  prometheus-prometheus-node-exporter-gbrzr                1/1     Running   0          4m38s
  prometheus-prometheus-node-exporter-j6l9h                1/1     Running   0          4m38s
  prometheus-prometheus-oper-operator-648f9ddc47-rxszj     1/1     Running   0          4m38s
  prometheus-prometheus-prometheus-oper-prometheus-0       3/3     Running   0          4m23s
  </code></pre>
  
  <p>I didn't find any configuration for node exporter in pod <code>prometheus-prometheus-prometheus-oper-prometheus-0</code> at <code>/etc/prometheus/prometheus.yml</code></p>