<p>I have to setup a monitoring environment for my EKS cluster.
Prometheus is running on external node and I am trying to use node exporter daemonset for getting metrics.
But on prometheus when I see the targets I am not able to see any target instead of just local host.</p>
<p><strong>Kubernetes_sd_config block</strong></p>
<pre><code>global:
  scrape_interval: 15s
scrape_configs:

- job_name: 'prometheus'
  scrape_interval: 15s
  static_configs:
    - targets: ['localhost:9100']


- job_name: 'kubernetes-apiservers'
  kubernetes_sd_configs:
  - role: endpoints
    api_server: https://{{ kubernetes_api_server_addr }}
    tls_config:
      insecure_skip_verify: true
    bearer_token_file: /etc/prometheus/token
  scheme: https
  tls_config:
    insecure_skip_verify: true
  bearer_token_file: /etc/prometheus/token
  relabel_configs:
  - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name]
    action: keep
    regex: default;kubernetes;https
  - target_label: __address__
    replacement: {{ kubernetes_api_server_addr }}


- job_name: 'kubernetes-kube-state'
  tls_config:
    insecure_skip_verify: true
  bearer_token_file: /etc/prometheus/token
  kubernetes_sd_configs:
  - role: pod
    api_server: https://{{ kubernetes_api_server_addr }}
    tls_config:
      insecure_skip_verify: true
    bearer_token_file: /etc/prometheus/token
  scheme: https
  relabel_configs:
  - action: labelmap
    regex: __meta_kubernetes_pod_label_(.+)
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_pod_name]
    action: replace
    target_label: kubernetes_pod_name
  - source_labels: [__meta_kubernetes_pod_label_grafanak8sapp]
    regex: .*true.*
    action: keep
  - target_label: __address__
    replacement: {{ kubernetes_api_server_addr }}
  - source_labels: ['__meta_kubernetes_pod_label_daemon', '__meta_kubernetes_pod_node_name']
    regex: 'node-exporter;(.*)'
    action: replace
    target_label: nodename
  - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_pod_name]
    regex: (.+);(.+)
    target_label: __metrics_path__
    replacement: /api/v1/namespaces/${1}/pods/${2}/proxy/metrics

###################################################################################
# Scrape config for nodes (kubelet).                                              #
#                                                                                 #
# Rather than connecting directly to the node, the scrape is proxied though the   #
# Kubernetes apiserver.  This means it will work if Prometheus is running out of  #
# cluster, or can't connect to nodes for some other reason (e.g. because of       #
# firewalling).                                                                   #
###################################################################################

- job_name: 'kubernetes-kubelet'
  scheme: https
  tls_config:
    insecure_skip_verify: true
  bearer_token_file: /etc/prometheus/token

  kubernetes_sd_configs:
  - role: node
    api_server: https://{{ kubernetes_api_server_addr }}
    tls_config:
      insecure_skip_verify: true
    bearer_token_file: /etc/prometheus/token
  relabel_configs:
  - action: labelmap
    regex: __meta_kubernetes_node_label_(.+)
  - target_label: __address__
    replacement: {{ kubernetes_api_server_addr }}
  - source_labels: [__meta_kubernetes_node_name]
    regex: (.+)
    target_label: __metrics_path__
    replacement: /api/v1/nodes/${1}/proxy/metrics

- job_name: 'kubernetes-cadvisor'
  scheme: https
  tls_config:
    insecure_skip_verify: true
  bearer_token_file: /etc/prometheus/token
  kubernetes_sd_configs:
  - role: node
    api_server: https://{{ kubernetes_api_server_addr }}
    tls_config:
      insecure_skip_verify: true
    bearer_token_file: /etc/prometheus/token
  relabel_configs:
  - action: labelmap
    regex: __meta_kubernetes_node_label_(.+)
  - target_label: __address__
    replacement: {{ kubernetes_api_server_addr }}
  - source_labels: [__meta_kubernetes_node_name]
    regex: (.+)
    target_label: __metrics_path__
    replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor


###################################################################################
# Example scrape config for service endpoints.                                    #
#                                                                                 #
# The relabeling allows the actual service scrape endpoint to be configured       #
# for all or only some endpoints.                                                 #
###################################################################################

- job_name: 'kubernetes-service-endpoints'

  kubernetes_sd_configs:
  - role: endpoints
    api_server: https://{{ kubernetes_api_server_addr }}
    tls_config:
      insecure_skip_verify: true
    bearer_token_file: /etc/prometheus/token

  relabel_configs:
  - action: labelmap
    regex: __meta_kubernetes_service_label_(.+)
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_service_name]
    action: replace
    target_label: kubernetes_name

#########################################################################################
# Example scrape config for probing services via the Blackbox Exporter.                 #
#                                                                                       #
# The relabeling allows the actual service scrape endpoint to be configured             #
# for all or only some services.                                                        #
#########################################################################################

- job_name: 'kubernetes-services'
  kubernetes_sd_configs:
  - role: service
    api_server: https://{{ kubernetes_api_server_addr }}
    tls_config:
      insecure_skip_verify: true
    bearer_token_file: /etc/prometheus/token
  scheme: https
  tls_config:
      insecure_skip_verify: true
  bearer_token_file: /etc/prometheus/token 
  relabel_configs:
  - action: labelmap
    regex: __meta_kubernetes_service_label_(.+)
  - target_label: __address__
    replacement: {{ kubernetes_api_server_addr }}
  - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name]
    regex: (.+);(.+)
    target_label: __metrics_path__
    replacement: /api/v1/namespaces/$1/services/$2/proxy/metrics

##################################################################################
# Example scrape config for pods                                                 #
#                                                                                #
# The relabeling allows the actual pod scrape to be configured                   #
# for all the declared ports (or port-free target if none is declared)           #
# or only some ports.                                                            #
##################################################################################

- job_name: 'kubernetes-pods'

  kubernetes_sd_configs:
  - role: pod
    api_server: https://{{ kubernetes_api_server_addr }}
    tls_config:
      insecure_skip_verify: true
    bearer_token_file: /etc/prometheus/token
  relabel_configs:
  - source_labels: [__address__, __meta_kubernetes_pod_annotation_example_io_scrape_port]
    action: replace
    regex: ([^:]+)(?::\d+)?;(\d+)
    replacement: $1:$2
    target_label: __address__
  - action: labelmap
    regex: __meta_kubernetes_pod_label_(.+)
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_pod_name]
    action: replace
    target_label: kubernetes_pods 
- job_name: 'kubernetes-service-endpoints-e'
  kubernetes_sd_configs:
  - role: endpoints
    api_server: https://{{ kubernetes_api_server_addr }}
    tls_config:
      insecure_skip_verify: true
    bearer_token_file: /etc/prometheus/token
  scheme: https
  tls_config:
    insecure_skip_verify: true
  bearer_token_file: /etc/prometheus/token
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
    action: keep
    regex: true
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port]
    action: replace
    regex: (\d+)
    target_label: __meta_kubernetes_pod_container_port_number
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
    action: replace
    regex: ()
    target_label: __meta_kubernetes_service_annotation_prometheus_io_path
    replacement: /metrics
  - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_pod_container_port_number, __meta_kubernetes_service_annotation_prometheus_io_path]
    target_label: __metrics_path__
    regex: (.+);(.+);(.+);(.+)
    replacement: /api/v1/namespaces/$1/services/$2:$3/proxy$4
  - target_label: __address__
    replacement: {{ kubernetes_api_server_addr }}
  - action: labelmap
    regex: __meta_kubernetes_service_label_(.+)
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_service_name]
    action: replace
    target_label: kubernetes_name
  - source_labels: [__meta_kubernetes_pod_node_name]
    action: replace
    target_label: instance
</code></pre>
<p>This is the Prometheus.yml file that I have on my prometheus instance.</p>
<p><strong>Prometheus instance logs /var/log/messages</strong></p>
<pre><code>Jul  1 15:18:53 ip-XXXXXXXXXXX prometheus: ts=2021-07-01T15:18:53.655Z caller=log.go:124 component=k8s_client_runtime level=debug func=Verbose.Infof msg=&quot;Listing and watching *v1.Endpoints from pkg/mod/k8s.io/client-go@v0.21.1/tools/cache/reflector.go:167&quot;
Jul  1 15:18:53 ip-XXXXXXXXXXX prometheus: ts=2021-07-01T15:18:53.676Z caller=log.go:124 component=k8s_client_runtime level=debug func=Infof msg=&quot;GET https://XXXXXXXXXXXXXXXXXXXXXXX.eks.amazonaws.com/api/v1/endpoints?limit=500&amp;resourceVersion=0  in 20 milliseconds&quot;
Jul  1 15:18:53 ip-XXXXXXXXXXX prometheus: ts=2021-07-01T15:18:53.676Z caller=log.go:124 component=k8s_client_runtime level=error func=ErrorDepth msg=&quot;pkg/mod/k8s.io/client-go@v0.21.1/tools/cache/reflector.go:167: Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: Get \&quot;https://XXXXXXXXXXXXXXXXXXXXXXX.eks.amazonaws.com/api/v1/endpoints?limit=500&amp;resourceVersion=0\&quot;: x509: certificate signed by unknown authority&quot;
Jul  1 15:18:56 ip-XXXXXXXXXXX prometheus: ts=2021-07-01T15:18:56.445Z caller=log.go:124 component=k8s_client_runtime level=debug func=Verbose.Infof msg=&quot;Listing and watching *v1.Pod from pkg/mod/k8s.io/client-go@v0.21.1/tools/cache/reflector.go:167&quot;
Jul  1 15:18:56 ip-XXXXXXXXXXX prometheus: ts=2021-07-01T15:18:56.445Z caller=log.go:124 component=k8s_client_runtime level=debug func=Infof msg=&quot;GET https://XXXXXXXXXXXXXXXXXXXXXXX.eks.amazonaws.com/api/v1/pods?limit=500&amp;resourceVersion=0  in 0 milliseconds&quot;
Jul  1 15:18:56 ip-XXXXXXXXXXX prometheus: ts=2021-07-01T15:18:56.445Z caller=log.go:124 component=k8s_client_runtime level=error func=ErrorDepth msg=&quot;pkg/mod/k8s.io/client-go@v0.21.1/tools/cache/reflector.go:167: Failed to watch *v1.Pod: failed to list *v1.Pod: Get \&quot;https://XXXXXXXXXXXXXXXXXXXXXXX.eks.amazonaws.com/api/v1/pods?limit=500&amp;resourceVersion=0\&quot;: unable to read authorization credentials file /etc/prometheus/token: open /etc/prometheus/token: no such file or directory&quot;
</code></pre>

meghashukla
<p>The logs you shared point to the problem:</p>
<pre><code>... unable to read authorization credentials file /etc/prometheus/token: open /etc/prometheus/token: no such file or directory&quot;
</code></pre>
<p>The token file for in-cluster workloads is by default mounted at <code>/var/run/secrets/kubernetes.io/serviceaccount/token</code> but since you mentioned Prometheus is running &quot;on external node&quot; (no idea what you mean by this) this may or may not be useful for you to be possible to change.</p>
