<p>I have configured Prometheus on one of the kubernetes cluster nodes using [this][1].  After that I added following <code>prometheus.yml</code> file. I can list nodes and apiservers but for pods, all the pods shows down and error:</p>
<pre><code>Get &quot;https:// xx.xx.xx:443 /metrics&quot;: dial tcp xx.xx.xx:443: connect: connection refused and for some pods the status is unknown. 
</code></pre>
<p>Can someone point me what am I doing wrong here?</p>
<pre><code>Cat prometheus.yml



global:
  scrape_interval: 1m
 
scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: \['localhost:9090'\]
 
# metrics for default/kubernetes api's from the kubernetes master
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
    - role: pod
      bearer_token_file: /dfgdjk/token
      api_server: https://masterapi.com:3343
      tls_config:
        insecure_skip_verify: true
    tls_config:
      insecure_skip_verify: true
    bearer_token_file: /dfgdjk/token
    scheme: https
    relabel_configs:
    - action: labelmap
      regex: __meta_kubernetes_pod_label_(.+)
    - source_labels: \[__meta_kubernetes_namespace\]
      action: replace
      target_label: kubernetes_namespace
    - source_labels: \[__meta_kubernetes_pod_name\]
      action: replace
      target_label: kubernetes_pod_name
 
# metrics for default/kubernetes api's from the kubernetes master
  - job_name: 'kubernetes-apiservers'
    kubernetes_sd_configs:
    - role: endpoints
      api_server: https://masterapi.com:3343
      bearer_token_file: /dfgdjk/token
      tls_config:
        insecure_skip_verify: true
    tls_config:
      insecure_skip_verify: true
    bearer_token_file: /dfgdjk/token
    scheme: https
    relabel_configs:
    - source_labels: \[__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name\]
      action: keep
      regex: default;kubernetes;https][1]
 


  [1]: https://devopscube.com/install-configure-prometheus-linux/
</code></pre>

<p>It's impossible to get metrics to external prometheus server without having any prometheus components inside the kubernetes cluster. This happens because cluster network is isolated with host's network and it's not possible to scrape metrics from pods directly from outside the cluster.</p>
<p>Please refer to <a href="https://github.com/prometheus/prometheus/issues/4633" rel="nofollow noreferrer">Monitoring kubernetes with prometheus from outside of k8s cluster GitHub issue</a></p>
<p>There options which can be done:</p>
<ul>
<li>install prometheus inside the cluster using <code>prometheus operator</code> or manually - <a href="https://sysdig.com/blog/kubernetes-monitoring-prometheus/#install" rel="nofollow noreferrer">example</a></li>
<li>use proxy solutions, for example this one from the same thread on GitHub - <a href="https://github.com/americanexpress/k8s-prometheus-proxy" rel="nofollow noreferrer">k8s-prometheus-proxy</a></li>
<li>on top of the prometheus installed within the cluster, it's possible to have external prometheus in <code>federation</code> so all logs are saved outside of the cluster. Please refer to <a href="https://prometheus.io/docs/prometheus/latest/federation/" rel="nofollow noreferrer">prometheus federation</a>.</li>
</ul>
<p>Also important part is <a href="https://github.com/kubernetes/kube-state-metrics" rel="nofollow noreferrer">kube state metrics</a> should be installed as well in kubernetes cluster. <a href="https://devopscube.com/setup-kube-state-metrics/" rel="nofollow noreferrer">How to set it up</a>.</p>
<p><strong>Edit:</strong> also you can refer to <a href="https://stackoverflow.com/questions/41845307">another SO question/answer</a> which confirms that only with additional steps or OP resolved it by another proxy solution.</p>
