<p>I run <a href="https://prometheus.io/" rel="nofollow noreferrer">prometheus</a> locally as http://localhost:9090/targets with</p>
<pre><code>docker run --name prometheus -d -p 127.0.0.1:9090:9090 prom/prometheus
</code></pre>
<p>and want to connect it to several Kubernetes (cluster) instances we have.
See that scraping works, try <a href="https://grafana.com/grafana/dashboards" rel="nofollow noreferrer">Grafana dashboards</a> etc.</p>
<p>And then I'll do the same on dedicated server that will be specially for monitoring.
However all googling gives me all different ways to configure prometheus that is already within one Kubernetes instance, and no way to read metrics from external Kubernetes.</p>
<p><strong>How to add Kubernetes scrape target to Prometheus instance that is NOT in Kubernetes?</strong></p>
<hr />
<p>I have read <a href="https://www.datadoghq.com/blog/monitoring-kubernetes-performance-metrics/#where-kubernetes-metrics-come-from" rel="nofollow noreferrer">Where Kubernetes metrics come from</a> and checked that my (first) Kubernetes cluster has the <strong>Metrics Server</strong>.</p>
<pre><code>kubectl get pods --all-namespaces | grep metrics-server 
</code></pre>
<p>There is definitely no sense to add Prometheus instance into every Kubernetes (cluster) instance. One Prometheus must be able to read metrics from many Kubernetes clusters and every node within them.</p>
<p>P.S. Some <a href="https://stackoverflow.com/questions/59132281/monitor-kubernetes-cluster-from-other-kubernetes-cluster-with-prometheus">old question</a> has answer to install Prometheus in every Kubernetes and then use federation, that is just opposite from what I am looking for.</p>
<p>P.P.S. It is also strange for me, why Kubernetes and Prometheus that are #1 and #2 projects from Cloud Native Foundation don't have simple &quot;add Kubernetes target in Prometheus&quot; button or simple step.</p>

Paul Verest
<p>If I understand your question, you want to monitor kubernetes cluster where prometheus is not installed on remote kubernetes cluster.</p>
<blockquote>
<p>I monitor many different kubernetes cluster from one prometheus which
is installed on a standalone server.</p>
</blockquote>
<p><em><strong>You can do this by generating a token on the kubernetes server using a service account which has proper permission to access the kubernetes api.</strong></em></p>
<p><strong>Kubernetes-api:</strong></p>
<p>Following are the details required to configure prometheus scrape job.</p>
<ol>
<li>Create a service account which has permissions to read and watch the
pods.</li>
<li>Generate token from the service account.</li>
<li>Create scrape job as following.</li>
</ol>
<pre><code>- job_name: kubernetes
  kubernetes_sd_configs:
  - role: node
    api_server: https://kubernetes-cluster-api.com
    tls_config:
      insecure_skip_verify: true
      bearer_token: &quot;xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx&quot;
  bearer_token: &quot;xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx&quot;
  scheme: https
  tls_config:
    insecure_skip_verify: true
  relabel_configs:
  - separator: ;
    regex: __meta_kubernetes_node_label_(.+)
    replacement: $1
    action: labelmap
</code></pre>
<p>I have explained the same in detail in the article<br />
&quot;Monitor remote kubernetes cluster using prometheus&quot;.
<a href="https://amjadhussain3751.medium.com/monitor-remote-kubernetes-cluster-using-prometheus-a3781b041745" rel="nofollow noreferrer">https://amjadhussain3751.medium.com/monitor-remote-kubernetes-cluster-using-prometheus-a3781b041745</a></p>
