<p>I have 3 kubernetes clusters (prod, test, monitoring). Iam new to prometheus so i have tested it by installing it in my test environment with the helm chart:</p>
<pre><code># https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack
helm install [RELEASE_NAME] prometheus-community/kube-prometheus-stack
</code></pre>
<p>But if i want to have metrics from the prod and test clusters, i have to repeat the same installation of the helm and each &quot;kube-prometheus-stack&quot; would be standalone in its own cluster. It is not ideal at all. Iam trying to find a way to have a single prometheus/grafana which would federate/agregate the metrics from each cluster's prometheus server.<br/></p>
<p>I found this link, saying about prometheus federation:</p>
<pre><code>https://prometheus.io/docs/prometheus/latest/federation/
</code></pre>
<p>If install the helm chart &quot;kube-prometheus-stack&quot; and get rid of grafana on the 2 other cluster, how can i make the 3rd &quot;kube-prometheus-stack&quot;, on the 3rd cluster, scrapes metrics from the 2 other ones?<br/>
thanks</p>

YoussHark
<p>You have to modify configuration for prometheus federate so it can scrape metrics from other clusters as described <a href="https://prometheus.io/docs/prometheus/latest/federation/#configuring-federation" rel="noreferrer">in documentation</a>:</p>
<pre><code>scrape_configs:
  - job_name: 'federate'
    scrape_interval: 15s

    honor_labels: true
    metrics_path: '/federate'

    params:
      'match[]':
        - '{job=&quot;prometheus&quot;}'
        - '{__name__=~&quot;job:.*&quot;}'

    static_configs:
      - targets:
        - 'source-prometheus-1:9090'
        - 'source-prometheus-2:9090'
        - 'source-prometheus-3:9090'
</code></pre>
<p><code>params</code> field checks for <a href="https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml#L248" rel="noreferrer">jobs to scrape</a> metrics from. In this particular example</p>
<blockquote>
<p>It will scrape any series with the label job=&quot;prometheus&quot; or a metric name starting
with job: from the Prometheus servers at
source-prometheus-{1,2,3}:9090</p>
</blockquote>
<p>You can check following articles to give you more insight of prometheus federation:</p>
<ol>
<li><p><a href="https://www.linkedin.com/pulse/monitoring-kubernetes-prometheus-outside-cluster-steven-acreman" rel="noreferrer">Monitoring Kubernetes with Prometheus - outside the cluster!</a></p>
</li>
<li><p><a href="https://medium.com/@jotak/prometheus-federation-in-kubernetes-4ce46bda834e" rel="noreferrer">Prometheus federation in Kubernetes</a></p>
</li>
<li><p><a href="https://banzaicloud.com/blog/prometheus-federation/" rel="noreferrer">Monitoring multiple federated clusters with Prometheus - the secure way</a></p>
</li>
<li><p><a href="https://developers.mattermost.com/blog/cloud-monitoring/" rel="noreferrer">Monitoring a Multi-Cluster Environment Using Prometheus Federation and Grafana</a></p>
</li>
</ol>
