<p>is it possible to setup a prometheus/grafana running on centos to monitor several K8S clusters in the lab? the architecture can be similar to the bellow one, although not strictly required. Right now the kubernetes clusters we have, do not have prometheus and grafana installed. The documentation is not very much clear if an additional component/agent remote-push is required or not and how the central prometheus and the K8S need to be configured to achieve the results?
Thanks.</p>
<p><img src="https://i.stack.imgur.com/bCWTX.png" alt="Required architecture" /></p>

<p>You have different solution in order to implement your use case :</p>
<ol>
<li>You can use <a href="https://prometheus.io/docs/prometheus/latest/federation/" rel="nofollow noreferrer">prometheus federation</a>. This will allow you to have a central prometheus server that will scrape samples from other prometheus servers.</li>
<li>You can use <a href="https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write" rel="nofollow noreferrer">remote_write configuration</a>. This will allow you to send your samples to a remote endpoint (and then eventually scrape that central endpoint). You'll also be able to apply relabeling rules with this configuration.</li>
<li>As @JulioHM said in the comment, you can use another tool like <a href="https://thanos.io/" rel="nofollow noreferrer">thanos</a> or <a href="https://github.com/cortexproject/cortex" rel="nofollow noreferrer">Cortex</a>. Those tools are great and allow you to do more stuff than just writing to a remote endpoint. You'll be able to implement horizontal scalling of your prometheus servers, long-term storage, etc.</li>
</ol>
