<p>We have several K8S clusters which we need to monitor from <strong>one</strong> operator cluster (cluster A)
We are using Prometheus on each cluster to monitor the cluster itself, now in addition we want to monitor from a specific api of application which will tell us if our cluster (according to our specific services) is functinal or not, im not talking about monitor the cluster ,we want the the operator will monitor 3 application on each cluster( all the 3 applications are deployed on all the monitored clusters)</p>
<blockquote>
<p>Cluster A (operator) should monitor service/apps on cluster B,C,D etc</p>
</blockquote>
<p>e.g. The operator cluster will call to deplyed app in clusterA like  <code>host://app1/status</code> to get the status if 0 or 1, and save the status in some DB. (maybe prometehusDB) and report them outside the cluster.</p>
<p>Currently after some search I found this option but maybe there is more which I dont khow</p>
<ol>
<li><p>Use blackbox exporter - <a href="https://github.com/prometheus/blackbox_exporter" rel="nofollow noreferrer">https://github.com/prometheus/blackbox_exporter</a></p>
</li>
<li><p>Create my own programs (in golang) which will like a cronjob and which will be runing in the operator cluster using prom lib.</p>
</li>
</ol>
<p><a href="https://github.com/prometheus/client_golang" rel="nofollow noreferrer">https://github.com/prometheus/client_golang</a></p>
<p>I mean running a rest call and use Prometheus api to store the status inside Prometheus <code>tsdb</code> via go &quot;github.com/prometheus/client_golang/prometheus/promhttp&quot; code. but not sure how..</p>
<ol start="3">
<li>Federation ??</li>
</ol>
<p>In addition, in case I was able to collect all the data from the clusters into the operator cluster, How and where should I keep it? in Prometheus db tsdb? other way ?</p>
<p><strong>What should be the best practice to support our case ?</strong> How should we do it ?</p>

<p>I have seen that you though about using Thanos, its not bad and we had it running in production, for a while. But it didn't fit well for our requirements, yours looks familiar to ours so I suggest you to take a look at VictoriaMetrics
you have a nice article just here : <a href="https://medium.com/faun/comparing-thanos-to-victoriametrics-cluster-b193bea1683" rel="nofollow noreferrer">https://medium.com/faun/comparing-thanos-to-victoriametrics-cluster-b193bea1683</a></p>
<p>Also a big up is their support on Slack!
Good luck implementing it!</p>
