<p>We have several K8S clusters which we need to monitor from <strong>one</strong> operator cluster (cluster A)
We are using Prometheus on each cluster to monitor the cluster itself, now in addition we want to monitor from a specific api of application which will tell us if our cluster (according to our specific services) is functinal or not, im not talking about monitor the cluster ,we want the the operator will monitor 3 application on each cluster( all the 3 applications are deployed on all the monitored clusters)</p>
<blockquote>
<p>Cluster A (operator) should monitor service/apps on cluster B,C,D etc</p>
</blockquote>
<p>e.g. The operator cluster will call to deplyed app in clusterA like  <code>host://app1/status</code> to get the status if 0 or 1, and save the status in some DB. (maybe prometehusDB) and report them outside the cluster.</p>
<p>Currently after some search I found this option but maybe there is more which I dont khow</p>
<ol>
<li><p>Use blackbox exporter - <a href="https://github.com/prometheus/blackbox_exporter" rel="nofollow noreferrer">https://github.com/prometheus/blackbox_exporter</a></p>
</li>
<li><p>Create my own programs (in golang) which will like a cronjob and which will be runing in the operator cluster using prom lib.</p>
</li>
</ol>
<p><a href="https://github.com/prometheus/client_golang" rel="nofollow noreferrer">https://github.com/prometheus/client_golang</a></p>
<p>I mean running a rest call and use Prometheus api to store the status inside Prometheus <code>tsdb</code> via go &quot;github.com/prometheus/client_golang/prometheus/promhttp&quot; code. but not sure how..</p>
<ol start="3">
<li>Federation ??</li>
</ol>
<p>In addition, in case I was able to collect all the data from the clusters into the operator cluster, How and where should I keep it? in Prometheus db tsdb? other way ?</p>
<p><strong>What should be the best practice to support our case ?</strong> How should we do it ?</p>

<p>Ideally you would instrument your code and expose Prometheus compatible metrics for whatever needs monitored. But, there is something to be said for blackbox and/or 3rd party monitoring/smoke testing.</p>
<p>The http module in Blackbox Exporter is probably what you want (I've used it similarly before). If that isn't flexible enough for the testing you need to do, I like to run custom testing scripts in Lambda that record the results in Cloudwatch (if running in AWS, otherwise use the equivalent in your environment). If you haven't done that before, there is a bit of a learning curve, but it is well worth the effort.</p>
<p>If the APIs are externally accessible, services like Pingdom and Site24x7 offer flexible testing options (for a price), and it is generally recommended to utilize a 3rd party for at least basic up-time testing for the cases where your entire environment goes down--along with all of your monitoring!</p>
<p>But, it does sound like you just want to do some basic blackbox style monitoring which the Blackbox Exporter would be well suited to. It will need a host to run on, and then you'll need to add a job for it to Prometheus' scrape config. Best practice is to use each host for a single purpose, so I'd provision a specific host for the purpose of running blackbox exporter (even if it is just another container in the cluster).</p>
