<p>I have run several Kubernetes Clusters on Azure AKS, so I intend to create <strong>Centralized Monitoring Cluster</strong> run on other Kubernetes cluster with Prometheus-Grafana.</p>

<p><a href="https://i.stack.imgur.com/74Txr.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/74Txr.png" alt="Centralized Monitoring"></a></p>

<p>My idea is that:</p>

<ul>
<li>Isolating &amp; centralizing monitoring cluster. </li>
<li>In case of any cluster is downed, Monitoring cluster is still alive to inspect downed cluster.</li>
<li>Run cross-cloud provider Kubernetes (if available)</li>
</ul>

<p>I'm confusing about connecting clusters, network, ingress, how does Prometheus discovery, pull metric from outside cluster... </p>

<p>Is there any best practice, instruction for my usecase. Thank you!</p>

<p>Yes, Prometheus is a very flexible monitoring solution wherein each Prometheus server is able to act as a target for another Prometheus server. Using prometheus federation, Prometheus servers can scrape selected time series data from other Prometheus servers.</p>

<pre><code>A typical Prometheus federation example configuration looks like this:

- job_name: 'federate'
  scrape_interval: 15s

  honor_labels: true
  metrics_path: '/federate'

  params:
    'match[]':
      - '{job="prometheus"}'
      - '{__name__=~"job:.*"}'

  static_configs:
    - targets:
      - 'source-prometheus-1:9090'
      - 'source-prometheus-2:9090'
      - 'source-prometheus-3:9090'
</code></pre>
