<p>I am newbie for using monitoring tools such as Prometheus in k8s..We have two separate cluster one for applications we deployed one for we only would like deploy monitoring,logging tools.</p>

<p>But I have some confusion how to handle this?</p>

<p>1.How cluster that serves prometheus can connect to application cluster and able to pull metrics?</p>

<p>2.How should I specify the namespace if I would like to set a network policy?</p>

<p>3.What should I do in application side for helm chart except exporting metrics?</p>

<pre><code># Allow traffic from pods with label app=prometheus in namespace with label name=monitoring
# to any pod in &lt;YOUR_APPLICATION_NAMESPACE&gt;
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: monitoring.prometheus.all
  namespace: &lt;YOUR_APPLICATION_NAMESPACE&gt;
spec:
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          name: monitoring
      podSelector:
        matchLabels:
          app: prometheus
  podSelector: {}
  policyTypes:
  - Ingress
</code></pre>

semural
<p>Isn't that what you want?</p>

<p>1) <a href="https://prometheus.io/docs/prometheus/latest/federation/" rel="nofollow noreferrer">Prometheus federation</a></p>

<blockquote>
  <p>Prometheus federation is a Prometheus server that can scrape data
  from other Prometheus servers. It supports hierarchical federation,
  which in our case resembles a tree.</p>
</blockquote>

<p>A default version of the Prometheus server is installed in each one of our clusters and a Prometheus federation server is deployed together with Grafana in a central monitoring cluster. Prometheus federation scrapes data from all the other Prometheus servers that run in our clusters. For future expansion, a central Prometheus federation can be used to scrape data from multiple Prometheus federation servers that scrape data from groups of tens of clusters.</p>

<p>More info here: <a href="https://developers.mattermost.com/blog/cloud-monitoring/" rel="nofollow noreferrer">https://developers.mattermost.com/blog/cloud-monitoring/</a></p>

<p>2) <a href="https://gist.github.com/sacreman/b61266d2ec52cf3a1af7c278d9d93450" rel="nofollow noreferrer">Prometheus configuration to scrape Kubernetes outside the cluster</a> yaml example</p>

<p>3) Linkedin <a href="https://www.linkedin.com/pulse/monitoring-kubernetes-prometheus-outside-cluster-steven-acreman" rel="nofollow noreferrer">Monitoring Kubernetes with Prometheus - outside the cluster!</a> article and Reddit  <a href="https://www.reddit.com/r/kubernetes/comments/c3gc39/monitoring_k8s_by_prometheus_outside_cluster/" rel="nofollow noreferrer">Monitoring K8s by Prometheus Outside Cluster</a> related discussion</p>
