<p>I have reached moment when I need to split my prometheus into smaller ones.
I have been reading about it <a href="https://www.robustperception.io/scaling-and-federating-prometheus/" rel="nofollow noreferrer">here</a> but it does not say anything about scaling in kubernetes. Below is my setup:</p>

<ul>
<li>one node of prometheus</li>
<li>one node of <a href="https://github.com/kubernetes/kube-state-metrics" rel="nofollow noreferrer">kube state metrics</a></li>
<li><a href="https://github.com/prometheus/node_exporter" rel="nofollow noreferrer">node exporter</a> on each cluster node</li>
</ul>

<p>and there are about 50 namespaces which produces thousands of metrics and one current setup with one prometheus is not enough. So I decided to split it to three instances like:</p>

<ul>
<li>one for <a href="https://github.com/kubernetes/kube-state-metrics" rel="nofollow noreferrer">kube state metrics</a> metrics</li>
<li>one for <a href="https://github.com/prometheus/node_exporter" rel="nofollow noreferrer">node exporter</a> metrics</li>
<li>one for kubernetes metrics</li>
</ul>

<p>But after while i realised that those metrics are scraped by <a href="https://prometheus.io/docs/operating/configuration/#&lt;kubernetes_sd_config" rel="nofollow noreferrer">kubernetes_sd_config</a> and there is no way to tell which metrics I want to scrape by which instance of prometheus or I am wrong.
One solution would be to split kubernetes cluster into smaller one but it is too much work for now.</p>

<p>So my question is if there is any possibility to tell prometheus that I want scrape only <a href="https://github.com/kubernetes/kube-state-metrics" rel="nofollow noreferrer">kube state metrics</a>, <a href="https://github.com/prometheus/node_exporter" rel="nofollow noreferrer">node exporter</a> or native kubernetes metrics ?</p>

<p>Another option would be going for a horizontally scalable, distributed Prometheus implementation: <a href="https://github.com/weaveworks/cortex" rel="nofollow noreferrer">https://github.com/weaveworks/cortex</a> (NB I wrote this.)</p>

<p>Its not ready for prime time yet, but we're using it internally and getting pretty good results.  It will be more effort to setup and operate than upstream Prometheus, but it should scale virtually indefinitely - and what's more we run it on Kubernetes, so it's really at home there.</p>

<p>Let me know if you're interested and I can walk you though setting it up.</p>
