<p><strong>Problem</strong>:</p>

<p>Duplicate data when querying from prometheus for metrics from <em>kube-state-metrics</em>.</p>

<p>Sample query and result with 3 instances of <em>kube-state-metrics</em> running:</p>

<p>Query:</p>

<pre><code>kube_pod_container_resource_requests_cpu_cores{namespace="ns-dummy"}
</code></pre>

<p>Metrics</p>

<pre><code>kube_pod_container_resource_requests_cpu_cores{container="appname",endpoint="http",instance="172.232.35.142:8080",job="kube-state-metrics",namespace="ns-dummy",node="ip-172-232-34-25.ec2.internal",pod="app1-appname-6bd9d8d978-gfk7f",service="prom-kube-state-metrics"}
1
kube_pod_container_resource_requests_cpu_cores{container="appname",endpoint="http",instance="172.232.35.142:8080",job="kube-state-metrics",namespace="ns-dummy",node="ip-172-232-35-22.ec2.internal",pod="app2-appname-ccbdfc7c8-g9x6s",service="prom-kube-state-metrics"}
1
kube_pod_container_resource_requests_cpu_cores{container="appname",endpoint="http",instance="172.232.35.17:8080",job="kube-state-metrics",namespace="ns-dummy",node="ip-172-232-34-25.ec2.internal",pod="app1-appname-6bd9d8d978-gfk7f",service="prom-kube-state-metrics"}
1
kube_pod_container_resource_requests_cpu_cores{container="appname",endpoint="http",instance="172.232.35.17:8080",job="kube-state-metrics",namespace="ns-dummy",node="ip-172-232-35-22.ec2.internal",pod="app2-appname-ccbdfc7c8-g9x6s",service="prom-kube-state-metrics"}
1
kube_pod_container_resource_requests_cpu_cores{container="appname",endpoint="http",instance="172.232.37.171:8080",job="kube-state-metrics",namespace="ns-dummy",node="ip-172-232-34-25.ec2.internal",pod="app1-appname-6bd9d8d978-gfk7f",service="prom-kube-state-metrics"}
1
kube_pod_container_resource_requests_cpu_cores{container="appname",endpoint="http",instance="172.232.37.171:8080",job="kube-state-metrics",namespace="ns-dummy",node="ip-172-232-35-22.ec2.internal",pod="app2-appname-ccbdfc7c8-g9x6s",service="prom-kube-state-metrics"}
</code></pre>

<p><strong>Observation</strong>:</p>

<p>Every metric is coming up Nx when N pods are running for <em>kube-state-metrics</em>. If it's a single pod running, we get the correct info.</p>

<p><strong>Possible solutions</strong>:</p>

<ol>
<li>Scale down to single instance of kube-state-metrics. (Reduced availability is a concern)</li>
<li>Enable sharding. (Solves duplication problem, still less available)</li>
</ol>

<p>According to the <a href="https://github.com/kubernetes/kube-state-metrics#horizontal-scaling-sharding" rel="nofollow noreferrer">docs</a>, for horizontal scaling we have to pass sharding arguments to the pods.</p>

<p>Shards are zero indexed. So we have to pass the index and total number of shards for each pod.</p>

<p>We are using <a href="https://github.com/helm/charts/tree/master/stable/kube-state-metrics" rel="nofollow noreferrer">Helm chart</a> and it is deployed as a deployment. </p>

<p><strong>Questions</strong>:</p>

<ol>
<li>How can we pass different arguments to different pods in this scenario, if its possible?</li>
<li>Should we be worried about availability of the <em>kube-state-metrics</em> considering the self-healing nature of k8s workloads? </li>
<li>When should we really scale it to multiple instances and how?</li>
</ol>

<p>You could use a 'self-healing' deployment with only a single replica of <code>kube-state-metric</code> if the container down, the deployment will start a new container. Since kube-state-metric <a href="https://github.com/kubernetes/kube-state-metrics" rel="nofollow noreferrer">is not focused on the health of the individual kubernetes components</a>. It only will affect you if your cluster is too big and make many objects changes per second.</p>

<blockquote>
  <p>It is not focused on the health of the individual Kubernetes components, but rather on the health of the various objects inside, such as deployments, nodes and pods.</p>
</blockquote>

<p>For small cluster there's is no problem use in this way, but you really need a high availability monitoring platform I recommend you take a look in this two articles:
<a href="https://medium.com/nephely/creating-a-well-designed-and-highly-available-monitoring-stack-for-servers-kubernetes-cluster-and-47e810ec55af" rel="nofollow noreferrer">creating a well designed and highly available monitoring stack for kubernetes</a> and 
<a href="https://logz.io/blog/kubernetes-monitoring/" rel="nofollow noreferrer">kubernetes monitoring</a></p>
