<p>I have an application running in Kubernetes (Azure AKS) in which each pod contains two containers. I also have Grafana set up to display various metrics some of which are coming from Prometheus. I'm trying to troubleshoot a separate issue and in doing so I've noticed that some metrics don't appear to match up between data sources.</p>

<p>For example, <code>kube_deployment_status_replicas_available</code> returns the value 30 whereas <code>kubectl -n XXXXXXXX get pod</code> lists 100 all of which are Running, and <code>kube_deployment_status_replicas_unavailable</code> returns a value of 0. Also, if I get the deployment in question using <code>kubectl</code> I'm seeing the expected value.</p>

<pre><code>$ kubectl get deployment XXXXXXXX
NAME       DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
XXXXXXXX   100       100       100          100         49d
</code></pre>

<p>There are other applications (namespaces) in the same cluster where all the values correlate correctly so I'm not sure where the fault may be or if there's any way to know for sure which value is the correct one. Any guidance would be appreciated. Thanks</p>

<p>Based on having the <code>kube_deployment_status_replicas_available</code> metric I assume that you have Prometheus scraping your metrics from <a href="https://github.com/kubernetes/kube-state-metrics" rel="nofollow noreferrer">kube-state-metrics</a>. It sounds like there's something quirky about its deployment. It could be:</p>

<ul>
<li>Cached metric data</li>
<li>And/or simply it can't pull current metrics from the <a href="https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/" rel="nofollow noreferrer">kube-apiserver</a></li>
</ul>

<p>I would:</p>

<ul>
<li>Check the version that you are running for <a href="https://github.com/kubernetes/kube-state-metrics" rel="nofollow noreferrer">kube-state-metrics</a> and see if it's compatible with your K8s version.</li>
<li>Restart the kube-state-metrics pod.</li>
<li>Check the logs <code>kubectl logs</code>kube-state-metrics` </li>
<li>Check the Prometheus logs

<ul>
<li>If you don't see anything try starting Prometheus with the <code>--log.level=debug</code> flag.</li>
</ul></li>
</ul>

<p>Hope it helps.</p>
