<p>I have a <a href="https://github.com/kubernetes-sigs/metrics-server" rel="nofollow noreferrer">metrics-server</a> and a horizontal pod autoscaler using this server, running on my cluster.<br />
This works perfectly fine, until i inject linkerd-proxies into the deployments of the namespace where my application is running. Running <code>kubectl top pod</code> in that namespace results in a <code>error: Metrics not available for pod &lt;name&gt;</code> error. However, nothing appears in the metrics-server pod's logs.<br />
The metrics-server clearly works fine in other namespaces, because top works in every namespace but the meshed one.</p>
<p>At first i thought it could be because the proxies' resource requests/limits weren't set, but after running the injection with them (<code>kubectl get -n &lt;namespace&gt; deploy -o yaml | linkerd inject - --proxy-cpu-request &quot;10m&quot; --proxy-cpu-limit &quot;1&quot; --proxy-memory-request &quot;64Mi&quot; --proxy-memory-limit &quot;256Mi&quot; | kubectl apply -f -</code>), the issue stays the same.</p>
<p>Is this a known problem, are there any possible solutions?</p>
<p>PS: I have a <a href="https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack" rel="nofollow noreferrer">kube-prometheus-stack</a> running in a different namespace, and this seems to be able to scrape the pod metrics from the meshed pods just fine <img src="https://i.imgur.com/bOogl3P.png" alt="grafana dashboard image showing prometheus can collect the data" /></p>

<p>The problem was apparently a bug in the cAdvisor stats provider with the CRI runtime. The linkerd-init containers keep producing metrics after they've terminated, which shouldn't happen. The metrics-server ignores stats from pods that contain containers that report zero values (to avoid reporting invalid metrics, like when a container is restarting, metrics aren't collected yet,...). You can follow up on the <a href="https://github.com/kubernetes/kubernetes/issues/103368" rel="nofollow noreferrer">issue</a> here. Solutions seem to be changing to another runtime or using the PodAndContainerStatsFromCRI flag, which will let the internal CRI stats provider be responsible instead of the cAdvisor one.</p>
