<p>I moved from gvisor-containerd-shim (Shim V1) to containerd-shim-runsc-v1 (Shim V2). The metrics server and the Horizontal Pod Autoscaler used to work just fine in the case of gvisor-containerd-shim.</p>
<p>But now, with containerd-shim-runsc-v1, I keep getting CPU and memory metrics for nodes and runc pods, but I only get memory metrics for runsc (gvisor) pods.</p>
<p>For example, I deployed a PHP server in a gvisor pod with containerd-shim-runsc-v1. I get the following metrics:</p>
<pre class="lang-sh prettyprint-override"><code>kubectl get hpa
NAME         REFERENCE               TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
php-apache   Deployment/php-apache   0%/50%    1         10        1          68s


kubectl top nodes
NAME         CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
snf-877559   549m         13%    2327Mi          39%


kubectl top pods
NAME                                 CPU(cores)   MEMORY(bytes)
php-apache-gvisor-6f7bb6cf84-28qdk   0m           52Mi
</code></pre>
<p>After sending some load to the php-apache-gvisor pod, I can see CPU and memory usage increment for the node and for the runc pod (load-generator). <strong>I can also see that php-apache-gvisor's memory is increased from 52 to 72 Mi but its CPU usage remains at 0%.</strong> Why does the cpu usage remain at 0%?</p>
<p>I also tried with different container images, but I keep getting same results.</p>
<p>With load I get the following metrics:</p>
<pre class="lang-sh prettyprint-override"><code>kubectl get hpa
NAME         REFERENCE               TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
php-apache   Deployment/php-apache   0%/50%    1         10        1          68s


kubectl top nodes
NAME         CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
snf-877559   946m         23%    2413Mi          41%


kubectl top pods
NAME                             CPU(cores)   MEMORY(bytes)
load-generator-7d549cd44-xmbqw   3m           1Mi
php-apache-gvisor-6f7bb6cf84-28qdk      0m           72Mi
</code></pre>
<p><strong>Further infos:</strong></p>
<p>kubeadm, kubernetes 1.15.3, containerd 1.3.3, runsc nightly/2019-09-18, flannel</p>
<pre class="lang-sh prettyprint-override"><code>kubectl logs metrics-server-74657b4dc4-8nlzn -n kube-system
I0728 09:33:42.449921       1 serving.go:312] Generated self-signed cert (/tmp/apiserver.crt, /tmp/apiserver.key)
I0728 09:33:44.153682       1 secure_serving.go:116] Serving securely on [::]:4443
E0728 09:35:24.579804       1 reststorage.go:160] unable to fetch pod metrics for pod default/php-apache-gvisor-6f7bb6cf84-28qdk: no metrics known for pod
E0728 09:35:39.940417       1 reststorage.go:160] unable to fetch pod metrics for pod default/php-apache-gvisor-6f7bb6cf84-28qdk: no metrics known for pod
</code></pre>
<p>/etc/containerd/config.toml (containerd-shim-runsc-v1)</p>
<pre><code>subreaper = true
oom_score = -999
disabled_plugins = [&quot;restart&quot;]


[debug]
    level = &quot;debug&quot;

[metrics]
    address = &quot;127.0.0.1:1338&quot;

[plugins.linux]
    runtime = &quot;runc&quot;
    shim_debug = true


[plugins.cri.containerd.runtimes.runsc]
  runtime_type = &quot;io.containerd.runsc.v1&quot;
</code></pre>
<p>/etc/containerd/config.toml (gvisor-containerd-shim)</p>
<pre><code>subreaper = true
oom_score = -999
disabled_plugins = [&quot;restart&quot;]


[debug]
    level = &quot;debug&quot;

[metrics]
    address = &quot;127.0.0.1:1338&quot;

[plugins.linux]
    runtime = &quot;runc&quot;
    shim_debug = true
    shim = &quot;/usr/local/bin/gvisor-containerd-shim&quot;


[plugins.cri.containerd.runtimes.runsc]
  runtime_type = &quot;io.containerd.runtime.v1.linux&quot;
  runtime_engine = &quot;/usr/local/bin/runsc&quot;
  runtime_root = &quot;/run/containerd/runsc&quot;
</code></pre>
<p>The metrics server yaml is based on <a href="https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.3.6/components.yaml" rel="nofollow noreferrer">https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.3.6/components.yaml</a> with the following args</p>
<pre class="lang-sh prettyprint-override"><code>....
      containers:
      - name: metrics-server
        image: k8s.gcr.io/metrics-server-amd64:v0.3.6
        imagePullPolicy: IfNotPresent
        args:
          - --kubelet-preferred-address-types=InternalIP
          - --kubelet-insecure-tls
          - --cert-dir=/tmp
          - --secure-port=4443
....
</code></pre>
<p>The current deployment has the below resources section</p>
<pre><code>  resources:
    limits:
      cpu: 500m
    requests:
      cpu: 200m
</code></pre>

<p>gVisor currently only reports memory and Pids on a per Pod basis.
See: <a href="https://github.com/google/gvisor/blob/add40fd/runsc/boot/events.go#L62-L68" rel="nofollow noreferrer">https://github.com/google/gvisor/blob/add40fd/runsc/boot/events.go#L62-L68</a></p>
<p>We are planning to export more stats and the issue for tracking that work is here:
<a href="https://gvisor.dev/issue/172" rel="nofollow noreferrer">https://gvisor.dev/issue/172</a></p>
