<p>After a few days of running <code>dockerd</code> on a kubernetes host, where pods are scheduled by kubelet, <code>dockerd</code> goes bad - consuming a lot of resources (50% memory - ~4gigs). </p>

<p>When it gets to this state, it is unable to act on commands for containers that appear to be running via <code>$ docker ps</code>. Also checking <code>ps -ef</code> on the host these containers don't map to any underlying host processes.</p>

<p><code>$ docker exec</code> yields - </p>

<pre><code>level=error msg="Error running exec in container: rpc error: code = 2 desc = containerd: container not found"

Cannot kill container 6a8d4....8: rpc error: code = 14 desc = grpc: the connection is unavailable"

level=fatal msg="open /var/run/docker/libcontainerd/containerd/7657...4/65...6/process.json: no such file or directory"
</code></pre>

<p>Looking through the process tree on the host there seem to be a lot of defunct processes which point to <code>dockerd</code> as the parent id. Any pointers on what the issue might be or where to look further?</p>

<p>Enabled debug on <code>dockerd</code> to see if the issue re-occurs, a dockerd restart fixes the issue.</p>

user2062360
<p>Sounds like you have a container misbehaving and docker is not able to reap it. I would take a look at what has been scheduled on the nodes where you see the problem. The error you are seeing seems like the docker daemon not responding to API requests issued by the docker CLI. Some pointers:</p>

<ul>
<li>Has the container exited successfully or with an error?</li>
<li>Did they containers get killed for some reason?</li>
<li>Check the kubelet logs</li>
<li>Check the kube-scheduler logs?</li>
<li>Follow the logs in the containers on your node <code>docker logs -f &lt;containerid&gt;</code></li>
</ul>
