<p><strong>Background:</strong></p>
<p>I have a bare metal Kubernetes cluster. The master node is diskful, however all of the worker nodes are diskless, and PXE boot off of a machine which provides their OS image and non-volatile NFS mount points.</p>
<p>We found early on that we needed some kind of non-volatile storage so that the diskless nodes could store their configurations, kubernetes secrets, SSL keys, etc... Without the non-volatile storage, the nodes would have to be deleted and rejoined to the cluster every time they rebooted, which is obviously a pain.</p>
<p>So we created NFS mount partitions for /etc/kubernetes and /var/lib/kubelet so that the nodes could remember who they were and rejoin the cluster after a reboot.</p>
<p><strong>The Problem:</strong></p>
<p>However, we have run into a problem with kubeadm: It unmounts/remounts /var/lib/kubelet whenever you run kubeadm join. This leads to kubelet puking and dying because it can't find the appropriate device. The specific error is:</p>
<pre><code>node6 kubelet: F0113 18:08:55.196706  210949 kubelet.go:1347] Failed to start ContainerManager failed to get rootfs info: failed to get device for dir &quot;/var/lib/kubelet&quot;: could not find dev
ice with major: 0, minor: 46 in cached partitions map
</code></pre>
<p>If we unmount the NFS mount for /var/lib/kubelet before running kubeadm join, it works just fine, but gets us back to our original problem: the diskless node will not survive a reboot and automatically rejoin the cluster.</p>
<p>We tried to get around this by changing kubelet's k8s root, but it still leads to the same crash where kubelet gets confused about the device.</p>
<p>So my question is this:</p>
<p><strong>How can I properly set up an NFS mount point for /var/lib/kubelet, or otherwise configure kubelet, on my diskless worker nodes so that even when kubeadm mounts/unmounts that point I don't get a crash, and kubeadm's setup files are non-volatile?</strong></p>

<p>This is a known issue but fortunately this is fixed just couple of days back - <a href="https://github.com/google/cadvisor/pull/2787" rel="nofollow noreferrer">https://github.com/google/cadvisor/pull/2787</a>.</p>
<p>This hasn't been cut into a release yet. While this might make into 1.20.x, not really sure about previous versions. Might want to request for a backport on the issue at <a href="https://github.com/kubernetes/kubernetes/issues/98009" rel="nofollow noreferrer">https://github.com/kubernetes/kubernetes/issues/98009</a> if required.</p>
