title: Kubernetes eviction manager evicting control plane pods to reclaim ephemeral storage
body: <p>I am using Kubernetes v1.13.0. My master is also functioning as a worker-node, so it has workload pods running on it, apart from control plane pods.</p>
  
  <p>The kubelet logs on my master show the following lines:</p>
  
  <pre>
  eviction_manager.go:340] eviction manager: must evict pod(s) to reclaim ephemeral-storage
  eviction_manager.go:358] eviction manager: pods ranked for eviction: kube-controller-manager-vm2_kube-system(1631c2c238e0c5117acac446b26d9f8c), kube-apiserver-vm2_kube-system(ce43eba098d219e13901c4a0b829f43b), etcd-vm2_kube-system(91ab2b0ddf4484a5ac6ee9661dbd0b1c)
  </pre>
  
  <p>Once the kube-apiserver pod is evicted, the cluster becomes unusable.</p>
  
  <p>What can I do to fix this? Should I add more ephemeral storage? How would I go about doing that? That means adding more space to the root partition on my host?</p>
  
  <p>My understanding is that ephemeral storage consists of <code>/var/log</code> and <code>/var/lib/kubelet</code> folders, which both come under the root partition.</p>
  
  <p>A <code>df -h</code> on my host shows:</p>
  
  <pre>
  Filesystem                               Size  Used Avail Use% Mounted on
  /dev/vda1                                 39G   33G  6.2G  85% /
  </pre>
  
  <p>So it looks like the root partition has lot of memory left, and there is no disk pressure. So what is causing this issue? Some of my worker pods must be doing something crazy with storage, but it's still 6G seems like plenty of room.</p>
  
  <p>Will adding more space to the root partition fix this issue temporarily?</p>
  
  <p><code>kubectl describe vm2</code> gives the following info:</p>
  
  <pre>
  Conditions:
    Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
    ----             ------  -----------------                 ------------------                ------                       -------
    MemoryPressure   False   Fri, 11 Jan 2019 21:25:43 +0000   Wed, 05 Dec 2018 19:16:41 +0000   KubeletHasSufficientMemory   kubelet has sufficient memory available
    DiskPressure     False   Fri, 11 Jan 2019 21:25:43 +0000   Fri, 11 Jan 2019 20:58:07 +0000   KubeletHasNoDiskPressure     kubelet has no disk pressure
    PIDPressure      False   Fri, 11 Jan 2019 21:25:43 +0000   Wed, 05 Dec 2018 19:16:41 +0000   KubeletHasSufficientPID      kubelet has sufficient PID available
    Ready            True    Fri, 11 Jan 2019 21:25:43 +0000   Thu, 06 Dec 2018 17:00:02 +0000   KubeletReady                 kubelet is posting ready status. AppArmor enabled
  Capacity:
   cpu:                8
   ephemeral-storage:  40593708Ki
   hugepages-1Gi:      0
   hugepages-2Mi:      0
   memory:             32946816Ki
   pods:               110
  Allocatable:
   cpu:                8
   ephemeral-storage:  37411161231
   hugepages-1Gi:      0
   hugepages-2Mi:      0
   memory:             32844416Ki
   pods:               110
  </pre>
  
  <p>It seems to me that there was pressure on ephemeral-storage, and the eviction manager is trying to reclaim some storage by evicting least recently used pods. But it should not evict the control plane pods, otherwise cluster is unusable.</p>
  
  <p>Currently, the Kubelet evicts the control plane pods. Then I try to manually start the apiserver and other control plane pods by adding and removing a space in the <code>/etc/kubernetes/manifests</code> files. This does start the apiserver, but then it again gets evicted. Ideally, the Kubelet should ensure that the static pods in <code>/etc/kubernetes/manifests</code> are always on and properly managed.</p>
  
  <p>I am trying to understand what is going on here, and how to fix this issue, so that my kubernetes cluster becomes more robust, and I don't have to keep manually restarting the apiserver.</p>