<p>I have a live Kubernetes cluster, version 1.12, in which I need to change the default pod hard eviction values for every kubelet. I've read through <a href="https://kubernetes.io/docs/setup/independent/kubelet-integration/" rel="nofollow noreferrer">https://kubernetes.io/docs/setup/independent/kubelet-integration/</a> but it falls short for my particular use case. In that article, it seems to implicitly assumes a static set of nodes in the cluster. In my case I have a cluster autoscaler managing several AWS autoscale groups. I need a way to reconfigure the kubelet on each live node as well as any future nodes which are dynamically started (via <code>kubeadm join</code>).</p>

<p>My thought is to manually edit the kubelet-config-1.12 configmap to change the eviction thresholds then update the live nodes using the method in the article listed above.</p>

<p>Is there any issue with manually editing th the kubelet-config-1.12 configmap? Will the edits get carried over to the 1.13 version when the cluster is upgraded to that version?</p>

<p>Or if anyone has a better solution I'd like to hear it.</p>

<p>In addition to what <a href="https://stackoverflow.com/users/9892073/aurelius">aurelius</a> wrote in his answer,</p>

<p><a href="https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/" rel="nofollow noreferrer">DynamicKubeletConfig</a> Feature Gate is enabled by default starting from Kubernetes v1.11, but you need some additional steps to activate it.</p>

<p>As mentioned in the <a href="https://kubernetes.io/docs/tasks/administer-cluster/reconfigure-kubelet/#reconfiguring-the-kubelet-on-a-live-node-in-your-cluster" rel="nofollow noreferrer">documentation</a> (but can be missed easily): </p>

<ul>
<li>The Kubelet’s <code>--dynamic-config-dir</code> flag must be set to a writable directory on the Node.</li>
</ul>

<p>and by <code>kubelet -h</code></p>

<blockquote>
  <p><strong>--dynamic-config-dir string</strong><br>
                                                                                   The Kubelet will use this directory for checkpointing downloaded configurations and tracking configuration health.<br>
  The Kubelet will create this directory if it does not already exist.<br>
  The path may be absolute or relative; relative paths start at the Kubelet's current working directory.<br>
  <strong>Providing this flag enables dynamic Kubelet configuration.</strong><br>
  The DynamicKubeletConfig feature gate must be enabled to pass this flag; this gate currently defaults to true because the feature is beta.</p>
</blockquote>

<p>The best place to set this flag (for Ubuntu) is to add it to <code>/etc/defaults/kubelet</code>:</p>

<pre><code>$KUBELET_EXTRA_ARGS=--dynamic-config-dir=/var/lib/kubelet-dynamic
</code></pre>

<p>Restart <code>kubelet</code> service after that:</p>

<pre><code>$ sudo systemctl restart kubelet

$ ps aux | grep kubelet
root      8610  4.1  1.1 1115992 90652 ?       Ssl  14:57   0:46 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf  
--kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml  
--cgroup-driver=cgroupfs --network-plugin=cni  
--pod-infra-container-image=k8s.gcr.io/pause:3.1  
--dynamic-config-dir=/var/lib/kubelet-dynamic
</code></pre>

<p>After that <code>kubelet</code> creates directory tree under this directory to maintain checkpoints:</p>

<pre><code>$ sudo tree /var/lib/kubelet-dynamic/
/var/lib/kubelet-dynamic/
└── store
    ├── checkpoints
    │   └── 009e03e7-62ad-11e9-9043-42010a9c0003
    │       └── 12399979
    │           └── kubelet
    └── meta
        ├── assigned
        └── last-known-good

</code></pre>

<p>From this point everything should work as mentioned in documentation.</p>
