<p>I've spent over a full day trying to make sense of Kubernetes' resource management. Specifically, I'm trying to set up eviction thresholds and resource reservations in such a way that there is always at least 1GiB of memory available.</p>
<p>Going on the documentation regarding <a href="https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/#eviction-thresholds" rel="nofollow noreferrer">resource reservations</a> and <a href="https://kubernetes.io/docs/tasks/administer-cluster/out-of-resource/#schedulable-resources-and-eviction-policies" rel="nofollow noreferrer">out-of-resource handling</a>, I figured setting the following eviction policy would suffice:</p>
<pre><code>--eviction-hard=memory.available&lt;1Gi
</code></pre>
<p>However, in practice, this does not work at all, as the computation the kubelet does seems to be different from the computation the kernel does when it needs to determine whether or not the OOM killer needs to be invoked. E.g. when I load up my system with a bunch of pods running an artificial memory hog, I get the following report from <code>free -m</code>:</p>
<pre><code>Total:      15866
Used:       14628
free:       161
shared:     53
buff/cache: 1077
available:  859
</code></pre>
<p>According to the kernel, there's 859 MiB memory available. Yet, the kubelet does not invoke its eviction policy. In fact, I've been able to invoke the system OOM killer before the kubelet eviction policy was invoked, even when ramping up memory usage incredibly slowly (to allow the kubelet housekeeing control loop to sleep 10 seconds, as per its default configuration).</p>
<p>I've found <a href="https://github.com/kubernetes/website/blob/release-1.18/content/en/docs/tasks/administer-cluster/memory-available.sh" rel="nofollow noreferrer">this script</a> which used to be in Kubernetes documentation and is supposed to calculate the available memory in the same way the Kubelet does. I ran it in parallel to <code>free -m</code> above and got the following result:</p>
<pre><code>memory.available_in_mb 1833
</code></pre>
<p>That's almost 1000M difference!</p>
<p>Now, I understand the calculation was by design, but that leaves me with the obvious question: how can I reliably manage system resource usage so that the system OOM killer does not get invoked? What eviction policy can I set so the kubelet will start evicting pods when there's less than a gigabyte of memory available?</p>

<p>According to documentation <a href="https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/" rel="nofollow noreferrer">https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/</a>, you should add the Kubelet flag <code>--system-reserved=memory=1024Mi</code></p>
