<p>I recently tried to bring up a Kubernetes cluster in AWS using kops. But when the worker node (Ubuntu 20.04) started, a <code>docker load</code> process on it kept getting OOMkilled even when it has enough memory (~14GiB). I tracked down the issue being I set <code>kubelet</code>'s  memory reservation too small (<code>--kube-reserved=memory=100Mi...</code>).</p>
<p>So now I have two questions related to the following paragraph in the documentation:</p>
<blockquote>
<p>kube-reserved is meant to capture resource reservation for kubernetes system daemons like the kubelet, container runtime, node problem detector, etc.</p>
</blockquote>
<p><a href="https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/#kube-reserved" rel="nofollow noreferrer">https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/#kube-reserved</a></p>
<p>First, I interpreted the &quot;reservation&quot; as &quot;the amount of memory guaranteed&quot;, similar to the concept of a pod's <code>.spec.resource.requests.memory</code>. However, it seems like the flag acts like a <code>limit</code> as well? Does this mean Kubernetes intend to manage Kubernetes system daemons with &quot;guaranteed&quot; QoS class concept?</p>
<p>Also, my container runtime, <code>docker</code>, does not seem to be in <code>/kube-reserved</code> cgroup, instead, it is in <code>/system.slice</code>:</p>
<pre><code>$ systemctl status $(pgrep dockerd) | grep CGroup
     CGroup: /system.slice/docker.service
</code></pre>
<p>So why is it getting limited by <code>/kube-reserved</code>? It is not even <code>kubelet</code> talking to <code>docker</code> through <code>CRI</code>, but just my manual <code>docker load</code> command.</p>

Yuhao Zhang
<p><code>kube-reserved</code> is a way to protect Kubernetes system daemons (which includes the Kubelet) from running out of memory should the pods consume too much. How is this achieved? The pods are limited by default to an &quot;allocatable&quot; value, equal to the memory capacity of the node minus several flag values defined in <a href="https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/#node-allocatable" rel="nofollow noreferrer">the URL you posted</a>, one of which is <code>kube-reserved</code>. Here's what this looks like for a 7-GiB DS2_v2 node in AKS:</p>
<p><a href="https://i.stack.imgur.com/mZdMA.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/mZdMA.jpg" alt="Node Allocatable and node memory capacity distribution for a 7-GiB DS2_v2 AKS node" /></a></p>
<p>But it's not always the Kubernetes system daemons that have to be protected from either pods or even OS components consuming too much memory. It can very well be the Kubernetes system daemons that could consume too much memory and start affecting the pods or other OS components. To protect against this scenario, there's an <a href="https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/#kube-reserved" rel="nofollow noreferrer">additional flag defined</a>:</p>
<blockquote>
<p>To optionally enforce <code>kube-reserved</code> on kubernetes system daemons,
specify the parent control group for kube daemons as the value for
<code>--kube-reserved-cgroup</code> kubelet flag.</p>
</blockquote>
<p>With this new flag in place, should the aggregated memory use of the Kubernetes system daemons exceed the cgroup limit, then the OOM killer will step in and terminate one of their processes. To apply this to the picture above, with the <code>--kube-reserved-cgroup</code> flag specified, the Kubernetes system daemons are prevented from going over 1,638 MiB.</p>
