<p>The Kubernetes docs on <a href="https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/" rel="noreferrer">https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/</a> state:</p>

<blockquote>
  <p>The scheduler ensures that, for each resource type, the sum of the resource requests of the scheduled Containers is less than the capacity of the node.</p>
</blockquote>

<p>Does Kubernetes consider the current state of the node when calculating capacity? To highlight what I mean, here is a concrete example:</p>

<p>Assuming I have a node with 10Gi of RAM, running 10 Pods each with 500Mi of resource requests, and no limits. Let's say they are "bursting", and each Pod is actually using 1Gi of RAM. In this case, the node is fully utilized (<code>10 x 1Gi = 10Gi</code>), but the resources requests are only <code>10 x 500Mi = 5Gi</code>. Would Kubernetes consider scheduling another pod on this node because only 50% of the memory capacity on the node has been <code>requested</code>, or would it use the fact that 100% of the memory is currently being utilized, and the node is at full capacity?</p>

<p>By default kubernetes will use cgroups to manage and monitor the  "allocatable" memory on a node for pods. It is possible to configure <code>kubelet</code> to entirely rely on the static <em>reservations</em> and pod <em>requests</em> from your deployments though so the method depends on your cluster deployment.</p>

<p>In either case, a node itself will track "memory pressure", which monitors the existing overall memory usage of a node. If a node is under memory pressure then no new pods will be scheduled and existing pods will be evicted.</p>

<p>It's best to set sensible memory <em>requests</em> and <em>limits</em> for all workloads to help the scheduler as much as possible. 
If a kubernetes deployment does not configure cgroup memory monitoring, setting <em>requests</em> is a requirement for <em>all</em> workloads.
If the deployment is using cgroup memory monitoring, at least setting <em>requests</em> give the scheduler extra detail as to whether the pods to be scheduled should fit on a node.  </p>

<h2>Capacity and Allocatable Resources</h2>

<p>The <a href="https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/" rel="noreferrer">Kubernetes Reserve Compute Resources docco</a> has a good overview of how memory is viewed on a node.</p>

<pre><code>      Node Capacity
---------------------------
|     kube-reserved       |
|-------------------------|
|     system-reserved     |
|-------------------------|
|    eviction-threshold   |
|-------------------------|
|                         |
|      allocatable        |
|   (available for pods)  |
|                         |
|                         |
---------------------------
</code></pre>

<p>The default scheduler checks a node isn't under memory pressure, then looks at the <em>allocatable</em> memory available on a node and whether the new pods <em>requests</em> will fit in it. </p>

<p>The <em>allocatable</em> memory available is the <code>total-available-memory - kube-reserved - system-reserved - eviction-threshold - scheduled-pods</code>.</p>

<h3>Scheduled Pods</h3>

<p>The value for <code>scheduled-pods</code> can be calculated via a dynamic cgroup, or statically via the pods <em>resource requests</em>.</p>

<p>The kubelet <code>--cgroups-per-qos</code> option which defaults to <code>true</code> enables cgroup tracking of scheduled pods. The pods kubernetes runs will be in </p>

<p>If <code>--cgroups-per-qos=false</code> then the <em>allocatable</em> memory will only be reduced by the <em>resource requests</em> that scheduled on a node. </p>

<h3>Eviction Threshold</h3>

<p>The <code>eviction-threshold</code> is the level of free memory when Kubernetes starts evicting pods. This defaults to 100MB but can be set via the kubelet command line. This setting is teid to both the <em>allocatable</em> value for a node and also the memory pressure state of a node in the next section.</p>

<h3>System Reserved</h3>

<p>kubelets <code>system-reserved</code> value can be configured as a static value (<code>--system-reserved=</code>) or monitored dynamically via cgroup (<code>--system-reserved-cgroup=</code>).
This is for any system daemons running outside of kubernetes (<code>sshd</code>, <code>systemd</code> etc). If you configure a cgroup, the processes all need to be placed in that cgroup. </p>

<h3>Kube Reserved</h3>

<p>kubelets <code>kube-reserved</code> value can be configured as a static value (via <code>--kube-reserved=</code>) or monitored dynamically via cgroup (<code>--kube-reserved-cgroup=</code>).
This is for any kubernetes services running outside of kubernetes, usually <code>kubelet</code> and a container runtime.</p>

<h3>Capacity and Availability on a Node</h3>

<p>Capacity is stored in the Node object.</p>

<pre><code>$ kubectl get node node01 -o json | jq '.status.capacity'
{
  "cpu": "2",
  "ephemeral-storage": "61252420Ki",
  "hugepages-1Gi": "0",
  "hugepages-2Mi": "0",
  "memory": "4042284Ki",
  "pods": "110"
}
</code></pre>

<p>The allocatable value can be found on the Node, you can note than existing usage doesn't change this value. Only schduleding pods with resource requests will take away from the <code>allocatable</code> value. </p>

<pre><code>$ kubectl get node node01 -o json | jq '.status.allocatable'
{
  "cpu": "2",
  "ephemeral-storage": "56450230179",
  "hugepages-1Gi": "0",
  "hugepages-2Mi": "0",
  "memory": "3939884Ki",
  "pods": "110"
}
</code></pre>

<h2>Memory Usage and Pressure</h2>

<p>A kube node can also have a "memory pressure" event. This check is done outside of the <em>allocatable</em> resource checks above and is more a system level catch all. Memory pressure looks at the current root cgroup memory usage minus the inactive file cache/buffers, similar to the calculation <code>free</code> does to remove the file cache. </p>

<p>A node under memory pressure will not have pods scheduled, and will actively try and evict existing pods until the memory pressure state is resolved. </p>

<p>You can set the eviction threshold amount of memory kubelet will maintain available via the <code>--eviction-hard=[memory.available&lt;500Mi]</code> flag. The memory requests and usage for pods can help informs the eviction process.</p>

<p><code>kubectl top node</code> will give you the existing memory stats for each node (if you have a metrics service running).</p>

<pre><code>$ kubectl top node
NAME                 CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
node01               141m         7%     865Mi           22%       
</code></pre>

<p>If you were not using <code>cgroups-per-qos</code> and a number of pods without resource limits, or a number of system daemons then the cluster is likely to have some problems scheduling on a memory constrained system as <em>allocatable</em> will be high but the actual value might be really low. </p>

<h3>Memory Pressure calculation</h3>

<p>Kubernetes <a href="https://kubernetes.io/docs/tasks/administer-cluster/out-of-resource/" rel="noreferrer">Out Of Resource Handling docco</a> includes a <a href="https://kubernetes.io/docs/tasks/administer-cluster/out-of-resource/memory-available.sh" rel="noreferrer">script</a> which emulates kubelets memory monitoring process: </p>

<pre><code># This script reproduces what the kubelet does
# to calculate memory.available relative to root cgroup.

# current memory usage
memory_capacity_in_kb=$(cat /proc/meminfo | grep MemTotal | awk '{print $2}')
memory_capacity_in_bytes=$((memory_capacity_in_kb * 1024))
memory_usage_in_bytes=$(cat /sys/fs/cgroup/memory/memory.usage_in_bytes)
memory_total_inactive_file=$(cat /sys/fs/cgroup/memory/memory.stat | grep total_inactive_file | awk '{print $2}')

memory_working_set=${memory_usage_in_bytes}
if [ "$memory_working_set" -lt "$memory_total_inactive_file" ];
then
    memory_working_set=0
else
    memory_working_set=$((memory_usage_in_bytes - memory_total_inactive_file))
fi

memory_available_in_bytes=$((memory_capacity_in_bytes - memory_working_set))
memory_available_in_kb=$((memory_available_in_bytes / 1024))
memory_available_in_mb=$((memory_available_in_kb / 1024))

echo "memory.capacity_in_bytes $memory_capacity_in_bytes"
echo "memory.usage_in_bytes $memory_usage_in_bytes"
echo "memory.total_inactive_file $memory_total_inactive_file"
echo "memory.working_set $memory_working_set"
echo "memory.available_in_bytes $memory_available_in_bytes"
echo "memory.available_in_kb $memory_available_in_kb"
echo "memory.available_in_mb $memory_available_in_mb"
</code></pre>
