<p>We are using a self-hosted microk8s cluster (single-node for now) for our internal staging workloads. From time to time, the server becomes unresponsive and I can't even ssh into it. The only way out is a restart.</p>
<p>I can see that before the server crashes, its memory usage goes to the limit and the CPU load shoots up to over 1000. So running out of resources is likely to blame.</p>
<p>That brings me to the question - <strong>how can I set global limits for microk8s to not consume <em>everything</em>?</strong></p>
<hr />
<p>I know there are resource limits that can be assigned to Kubernetes pods, and ResourceQuotas to limit aggregate namespace resources. But that has the downside of low resource utilization (if I understand those right). For simplicity, let's say:</p>
<ul>
<li>each pod is the same</li>
<li>its real memory needs can go from <code>50 MiB</code> to <code>500 MiB</code></li>
<li>each pod is running in its own namespace</li>
<li>there are 30 pods</li>
<li>the server has 8 GiB of RAM</li>
</ul>
<ol>
<li><p>I assign <code>request: 50 Mi</code> and <code>limit: 500 Mi</code> to the pod. As long as the node has at least <code>50 * 30 Mi = 1500 Mi</code> of memory, it should run all the requested pods. But there is nothing stopping all of the pods using <code>450 Mi</code> of memory each, which is under the individual limits, but still in total being <code>450 Mi * 30 = 13500 Mi</code>, which is more than the server can handle. And I suspect this is what leads to the server crash in my case.</p>
</li>
<li><p>I assign <code>request: 500 Mi</code> and <code>limit: 500 Mi</code> to the pod to ensure the total memory usage never goes above what I anticipate. This will of course allow me to only schedule 16 pods. But when the pods run with no real load and using just <code>50 Mi</code> of memory, there is severe RAM underutilization.</p>
</li>
<li><p>I am looking for a third option. Something to let me schedule pods freely and <em>only</em> start evicting/killing them when the total memory usage goes above a certain limit. And that limit needs to be configurable and lower than the total memory of the server, so that it does not die.</p>
</li>
</ol>
<hr />
<p>We are using microk8s but I expect this is a problem all self-hosted nodes face, as well as something AWS/Google/Azure have to deal with too.</p>
<p>Thanks</p>

Martin Melka
<p>Since microk8s running on the host machine, then all resources of the host are allocated for it. That is why if you want to keep your cluster resources in borders, you have to manage them in one of the ways below:</p>
<ol>
<li>Setup <a href="https://kubernetes.io/docs/concepts/policy/limit-range/" rel="nofollow noreferrer">LimitRange</a> policy for pods in a namespace.</li>
</ol>
<blockquote>
<p>A  <em>LimitRange</em>  provides constraints that can:</p>
<ul>
<li>Enforce minimum and maximum compute resources usage per Pod or Container in a namespace.</li>
<li>Enforce minimum and maximum storage request per PersistentVolumeClaim in a namespace.</li>
<li>Enforce a ratio between request and limit for a resource in a namespace.</li>
<li>Set default request/limit for compute resources in a namespace and automatically inject them to Containers at runtime.</li>
</ul>
</blockquote>
<ol start="2">
<li>Use <a href="https://kubernetes.io/docs/concepts/policy/resource-quotas/#compute-resource-quota" rel="nofollow noreferrer">Resource Quotas</a> per namespace.</li>
</ol>
<blockquote>
<p>A resource quota, defined by a <em>ResourceQuota</em> object, provides
constraints that limit aggregate resource consumption per namespace.
It can limit the quantity of objects that can be created in a
namespace by type, as well as the total amount of compute resources
that may be consumed by resources in that namespace.</p>
</blockquote>
<ol start="3">
<li>Assign necessary <a href="https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#requests-and-limits" rel="nofollow noreferrer">requests and limits</a> for each pod.</li>
</ol>
<blockquote>
<p>When you specify the resource  <em>request</em>  for Containers in a Pod, the
scheduler uses this information to decide which node to place the Pod
on. When you specify a resource  <em>limit</em>  for a Container, the kubelet
enforces those limits so that the running container is not allowed to
use more of that resource than the limit you set. The kubelet also
reserves at least the  <em>request</em>  amount of that system resource
specifically for that container to use.</p>
</blockquote>
