<p>The resource limit of Pod has been set as:</p>

<pre><code>resource
  limit
    cpu: 500m
    memory: 5Gi
</code></pre>

<p>and there's <code>10G</code> mem left on the node.</p>

<p>I've created <code>5</code> pods in a short time successfully, and the node maybe still have some mem left, e.g. <code>8G</code>.</p>

<p>The mem usage is growing as the time goes on, and reach the limit (<code>5G x 5 = 25G &gt; 10G</code>), then the node will be out of response.</p>

<p>In order to ensure the usability, is there a way to set the resource limit on the node? </p>

<h2>Update</h2>

<p>The core problem is that pod memory usage does not always equal to the limit, especially in the time when it just starts. So there can be unlimited pods created as soon as possible, then make all nodes full load. That's not good. There might be something to allocate resources rather than setting the limit.</p>

<h2>Update 2</h2>

<p>I've tested again for the limits and resources:</p>

<pre><code>resources:
  limits:
    cpu: 500m
    memory: 5Gi
  requests:
    cpu: 500m
    memory: 5Gi
</code></pre>

<p>The total mem is 15G and left 14G, but 3 pods are scheduled and running successfully:</p>

<pre><code>&gt; free -mh
              total        used        free      shared  buff/cache   available
Mem:            15G        1.1G        8.3G        3.4M        6.2G         14G
Swap:            0B          0B          0B

&gt; docker stats

CONTAINER           CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O
44eaa3e2d68c        0.63%               1.939 GB / 5.369 GB   36.11%              0 B / 0 B           47.84 MB / 0 B
87099000037c        0.58%               2.187 GB / 5.369 GB   40.74%              0 B / 0 B           48.01 MB / 0 B
d5954ab37642        0.58%               1.936 GB / 5.369 GB   36.07%              0 B / 0 B           47.81 MB / 0 B
</code></pre>

<p>It seems that the node will be crushed soon XD</p>

<h2>Update 3</h2>

<p>Now I change the resources limits, request <code>5G</code> and limit <code>8G</code>:</p>

<pre><code>resources:
  limits:
    cpu: 500m
    memory: 5Gi
  requests:
    cpu: 500m
    memory: 8Gi
</code></pre>

<p>The results are:
<a href="https://i.stack.imgur.com/rhpEB.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/rhpEB.jpg" alt="enter image description here"></a></p>

<p>According to the <a href="https://github.com/kubernetes/kubernetes/blob/8c60068b368f8ec1cd21fdf25d44c7b165aa9d27/plugin/pkg/scheduler/algorithm/predicates/predicates.go#L393" rel="nofollow noreferrer">k8s source code about the resource check</a>:</p>

<p><a href="https://i.stack.imgur.com/auyM2.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/auyM2.jpg" alt="enter image description here"></a></p>

<p>The total memory is only <code>15G</code>, and all the pods needs <code>24G</code>, so all the pods may be killed. (my single one container will cost more than <code>16G</code> usually if not limited.)</p>

<p>It means that you'd better keep the <code>requests</code> <strong>exactly equals</strong> to the <code>limits</code> in order to avoid pod killed or node crush. As if the <code>requests</code> value is not specified, it will be <a href="http://kubernetes.io/v1.1/docs/user-guide/compute-resources.html" rel="nofollow noreferrer">set to the <code>limit</code> as default</a>, so what exactly <code>requests</code> used for? I think only <code>limits</code> is totally enough, or IMO, on the contrary of what K8s claimed, I rather like to <strong>set the resource request greater than the limit</strong>, in order to ensure the usability of nodes.</p>

<h2>Update 4</h2>

<p>Kubernetes <a href="https://github.com/kubernetes/kubernetes/blob/1ef28633870d66fba6883bf1d667dbcfd83f200b/plugin/pkg/scheduler/algorithm/predicates/predicates.go#L120" rel="nofollow noreferrer">1.1 schedule the pods mem requests</a> via the formula:</p>

<p><code>(capacity - memoryRequested) &gt;= podRequest.memory</code></p>

<p>It seems that kubernetes is not caring about memory usage as <a href="https://stackoverflow.com/users/5289649/vishnu-kannan">Vishnu Kannan</a> said.  So the node will be crushed if the mem used much by other apps.</p>

<p>Fortunately, from the commit <a href="https://github.com/kubernetes/kubernetes/commit/e64fe82245303f08f32b0fb9a2fa9efe4a08f91f" rel="nofollow noreferrer">e64fe822</a>, the formula has been changed as:</p>

<p><code>(allocatable - memoryRequested) &gt;= podRequest.memory</code></p>

<p>waiting for the k8s v1.2!</p>

<p>Kubernetes resource specifications have two fields, <code>request</code> and <code>limit</code>.</p>

<p><code>limits</code> place a cap on how much of a resource a container can use. For memory, if a container goes above its limits, it will be OOM killed. For CPU, its usage may be throttled.</p>

<p><code>requests</code> are different in that they ensure the node that the pod is put on has at least that much capacity available for it. If you want to make sure that your pods will be able to grow to a particular size without the node running out of resources, specify a request of that size. This will limit how many pods you can schedule, though -- a 10G node will only be able to fit 2 pods with a 5G memory request.</p>
