<p>I have a number of pods running and horizontal pod auto scaler assigned to target them, the cluster I am using can also add nodes and remove nodes automatically based on current load.</p>
<p>BUT we recently had the cluster go offline with OOM errors and this caused a disruption in service.</p>
<p>Is there a way to monitor the load on each node and if usage reaches say 80% of the memory on a node, Kubernetes should not schedule more pods on that node but wait for another node to come online.</p>

Margach Chris
<p>The pending pods are what one should monitor and define <strong>Resource requests</strong> which affect scheduling.</p>
<p>The Scheduler uses Resource requests Information when scheduling the pod
to a node. Each node has a certain amount of CPU and memory it can allocate to
pods. When scheduling a pod, the Scheduler will only consider nodes with enough
unallocated resources to meet the pod’s resource requirements. If the amount of
unallocated CPU or memory is less than what the pod requests, Kubernetes will not
schedule the pod to that node, because the node can’t provide the minimum amount
required by the pod. The new Pods will remain in Pending state until new nodes come into the cluster.</p>
<p>Example:</p>
<pre><code>apiVersion: v1
kind: Pod
metadata:
name: requests-pod
spec:
 containers:
 - image: busybox
   command: [&quot;dd&quot;, &quot;if=/dev/zero&quot;, &quot;of=/dev/null&quot;]
   name: main
   resources:
     requests:
       cpu: 200m
       memory: 10Mi
</code></pre>
<p>When you don’t specify a request for CPU, you’re saying you don’t care how much
CPU time the process running in your container is allotted. In the worst case, it may
not get any CPU time at all (this happens when a heavy demand by other processes
exists on the CPU). Although this may be fine for low-priority batch jobs, which aren’t
time-critical, it obviously isn’t appropriate for containers handling user requests.</p>
