<p>Recently we've experienced issues with both Non-Production and Production clusters where the nodes encountered 'System OOM encountered' issue.</p>

<p>The nodes within the Non-Production cluster don't seem to be sharing the pods. It seems like a given node is running all the pods and putting a load on the system.</p>

<p>Also, the Pods are stuck in this status: 'Waiting: ContainerCreating'.</p>

<p>Any help/guidance with the above issues would be greatly appreciated. We are building more and more services in this cluster and want to make sure there's no instability and/or environment issues and place proper checks/configuration in place before we go live.</p>

<p>"I would recommend you manage container compute resources properly within your Kubernetes cluster. When creating a Pod, you can optionally specify how much CPU and memory (RAM) each Container needs to avoid OOM situations.</p>

<p>When Containers have resource requests specified, the scheduler can make better decisions about which nodes to place Pods on. And when Containers have their limits specified, contention for resources on a node can be handled in a specified manner. CPU specifications are in units of cores, and memory is specified in units of bytes.</p>

<p>An event is produced each time the scheduler fails, use the command below to see the status of events:</p>

<pre><code>$ kubectl describe pod &lt;pod-name&gt;| grep Events
</code></pre>

<p>Also, read the official Kubernetes guide on “Configure Out Of Resource Handling”. Always make sure to:</p>

<p>reserve <code>10-20%</code> of memory capacity for system daemons like kubelet and OS kernel
identify pods which can be evicted at <code>90-95%</code> memory utilization to reduce thrashing and incidence of system OOM.</p>

<p>To facilitate this kind of scenario, the kubelet would be launched with options like below:</p>

<pre><code>--eviction-hard=memory.available&lt;xMi
--system-reserved=memory=yGi
</code></pre>

<p>Replacing x and y with actual memory values.</p>

<p>Having <a href="https://github.com/kubernetes/heapster" rel="nofollow noreferrer">Heapster container monitoring</a> in place should be helpful for visualization".</p>

<p>Read more reading on <a href="https://www.aquasec.com/wiki/display/containers/Kubernetes" rel="nofollow noreferrer">Kubernetes and Docker Administration</a></p>
