<p>We are getting very frequently below message in <code>/var/log/messages</code></p>

<pre><code>kernel: SLUB: Unable to allocate memory on node -1 (gfp=0x8020)
</code></pre>

<p>In some cases followed by an allocation table</p>

<pre><code>kernel:    cache: sigqueue(12019:454c4ebd186d964699132181ad7367c669700f7d8991c47d4bc053ed101675bc), object size: 160, buffer size: 160, default order: 0, min order: 0
kernel:    node 0: slabs: 57, objs: 23313, free: 0
kernel:    node 1: slabs: 35, objs: 14315, free: 0
</code></pre>

<p>Ok, free is 0, but how may this be tuned?</p>

<p>Following is set information</p>

<hr>

<pre><code>OS - Centos7.3
Kernel - 3.10.0-327.36.3.el7.x86_64
Docker - 1.12.6
Kubernetes - 1.5.5
</code></pre>

<p>We have private cloud powered by kurbernetes, having 10 nodes; it was working fine till last month and now we are getting these alerts very frequently on every nodes, pods/container also increased in last few days.</p>

<p>We have enough  <code>memory</code> and <code>cpu</code> available on each node. </p>

<p>Any fine tuning for these alert will be very helpful.</p>

<p>Additional information:
<strong><code>sysctl.conf</code> options</strong></p>

<pre><code>net.ipv4.tcp_timestamps = 0 
net.ipv4.tcp_max_syn_backlog = 4096 
net.core.somaxconn = 1024 
net.ipv4.tcp_syncookies = 1 
net.core.rmem_max = 16777216 
net.core.wmem_max = 16777216 
net.core.rmem_default = 65535 
net.core.wmem_default = 65535 
net.ipv4.tcp_rmem = 4096 87380 16777216 
net.ipv4.tcp_wmem = 4096 65536 16777216 
net.ipv4.ip_local_port_range = 1024 65535 
vm.max_map_count = 262144 
vm.swappiness=10 
vm.vfs_cache_pressure=100
</code></pre>

<p>Please look at this: <a href="https://pingcap.com/blog/try-to-fix-two-linux-kernel-bugs-while-testing-tidb-operator-in-k8s/" rel="nofollow noreferrer">https://pingcap.com/blog/try-to-fix-two-linux-kernel-bugs-while-testing-tidb-operator-in-k8s/</a>. It's a kernel bug.</p>
