<p>I am using kube-aws to create the kubernetes cluster on AWS, I have the kube-aws version v0.12.3, I am getting frequent issue on worker nodes as "too many open files in the system" when I try to ssh into the worker node and the nodes becomes unresponsive and gets restarted.</p>

<p>Because of this The pods running on the nodes gets rescheduled frequently on different nodes and application goes down for some time.</p>

<p>How can I resolve this issue.</p>

<p>✗ kubectl version
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3", GitCommit:"a4529464e4629c21224b3d52edfe0ea91b072862", GitTreeState:"clean", BuildDate:"2018-09-09T18:02:47Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3", GitCommit:"a4529464e4629c21224b3d52edfe0ea91b072862", GitTreeState:"clean", BuildDate:"2018-09-09T17:53:03Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}</p>

<p>Worker Node :
node|k8s-<strong>-</strong> core@ip-10-0-214-11 ~ $ ulimit -a </p>

<p>core file size          (blocks, -c) unlimited</p>

<p>data seg size           (kbytes, -d) unlimited</p>

<p>scheduling priority             (-e) 0</p>

<p>file size               (blocks, -f) unlimited</p>

<p>pending signals                 (-i) 251640</p>

<p>max locked memory       (kbytes, -l) 16384</p>

<p>max memory size         (kbytes, -m) unlimited</p>

<p><strong>open files                      (-n) 1024</strong></p>

<p>pipe size            (512 bytes, -p) 8</p>

<p>POSIX message queues     (bytes, -q) 819200</p>

<p>real-time priority              (-r) 0</p>

<p>stack size              (kbytes, -s) 8192</p>

<p>cpu time               (seconds, -t) unlimited</p>

<p>max user processes              (-u) 251640</p>

<p>virtual memory          (kbytes, -v) unlimited</p>

<p>file locks                      (-x) unlimited</p>

<p>As you can see the maximum number of open files is set to quite small value (<code>1024</code>). Perhaps this is inherited from the AWS template used for the worker node instance. </p>

<p>You should increase this value but this should be done with clear understanding of what level it should be set on: </p>

<ul>
<li>globally or for a specific security principal;   </li>
<li>what exact principal this limit has to be applied to: user/system/daemon account or a group;</li>
<li>login service (su, ssh, telnet, etc)</li>
</ul>

<p>Also, you should be careful in order to not to exceed the kernel limit. </p>

<p>For a simple case just add the two strings like below to the end of the /etc/security/limits.conf file: </p>

<pre><code>mike           soft    nofile          4096
mike           hard    nofile          65536
</code></pre>

<p>and then re-login or restart the service which account you make the changes for.</p>

<p>You could find further explanations in the Internet; 
one of many is available here: <a href="https://www.suse.com/documentation/sles-12/book_hardening/data/sec_sec_prot_dos.html" rel="nofollow noreferrer">Security and Hardening Guide</a></p>

<p>In order to keep those settings applied to your AWS instance during the launch, you might compose a simple script code like this: </p>

<pre><code>#!/bin/bash
cd /etc/security
cp limits.conf limits.conf.$(date "+%Y%m%d")
cat &lt;&lt;EndOfMyStrings &gt;&gt; limits.conf
mike           soft    nofile          4096
mike           hard    nofile          65536
EndOfMyStrings
</code></pre>

<p>and then add it into the "User data" field of the Launch Instance Wizard as described here: <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html" rel="nofollow noreferrer">Running Commands on Your Linux Instance at Launch</a> </p>
