<p>I'm using <a href="https://github.com/Unitech/pm2" rel="nofollow noreferrer">pm2</a> to watch the directory holding the source code for my app-server's NodeJS program, running within a Kubernetes cluster.</p>
<p>However, I am getting this error:</p>
<pre><code>ENOSPC: System limit for number of file watchers reached
</code></pre>
<p>I searched on that error, and found this answer: <a href="https://stackoverflow.com/a/55763478">https://stackoverflow.com/a/55763478</a></p>
<pre><code># insert the new value into the system config
echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf &amp;&amp; sudo sysctl -p
</code></pre>
<p>However, I tried running that in a pod on the target k8s node, and it says the command <code>sudo</code> was not found. If I remove the <code>sudo</code>, I get this error:</p>
<pre><code>sysctl: setting key &quot;fs.inotify.max_user_watches&quot;: Read-only file system
</code></pre>
<p>How can I modify the file-system watcher limit from the 8192 found on my Kubernetes node, to a higher value such as 524288?</p>

<p>I found a solution: use a privileged Daemon Set that runs on each node in the cluster, which has the ability to modify the <code>fs.inotify.max_user_watches</code> variable.</p>
<p>Add the following to a <code>node-setup-daemon-set.yaml</code> file, included in your Kubernetes cluster:</p>
<pre><code>apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-setup
  namespace: kube-system
  labels:
    k8s-app: node-setup
spec:
  selector:
    matchLabels:
      name: node-setup
  template:
    metadata:
      labels:
        name: node-setup
    spec:
      containers:
      - name: node-setup
        image: ubuntu
        command: [&quot;/bin/sh&quot;,&quot;-c&quot;]
        args: [&quot;/script/node-setup.sh; while true; do echo Sleeping &amp;&amp; sleep 3600; done&quot;]
        env:
          - name: PARTITION_NUMBER
            valueFrom:
              configMapKeyRef:
                name: node-setup-config
                key: partition_number
        volumeMounts:
          - name: node-setup-script
            mountPath: /script
          - name: dev
            mountPath: /dev
          - name: etc-lvm
            mountPath: /etc/lvm
        securityContext:
          allowPrivilegeEscalation: true
          privileged: true
      volumes:
        - name: node-setup-script
          configMap:
            name: node-setup-script
            defaultMode: 0755
        - name: dev
          hostPath:
            path: /dev
        - name: etc-lvm
          hostPath:
            path: /etc/lvm
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: node-setup-config
  namespace: kube-system
data:
  partition_number: &quot;3&quot;
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: node-setup-script
  namespace: kube-system
data:
  node-setup.sh: |
    #!/bin/bash
    set -e

    # change the file-watcher max-count on each node to 524288

    # insert the new value into the system config
    sysctl -w fs.inotify.max_user_watches=524288

    # check that the new value was applied
    cat /proc/sys/fs/inotify/max_user_watches
</code></pre>
<p>Note: The file above could probably be simplified quite a bit. (I was basing it on <a href="https://docs.ovh.com/gb/en/kubernetes/formating-nvme-disk-on-iops-nodes" rel="nofollow noreferrer">this guide</a>, and left in a lot of stuff that's probably not necessary for simply running the <code>sysctl</code> command.) If others succeed in trimming it further, while confirming that it still works, feel free to make/suggest those edits to my answer.</p>
