<p>My Jenkins X installation, mid-project, is now becoming very unstable. (Mainly) Jenkins pods are failing to start due to disk pressure.</p>

<p>Commonly, many pods are failing with</p>

<blockquote>
  <p>The node was low on resource: [DiskPressure].</p>
</blockquote>

<p>or</p>

<blockquote>
  <p>0/4 nodes are available: 1 Insufficient cpu, 1 node(s) had disk pressure, 2 node(s) had no available volume zone.
  Unable to mount volumes for pod "jenkins-x-chartmuseum-blah": timeout expired waiting for volumes to attach or mount for pod "jx"/"jenkins-x-chartmuseum-blah". list of unmounted volumes=[storage-volume]. list of unattached volumes=[storage-volume default-token-blah]
  Multi-Attach error for volume "pvc-blah" Volume is already exclusively attached to one node and can't be attached to another</p>
</blockquote>

<p>This may have become more pronounced with more preview builds for projects with npm and the massive <code>node-modules</code> directories it generates. I'm also not sure if Jenkins is cleaning up after itself.</p>

<p>Rebooting the nodes helps, but not for very long.</p>

<p>Let's approach this from the Kubernetes side. 
There are few things you could do to fix this:</p>

<ol>
<li>As mentioned by @Vasily check what is causing disk pressure on nodes. You may also need to check logs from:

<ul>
<li><code>kubeclt logs: kube-scheduler events logs</code></li>
<li><code>journalctl -u kubelet: kubelet logs</code></li>
<li><code>/var/log/kube-scheduler.log</code></li>
</ul></li>
</ol>

<p>More about why those logs below.</p>

<ol start="2">
<li><p>Check your Eviction Thresholds. Adjust Kubelet and Kube-Scheduler configuration if needed. See what is happening with both of them (logs mentioned earlier might be useful now). More info can be found <a href="https://kubernetes.io/docs/tasks/administer-cluster/out-of-resource/" rel="nofollow noreferrer">here</a></p></li>
<li><p>Check if you got a correctly running Horizontal Pod Autoscaler: <code>kubectl get hpa</code>
You can use standard kubectl commands to setup and <a href="https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#support-for-horizontal-pod-autoscaler-in-kubectl" rel="nofollow noreferrer">manage your HPA.</a></p></li>
<li><p>Finally, the volume related errors that you receive indicates that we might have problem with PVC and/or PV. Make sure you have your volume in the same zone as node. If you want to mount the volume to a specific container make sure it is not exclusively attached to another one. More info can be found <a href="https://kubernetes.io/docs/concepts/storage/volumes/" rel="nofollow noreferrer">here</a> and <a href="https://kubernetes.io/docs/concepts/storage/persistent-volumes/" rel="nofollow noreferrer">here</a></p></li>
</ol>

<p>I did not test it myself because more info is needed in order to reproduce the whole scenario but I hope that above suggestion will be useful.</p>

<p>Please let me know if that helped.</p>
