<p>I have a statefulset deployed with 1 replica for jenkins. few days back the node on which jenkins pod was running went into NotReady State . Once Node went in NotReady state, Jenkins pod went in Terminating state and stuck there for long time until Node went back in Ready State.</p>
<p>Ideally, my Jenkins pod should have been re-scheduled to a healthy node in case my current node is not healthy. due to this my jenkins application had downtime for the time node was in Not Ready State.</p>
<p>Can anything be done in this case in order to prevent such downtime in statefulset pods</p>
<p>Kubectl version:</p>
<p>Client Version: version.Info{Major:&quot;1&quot;, Minor:&quot;19&quot;, GitVersion:&quot;v1.19.13&quot;, GitCommit:&quot;53c7b65d4531a749cd3a7004c5212d23daa044a9&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2021-07-15T20:58:11Z&quot;, GoVersion:&quot;go1.15.14&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}
Server Version: version.Info{Major:&quot;1&quot;, Minor:&quot;19&quot;, GitVersion:&quot;v1.19.13&quot;, GitCommit:&quot;53c7b65d4531a749cd3a7004c5212d23daa044a9&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2021-07-15T20:53:19Z&quot;, GoVersion:&quot;go1.15.14&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}</p>

user3426197
<p>Your issue could be because the cluster doesn't have enough resources or because the pod scheduler can't make a decision because there aren't enough labels on the nodes. You might want to look into the Kubernetes scheduler's logs to find out exactly what happened in this case. To better distribute the workload, you might want to think about expanding the size of your cluster or adding additional nodes if the resources on the cluster were insufficient. In addition, you might want to make sure that all of the cluster's nodes have accurate labels so that the pod scheduler knows which node is best for running your Jenkins pod.</p>
<p>A <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#node-affinity" rel="nofollow noreferrer">node affinity</a> rule in your Jenkins statefulset's deployment manifest can be set up to prevent downtime caused by a node entering the Not Ready state. The pod will only be scheduled for nodes that match a particular label, such as &quot;Ready=true,&quot; as a result of this. Additionally, you can use additional labels to indicate the health status of your cluster's nodes in your affinity rule. The pod will be rescheduled to a healthy node whenever a node enters the Not Ready state in this manner. Last but not least, you might want to think about using a pod disruption budget for your statefulset. This will make it less likely that the pod will be evicted from a node when it is not healthy.</p>
<p>Attaching supporting <a href="https://komodor.com/learn/how-to-fix-kubernetes-node-not-ready-error/" rel="nofollow noreferrer">blog-1</a> and <a href="https://www.datadoghq.com/blog/debug-kubernetes-pending-pods/" rel="nofollow noreferrer">blog-2</a> for reference.</p>
