<p>I'm using simple pattern where one Node had one Pod in it, and that Pod is controlled by a Deployment with one replicas set.</p>
<p>Deployment is there to ensure Pod restarts when it gets evicted by <code>DiskPressureEviction</code>.
The problem I'm facing is caused by Deployment retrying to restart the Pod too fast. As the Pod is set to be in specific Node that hasn't cleaned up <code>DiskPressure</code> yet, restarting Pod fails sequentially before Node is ready to accept new Pod:</p>
<pre><code>NAME                                 READY   STATUS              RESTARTS   AGE
deployment-adid-7bb998fccc-4v9dx     0/1     Evicted             0          6m17s
deployment-adid-7bb998fccc-59kvv     0/1     Evicted             0          6m20s
deployment-adid-7bb998fccc-59zzl     0/1     Evicted             0          6m20s
deployment-adid-7bb998fccc-dmm9k     0/1     Evicted             0          6m16s
deployment-adid-7bb998fccc-gn59z     0/1     Evicted             0          6m20s
deployment-adid-7bb998fccc-j4v25     0/1     Evicted             0          6m18s
deployment-adid-7bb998fccc-mw4ps     0/1     Evicted             0          6m20s
deployment-adid-7bb998fccc-n7krq     0/1     Evicted             0          18h
deployment-adid-7bb998fccc-rm4tr     0/1     Evicted             0          6m18s
deployment-adid-7bb998fccc-vn44q     0/1     ContainerCreating   0          6m15s
</code></pre>
<p>Here, 8 Pods are created and evicted in like 5 seconds before 9th get accepted by the designated Node.</p>
<p>While the last Pod finally becomes <code>Running</code> I don't like making garbage Pods. Would be nice if Pod can wait for the Node becoming ready, yet if it's impossible, I suppose restarting could be simply delayed. Presumably by describing <code>waitTime</code> before start recreating a Pod, or describing <code>restartInterval</code> that says in which interval should Deployment try to restart Pod.</p>
<p>So how can I set this kind of control in Deployment's spec?</p>
<p>ADDITION:</p>
<p>With excluding meaningless labels, Deployment spec is something like this:</p>
<pre><code>deployment_template = {
    'apiVersion': 'apps/v1',
    'kind': 'Deployment',
    'metadata': {
        'name': 'first',
    },
    'spec': {
        'replicas': '1',
        'selector': {
            'matchLabels': {
                &quot;podName&quot; : &quot;first&quot;
            }
        },
        'template': {
            'metadata': {
                'labels': {
                    &quot;podName&quot; : &quot;first&quot;
                }
            },
            'spec': {
                'nodeSelector': {
                    &quot;node&quot;: &quot;1&quot;
                },
                'restartPolicy': 'Always',
                'hostNetwork': True,
                'dnsPolicy': 'ClusterFirstWithHostNet',
                'containers': [
                    {
                        'name': 'containername',
                        'image': &quot;somecontainerimage&quot;,
                        'imagePullPolicy': 'Always',
                    }
                ]
            }
        }
    }
}
</code></pre>

김기영
<p>First I'd suggest updating to the newest, supported Kubernetes version. The maintenance support for version 1.17 that you are using <a href="https://endoflife.date/kubernetes" rel="nofollow noreferrer">ended 11 months ago</a>. <a href="https://kubernetes.io/blog/2021/12/07/kubernetes-1-23-release-announcement/" rel="nofollow noreferrer">The actual version (as of today 15.12.2021) is v1.23</a>. Since Kubernetes v1.18 <a href="https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/#feature-gates-for-graduated-or-deprecated-features" rel="nofollow noreferrer">the feature <code>TaintBasedEvictions</code></a> is in stable mode.</p>
<p>Another thing is that, instead of trying to delay the deployment which is kind of a workaround and not the best practice and better to fix a main issue which is disk pressure eviction that you are occurring. You should consider changing behaviour of your application, or at least try to avoid disk pressure on node by increasing it's storage size.</p>
<p>Anyway, If you want to keep it in that way, you may try to setup some additional parameters. You can't itself delay the deployment, but you can change the behaviour of the <a href="https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/" rel="nofollow noreferrer">kubelet agent</a> on your node.</p>
<hr />
<p>Below example is for the Kubernetes version 1.23. Keep in mind that for version 1.17 it may differ.</p>
<p>I created a cluster with one master node and one worker node, the pods are only scheduled on the worker node. I am fulfilling worker storage to create <code>node.kubernetes.io/disk-pressure</code>. By default the behaviour is similar to yours, many pods are created in <code>Evicted</code> state, which, worth to note, it's totally normal and it's expected behaviour. They are creating until node get taint <code>disk-pressure</code>, <a href="https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/" rel="nofollow noreferrer">which is occurring after ~10 seconds by default</a>:</p>
<blockquote>
<p>nodeStatusUpdateFrequency is the frequency that kubelet computes node status. ... Default: &quot;10s&quot;</p>
</blockquote>
<p>After that time, as you can observe, there are no pods created in <code>Evicted</code> state. The taint is deleted (i.e in you case the disk storage on node is back to the proper value) after ~5 min, it is <a href="https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/#kubelet-config-k8s-io-v1beta1-KubeletConfiguration" rel="nofollow noreferrer">defined by <code>evictionPressureTransitionPeriod</code> parameter</a>:</p>
<blockquote>
<p>evictionPressureTransitionPeriod is the duration for which the kubelet has to wait before transitioning out of an eviction pressure condition. ... Default: &quot;5m&quot;</p>
</blockquote>
<p>Okay, let's change some configuration by editing <a href="https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/#kubelet-config-k8s-io-v1beta1-KubeletConfiguration" rel="nofollow noreferrer">kubelet config file</a> on the worker node- it is located at <code>/var/lib/kubelet/config.yaml</code> for kubeadm.</p>
<p>I will change three parameters:</p>
<ul>
<li>earlier mentioned <a href="https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/#kubelet-config-k8s-io-v1beta1-KubeletConfiguration" rel="nofollow noreferrer"><code>evictionPressureTransitionPeriod</code> parameter</a> set to 120s so taint will be deleted faster</li>
<li><a href="https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/#kubelet-config-k8s-io-v1beta1-KubeletConfiguration" rel="nofollow noreferrer"><code>evictionSoft</code></a> to define a soft eviction - in my case it will occur when worker node has available less than 15GB of the storage</li>
<li><a href="https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/#kubelet-config-k8s-io-v1beta1-KubeletConfiguration" rel="nofollow noreferrer"><code>evictionSoftGracePeriod</code></a> to define a period after pod will enter into eviction state if defined <code>evictionSoft</code>occurs, in my case it's 60 seconds</li>
</ul>
<p>The file <code>var/lib/kubelet/config.yaml</code> - only the changed / added fields:</p>
<pre><code>evictionPressureTransitionPeriod: 120s
evictionSoftGracePeriod: 
  nodefs.available: 60s
evictionSoft:
  nodefs.available: 15Gi 
</code></pre>
<p>To sum up - after my node storage is less than 15 GB, the pod will be in running state for 60 seconds. After that, is storage is still less than 15 GB, pods will enter into <code>Evicted</code> / <code>Completed</code> state, the new pods will occur in <code>Pending</code> state:</p>
<pre><code>NAME                                   READY   STATUS      RESTARTS   AGE
my-nginx-deployment-6cf77b6d6b-2hr2s   0/1     Completed   0          115m
my-nginx-deployment-6cf77b6d6b-8f8wv   0/1     Completed   0          115m
my-nginx-deployment-6cf77b6d6b-9kpc9   0/1     Pending     0          108s
my-nginx-deployment-6cf77b6d6b-jbx5g   0/1     Pending     0          107s
</code></pre>
<p>After the available storage is higher than 15 GB, it will take 2 minutes to remove the taint and create new pods.</p>
<p>If during these 60 seconds the available storage will be again higher than 15GB, then no action will be done, the pods will be still in <code>Running</code> state.</p>
<p>If you have any garbage pods running, run this command to delete them:</p>
<pre><code>kubectl get pods | grep -e &quot;ContainerStatusUnknown&quot; -e &quot;Evicted&quot; -e &quot;Completed&quot; -e &quot;Error&quot; | awk '{print $1}' | xargs kubectl delete pod
</code></pre>
<p>Keep in mind that pod eviction may behave differently for different <a href="https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/" rel="nofollow noreferrer">QoS classes</a> and <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/#priorityclass" rel="nofollow noreferrer">priority classes</a>- check this article -&gt; <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/#pod-selection-for-kubelet-eviction" rel="nofollow noreferrer">Node-pressure Eviction - Pod selection for kubelet eviction</a> for more information.</p>
<p>You should try to monitor how exactly the disk pressure is happening on your node and you can adjust the kubelet configuration accordingly. Also check these articles:</p>
<ul>
<li><a href="https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/" rel="nofollow noreferrer">Node-pressure Eviction</a>.</li>
<li><a href="https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/" rel="nofollow noreferrer">Parameters to configure in kubelet</a></li>
</ul>
