<p>I have a preStop hook defined in a statefulset pod resource that runs a bash script to make sure not to kill the pod until few processes finishes/cancels/errors within an application. I don't have the terminationGracePeriodSeconds defined. Now when I delete the pod, I tested that the script that is part of preStop hook is run as expected. But after adding terminationGracePeriodSeconds for 10 min, first the bash script is run as part of preStop hook successfully for couple of minutes and it is supposed to kill the pod. But the pod is hanging in TERMINATING status and it is killed only after 10 min.</p>
<ol>
<li>Why is the pod is hanging? Unable to find an answer for this.</li>
<li>When the terminationGracePeriodSeconds was not added, the flow was working as expected by killing the pod as soon as finishing the script or within 30 sec which is the terminationGracePeriodSeconds. But when I added the grace period of 10 min or more, it is waiting until that time and then killing the pod.</li>
</ol>
<p>How to solve this issue. Is there a way to send SIGTERM or SIGKILL to the pod. Any ideas? Thank you in advance!</p>
<p>STATEFULSET.YAML</p>
<pre><code>apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    app: appx
  name: appx
spec:
  serviceName: appx
  replicas: 1
  updateStrategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app: appx
  template:
    metadata:
      labels:
        app: appx
    spec:
      #removed some of the sensitive info
      terminationGracePeriodSeconds: 600
      containers:
        - image: appx
          imagePullPolicy: IfNotPresent
          name: appx
          lifecycle:
            preStop:
              exec:
                command: [&quot;/bin/sh&quot;, &quot;-c&quot;, &quot;sleep 30 &amp;&amp; bash /var/tmp/runscript.sh; sleep10&quot;]
</code></pre>
<p>KUBECTL DESCRIBE POD</p>
<pre><code>
**kubectl describe pod appx**
Name:           appx
Namespace:      default
Priority:       0
Node:           docker-desktop/192.168.65.3
Start Time:     Mon, 21 Sep 2020 07:30:55 -0500
Labels:         app=appx
Annotations:    &lt;none&gt;
Status:         Running
IP:             x.x.x.x
Controlled By:  StatefulSet/appx
Containers:
  appx:
    Container ID:   docker://dfdgfgfgfgfgfgfg
    Image:          appx
    Image ID:       docker://sha256:49dfgfgfgfgfgfgfgfgfg96a6fc
    Port:           &lt;none&gt;
    Host Port:      &lt;none&gt;
    State:          Running
      Started:      Mon, 21 Sep 2020 07:30:56 -0500
    Ready:          True
    Restart Count:  0
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  data:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  &lt;unset&gt;
  appx-token-xj6q9:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  appx-token-fhfdlf
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  &lt;none&gt;
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age    From                     Message
  ----    ------     ----   ----                     -------
  Normal  Scheduled  2m43s  default-scheduler        Successfully assigned default/appx to docker-desktop
  Normal  Pulled     2m42s  kubelet, docker-desktop  Container image &quot;appx&quot; already present on machine
  Normal  Created    2m42s  kubelet, docker-desktop  Created container appx
  Normal  Started    2m42s  kubelet, docker-desktop  Started container appx
</code></pre>

<p><code>preStop</code> hook and  <code>terminationGracePeriodSeconds</code> are asynchronous. It means that as soon as the kubelet sees that a Pod has been marked as terminating, the kubelet begins the local Pod shutdown process.  This means that if container doesn`t terminate within the grace period, a SIGKILL singal will be sent and the container will be killed regardless of wehther the commands in the preStop hook are completed.</p>
<blockquote>
<ol start="2">
<li>When the terminationGracePeriodSeconds was not added, the flow was working as expected by killing the pod as soon as finishing the script
or within 30 sec which is the terminationGracePeriodSeconds. But when
I added the grace period of 10 min or more, it is waiting until that
time and then killing the pod.</li>
</ol>
</blockquote>
<p><code>terminationGracePeriodSeconds</code> grace period is always added. It just defaults to the 30 seconds as I already mentioned in the comments.  So, what  if <code>terminationGracePeriodSeconds</code> is less than the time to complete the preStop hook?</p>
<p><strong>Then the container will terminate at the end of the <code>terminationGracePeriodSeconds</code> and the preStop hook will not finish/run.</strong></p>
<p>When terminationGracePeriodSeconds is set to 600s, the preStop hook script is hanging (currently unclear whether it ever worked as it wasn’t properly tested with the default 30s terminationGracePeriodSeconds due to preemptive termination). It means that some processes are not handling SIGTERM correctly which is currently not corrected for in the preStop hook, meaning that the container is instead waiting for the SIGKILL to be sent after the 10 min terminationGracePeriod ends.</p>
<p>If you take a look <a href="https://github.com/kubernetes/kubernetes/issues/24695" rel="nofollow noreferrer">here</a> you will find out that even though the user specified a preStop hook, they needed to SIGTERM nginx for a graceful shutdown.</p>
<p>In the case whereas you have set <code>terminationGracePeriodSeconds</code> to 10 minutes, even though your preStop hook  executed successfully Kubernetes waited 10 minutes before terminating your container because that is exactly what you told him to do.   Termination signal is being sent by kubelet, but it is not being passed to the application insider the container. Most common reason for that is when your container runs a shell which runs the application process the signal might be consumed/interrupted by shell itself instead of passed towards the child process.  Also, since it is unclear what your <code>runscript.sh</code> is doing it is difficult to make any other suggestions to what processes are failing to handle SIGTERM.</p>
<p>What you can do in this case?  The options for ending sooner are :</p>
<ul>
<li>Decrease terminationGracePeriodSeconds</li>
<li>Send a signal for a graceful shutdown by ensuring SIGTERM is handled correctly and all running processes are listening for termination. Examples of how to do this are <a href="https://medium.com/flant-com/kubernetes-graceful-shutdown-nginx-php-fpm-d5ab266963c2" rel="nofollow noreferrer">here</a>. You can see that they use the “quit” command for NGINX.</li>
</ul>
<p>For more information about you can find great articles <a href="https://cloud.google.com/blog/products/gcp/kubernetes-best-practices-terminating-with-grace" rel="nofollow noreferrer">here</a> and <a href="https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination" rel="nofollow noreferrer">here</a>.</p>
