<p>We are shutting down the AKS Cluster for our Non-Production Environments during night-time to save cost.
For the shutdown we first scale the user nodepool to 0 and then shut down the cluster using the following PowerShell Commands:</p>
<pre><code>Update-AzAksNodePool -Name $NodePoolName -ClusterName $AksClusterName -ResourceGroupName $ResourceGroupName -NodeCount 0
Stop-AzAksCluster -Name $AksClusterName -ResourceGroupName $ResourceGroupName
</code></pre>
<p>Startup is done in the opposite order</p>
<pre><code>Start-AzAksCluster -Name $AksClusterName -ResourceGroupName $ResourceGroupName
Update-AzAksNodePool -Name $NodePoolName -ClusterName $AksClusterName -ResourceGroupName $ResourceGroupName -NodeCount 1
</code></pre>
<p>When the cluster is started again, new nodes are assigned to it and i see some pods are stuck in the state Terminating on Nodes that are not existing anymore.</p>
<p>What's even crazier is that some pods of the deployment are in state &quot;Running&quot; but on a node that does not exist anymore. So for K8s the deployment seems to be satisfied but in reality there is no pod running from the deployment.</p>
<pre><code>server-67d4fc5bd5-6n7pp             1/1     Terminating   0          47h     10.244.0.25   aks-ocpp-41161936-vmss00000t        &lt;none&gt;           &lt;none&gt;
server-67d4fc5bd5-982jf             1/1     Terminating   0          4d20h   10.244.0.33   aks-ocpp-41161936-vmss00000q        &lt;none&gt;           &lt;none&gt;
server-67d4fc5bd5-fx8g6             1/1     Terminating   0          47h     10.244.0.24   aks-ocpp-41161936-vmss00000t        &lt;none&gt;           &lt;none&gt;
server-67d4fc5bd5-ptmzt             1/1     Terminating   0          4d20h   10.244.0.34   aks-ocpp-41161936-vmss00000q        &lt;none&gt;           &lt;none&gt;
server-68b79b75dd-kn5cx             1/1     Terminating   0          6d14h   10.244.0.26   aks-ocpp-41161936-vmss00000o        &lt;none&gt;           &lt;none&gt;
server-68b79b75dd-nnjwb             1/1     Terminating   0          6d14h   10.244.0.27   aks-ocpp-41161936-vmss00000o        &lt;none&gt;           &lt;none&gt;
server-b699b9cc7-f64lc              1/1     Running       0          21h     10.244.0.11   aks-ocpp-41161936-vmss00000u        &lt;none&gt;           &lt;none&gt;
server-b699b9cc7-hhngk              1/1     Running       0          21h     10.244.0.10   aks-ocpp-41161936-vmss00000u        &lt;none&gt;           &lt;none&gt;
</code></pre>
<p>When i output the current nodes i see that the nodes mentioned above where nodes seem to be still terminating are not existing anymore:</p>
<pre><code>NAME                                STATUS   ROLES   AGE    VERSION
aks-agentpool-36225625-vmss00001o   Ready    agent   107m   v1.26.0
aks-agentpool-36225625-vmss00001p   Ready    agent   105m   v1.26.0
aks-ocpp-41161936-vmss00000v        Ready    agent   107m   v1.26.0
</code></pre>
<p>I know i can kill the pods with <code>kubectl delete pod &lt;PODNAME&gt; --grace-period=0 --force --namespace &lt;NAMESPACE&gt;</code> but i would like to find out the core issue.
An ordinary kubectl delete pod will wait forever (probably as the pod does not run anymore), i really have to apply the additional flags to force-delete it.</p>
<p>When i delete the pods with the cluster up and running they don't have any issue and just restart as expected.</p>
<p>Is this a problem related to the implementation of the services running in the pod? Is the service failing to react on some shutdown-command?
Or is there a problem in the way we shut down our cluster?</p>

Markus S.
<p>It seems pods were not gracefully terminated before the nodes are deleted. As per your comment, same thing is happening while taking AKS cluster down. It can happen if the pods are not given enough time to gracefully terminate before the nodes are deleted.</p>
<p>To prevent this issue happening in future, you can perform cordon and drain on all nodes in the node pool before deleting the nodes or taking AKS cluster down using the below command.</p>
<pre><code>kubectl drain &lt;node-name&gt; --ignore-daemonsets
</code></pre>
<p><img src="https://i.imgur.com/bHZmSWy.png" alt="enter image description here" />
<img src="https://i.imgur.com/jUB8VDL.png" alt="enter image description here" /></p>
<p>Once you have applied <strong>kubectl drain</strong> command, then <strong>you can down the AKS cluster</strong>.</p>
<pre><code>Stop-AzAksCluster -Name cluster-name -ResourceGroupName rg-name
</code></pre>
<p>You can also use <code>terminationGracePeriodSeconds</code> field in the <strong>pod spec</strong> to ensure that pods are terminated quickly when the node is deleted. This will ensure that pods are not stuck in the running state for a long time. Refer to this link <a href="https://livebook.manning.com/concept/kubernetes/terminationgraceperiodsecond" rel="nofollow noreferrer">terminationgraceperiodsecond | Kubernetes | Manning Publications Co</a> for more information.</p>
