<p>We have a bunch of pods that use RabbitMQ. If the pods are shut down by K8S with SIGTERM, we have found that our RMQ client (Python Pika) has no time to close the connection to RMQ Server causing it to think those clients are still alive until 2 heartbeats are missed.</p>
<p>Our investigation has turned up that on SIGTERM, K8S kills all in- and most importantly OUTbound TCP connections, among other things (removing endpoints, etc.) Tried to see if any connections were still possible during preStop hooks, but preStop seems very internally focused and no traffic got out.</p>
<p>Has anybody else experienced this issue and solved it? All we need to do is be able to get a message out the door before kubelet slams the door. Our pods are not K8S &quot;Services&quot; so some <a href="https://stackoverflow.com/questions/62567844/kubernetes-graceful-shutdown-continue-to-serve-traffic-during-termination">suggestions</a> didn't help.</p>
<p>Steps to reproduce:</p>
<ol>
<li>add preStop hook sleep 30s to Sender pod</li>
<li>tail logs of Receiver pod to see inbound requests</li>
<li>enter Sender container's shell &amp; loop curl Receiver - requests appear in the logs</li>
<li><code>k delete pod</code> to start termination of Sender pod</li>
<li>curl requests immediately begin to hang in Sender, nothing in the Receiver logs</li>
</ol>

<p>TERM kills nothing, it's up to your application to decide how to handle it. SIGKILL is sent some time later which does forcibly nuke the process but 1) it also closes all sockets which RMQ can detect and 2) you control how long the container has to close cleanly via terminationGracePeriodSeconds</p>
