<p>I am hosting my services on azure cloud, sometimes I get "BackendConnectionFailure" without any apparent reason, after investigation I found a correlation between this exception and autoscale (scaling down) almost at the same second in most of the cases.</p>

<p>According to documentation termination grace period by default is 30 seconds, which is the case. The pod will be marked terminating and the loadbalancer will not consider it anymore, so receiving no more requests. According to this if my service takes far less time than 30 seconds, I should not need prestop hook or any special implementation in my application (please correct me if I am wrong).</p>

<p>If the previous paragraph is correct, why does this exception occur relatively frequent? My thought is when the pod is marked terminating and the loadbalancer does not forward anymore requests to the pod while it should do.</p>

<p>Edit 1:</p>

<p>The Architecture is simply like this</p>

<p>Client -> Firewall(azure) -> API(azure APIM) -> Microservices(Spring boot) -> backend(third party) or azure RDB depending on the service</p>

<p>I think the Exception comes from APIM, I found two patterns for this exception:</p>

<ol>
<li><p><code>Message   The underlying connection was closed: The connection was closed unexpectedly.
Exception type  BackendConnectionFailure 
Failed method   forward-request</code></p></li>
</ol>

<p><code>Response time  10.0 s</code></p>

<ol start="2">
<li><p><code>Message    The underlying connection was closed: A connection that was expected to be kept alive was closed by the server.
Exception type  BackendConnectionFailure
Failed method   forward-request</code></p></li>
</ol>

<p><code>Response time  3.6 ms</code></p>

<p>Spring Boot <a href="https://github.com/spring-projects/spring-boot/issues/4657" rel="nofollow noreferrer">doesn't do graceful termination</a> by default. </p>

<p>The Spring Boot app and it's application container (not linux container) are in control of what happens to existing connections during the termination grace period. The protocols being used and how a client reacts to a "close" also have a part to play.</p>

<p>If you get to the end of the grace period, then everything gets a hard reset. </p>

<h3>Kubernetes</h3>

<p><a href="https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods" rel="nofollow noreferrer">When a pod is deleted in k8s</a>, the Pod Endpoint removal from Services is triggered at the same time as the <code>SIGTERM</code> signal to the container(s).</p>

<p>At this point the cluster nodes will be reconfigured to remove any rules directing <strong><em>new</em></strong> traffic to the Pod. Any existing TCP connections to the Pod/containers will remain in connection tracking until they are closed (by the client, server or network stack).</p>

<p>For HTTP Keep Alive or HTTP/2 services, the client will continue hitting the same Pod Endpoint until it is told to close the connection (or it is forcibly reset)</p>

<h3>App</h3>

<p>The basic rules are, on SIGTERM the application should:</p>

<ul>
<li>Allow running transactions to complete</li>
<li>Do any application cleanup required</li>
<li>Stop accepting new connections, just in case</li>
<li>Close any inactive connections it can (keep alive requests, websockets)</li>
</ul>

<p>Some circumstances you might not be able to handle (depends on the client)</p>

<ul>
<li>A keep alive connection that doesn't complete a request in the grace period, can't get a <code>Connection: close</code> header. It will need a TCP level FIN close. </li>
<li>A slow client with a long transfer, in a one way HTTP transfer these will have to be waited for or forcibly closed.</li>
</ul>

<p>Although keepalive clients should respect a TCP FIN close, every client reacts differently. Microsoft APIM might be sensitive and produce the error even though there was no real world impact. It's best to load test your setup while scaling to see if there is a real world impact. </p>

<p>For more spring boot info see:</p>

<p><a href="https://github.com/spring-projects/spring-boot/issues/4657" rel="nofollow noreferrer">https://github.com/spring-projects/spring-boot/issues/4657</a>
<a href="https://github.com/corentin59/spring-boot-graceful-shutdown" rel="nofollow noreferrer">https://github.com/corentin59/spring-boot-graceful-shutdown</a>
<a href="https://github.com/SchweizerischeBundesbahnen/springboot-graceful-shutdown" rel="nofollow noreferrer">https://github.com/SchweizerischeBundesbahnen/springboot-graceful-shutdown</a></p>
