<p>We have a number of different REST-based services running in Azure within a Kubernetes (version 1.9.6) cluster.  </p>

<p>Two of the services, let's say A and B needs to communicate with each other using REST-calls.  Typically, something like the following:</p>

<pre><code>Client calls A (original request)
A calls B (request 1)
B calls A (request 2)
A responds to B (request 2)
B responds to A (request 1)
A responds to the original request
</code></pre>

<p>The above being a typical intertwined micro-services architecture.  Manually running the docker instances works perfectly on our local test servers.</p>

<p>The moment we run this in Kubernetes on Azure we get intermittent timeouts (60+ seconds) on the micro-services calling each other through Kubernetes' networking services.  After a timeout, repeating the request would then often give correct responses in a few micro-seconds.</p>

<p>I am stuck at this point as I have no idea what could be causing this.  Could it be the dynamic routing?  The virtualised network? Kubernetes configuration? </p>

<p>Any ideas?</p>

<p>Finally figured this out.</p>

<blockquote>
  <p>Azure Load Balancers / Public IP addresses have a default 4 minute idle connection timeout.</p>
</blockquote>

<p>Essentially anything running through a Load Balancer (whether created by an Azure AKS Kubernetes Ingress or otherwise) has to abide by this.  While you CAN change the timeout there is no way to eliminate it entirely (longest idle timeout duration possible is 30 minutes).</p>

<p>For that reason it makes sense to implement a connection pooling / monitoring solution that will track the idle time that has elapsed on each of your connections (through the load balancer / public IP) and then disconnect / re-create any connection that gets close to the 4 minute cutoff.</p>

<p>We ended up implementing PGbouncer (<a href="https://github.com/pgbouncer/pgbouncer" rel="nofollow noreferrer">https://github.com/pgbouncer/pgbouncer</a>) as an additional container on our Azure AKS / Kubernetes Cluster via the awesome directions which can be found here: <a href="https://github.com/edoburu/docker-pgbouncer/tree/master/examples/kubernetes/singleuser" rel="nofollow noreferrer">https://github.com/edoburu/docker-pgbouncer/tree/master/examples/kubernetes/singleuser</a></p>

<p>Overall I can see the need for the timeout but MAN was it hard to troubleshoot.  Hope this saves you guys some time!</p>

<p>More details can be found on my full post over here: <a href="https://stackoverflow.com/questions/50706483/what-azure-kubernetes-aks-time-out-happens-to-disconnect-connections-in-out">What Azure Kubernetes (AKS) &#39;Time-out&#39; happens to disconnect connections in/out of a Pod in my Cluster?</a></p>
