<p>We are running spring-boot microservices on k8s on Amazon EC2, using undertow as our embedded web server. </p>

<p>Whenever - for whatever reason - our downstream services are overwhelmed by incoming requests, and the downstream pods' worker queue grows too large (i've seen this issue happen at 400-ish), then spring-boot stops processing queued requests completely and the app goes silent.</p>

<p>Monitoring the queue size via JMX we can see that the queue size continues to grow as more requests are queued by the IO worker - but by this point no queued requests are ever processed by any worker threads.</p>

<p>We can't see any log output or anything to indicate why this might be happening. </p>

<p>This issue cascades upstream, whereby the paralyzed downstream pods cause the traffic in the upstream pods to experience the same issue and they too become unresponsive - even when we turn off all incoming traffic through the API gateway.</p>

<p>To resolve the issue we have to stop incoming traffic upstream, and then kill all of the affected pods, before bringing them back up in greater numbers and turning the traffic back on.</p>

<p>Does anyone have any ideas about this? 
Is it expected behaviour? 
If so, how can we make undertow refuse connections before the queue size grows too large and kills the service?
If not, whhat is causing this behaviour?</p>

<p>Many thanks.
Aaron.</p>

<p>I am not entirely sure if tweaking spring boot version / embedded web server will fix this, but below is how you can scale this up using Kubernetes / Istio .</p>

<ul>
<li><strong>livenessProbe</strong></li>
</ul>

<p>If livenessProbe is configured correctly then Kubernetes restarts pods if they aren't alive. <a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#define-a-liveness-http-request" rel="nofollow noreferrer">https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#define-a-liveness-http-request</a></p>

<ul>
<li><strong>Horizontal Pod Autoscaller</strong></li>
</ul>

<p>Increases/Decreases the number of replicas of the pods based on CPU utilization or custom metrics. <a href="https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/" rel="nofollow noreferrer">https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/</a></p>

<ul>
<li><strong>Vertical Pod Autoscaller</strong></li>
</ul>

<p>Increase/Decrease the CPU / RAM of the POD based on the load. <a href="https://cloud.google.com/kubernetes-engine/docs/concepts/verticalpodautoscaler" rel="nofollow noreferrer">https://cloud.google.com/kubernetes-engine/docs/concepts/verticalpodautoscaler</a></p>

<ul>
<li><strong>Cluster Autoscaller</strong></li>
</ul>

<p>Increase/Decrease the number of nodes in the cluster based on load. <a href="https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler" rel="nofollow noreferrer">https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler</a></p>

<ul>
<li><strong>Istio Rate limiting &amp; Retry mechanism</strong></li>
</ul>

<p>Limit the number of requests that the service will receive &amp; have a retry mechanism for the requests which couldn't get executed
<a href="https://istio.io/docs/tasks/traffic-management/request-timeouts/" rel="nofollow noreferrer">https://istio.io/docs/tasks/traffic-management/request-timeouts/</a>
<a href="https://istio.io/docs/concepts/traffic-management/#network-resilience-and-testing" rel="nofollow noreferrer">https://istio.io/docs/concepts/traffic-management/#network-resilience-and-testing</a></p>
