<p>I am using Spring Boot with Embedded Tomcat 9.0.36. It is used as a Docker image in Kubernetes. Recently after upgrading envoy, I started getting exceptions.</p>

<pre><code>  "upstream connect error or disconnect/reset before headers. reset reason: connection termination" with 503 status code
</code></pre>

<p>Some people suggested increasing idle connection Time out to 60 seconds but it spring-boot I was able to find out "Connection Time Out" &amp; "Keep-Alive Time Out". I increased them to 5 minutes using the below code.</p>

<p><div class="snippet" data-lang="js" data-hide="false" data-console="true" data-babel="false">
<div class="snippet-code">
<pre class="snippet-code-html lang-html prettyprint-override"><code>@Configuration
public class TomcatCustomizer implements WebServerFactoryCustomizer&lt;TomcatServletWebServerFactory&gt; {

    private static final Logger LOGGER = LoggerFactory.getLogger(TomcatCustomizer.class);

    @Override
    public void customize(TomcatServletWebServerFactory factory) {

        factory.addConnectorCustomizers(connector -&gt; {
            AbstractHttp11Protocol protocol = (AbstractHttp11Protocol) connector.getProtocolHandler();
            //Setting up connection time out
            protocol.setKeepAliveTimeout(360000);
            protocol.setConnectionTimeout(360000);
            protocol.setMaxKeepAliveRequests(120);
        });
    }
}</code></pre>
</div>
</div>
</p>

<p>Still, I am getting the same error. This application calls another service internally which is also hosted in Kubernetes. I am able to see a successful response in my service but after that, I don't see any logs.</p>

<p>I spent a week analyzing this from the application point of view. I followed a few steps that were suggested by the Ops Team.</p>

<ul>
<li>Increase Timeout in Tomcat Server to 60 seconds because they have configured the same in Envoy</li>
<li>I did increase the time but not able to solve the issue.</li>
<li>I was using Spring Cloud Gateway for Gateway service I thought that is the issue so I changed it into Rest Templates but that also didn't solve the issue.</li>
<li>Luckily Health Check APIs working fine except those which were communicating to other services internally. In Health APIs they were also communicating with other services to check their Health but I was not returning response directly. I was wrapping up the response body modifying it and den forwarding it to UI. I also applied the same and use the below code which you can understand easily.I created a new Response Entity and dropped all headers which I received from internal APIs and returned to UI. It worked like charm. </li>
</ul>

<p><div class="snippet" data-lang="js" data-hide="false" data-console="true" data-babel="false">
<div class="snippet-code">
<pre class="snippet-code-html lang-html prettyprint-override"><code>//Earlier (Forwarding same headers received from internal service to UI)
ResponseEntity responseEntity = //Received by calling other APIs;
return responseEntity;

//Now (Dropped headers)
ResponseEntity responseEntity = //Received by calling other APIs;
MultiValueMap&lt;String, String&gt; newHeaders = new LinkedMultiValueMap&lt;&gt;();          
if (Objects.nonNull(responseEntity) &amp;&amp; Objects.nonNull(responseEntity.getBody())) {
    newHeaders.set("Content-type", responseEntity.getHeaders().getContentType().toString());
    return new ResponseEntity(responseEntity.getBody(), newHeaders, responseEntity.getStatusCode());
}</code></pre>
</div>
</div>
</p>
