<p>I have two API that are publicly exposed, let's say xyz.com/apiA and xyz.com/apiB.</p>
<p>Both these API are running node and are dockerized services running as individual pods in the same namespace of a Kubernetes cluster.</p>
<p>Now, apiA calls apiB internally as part of its code logic. The apiA service makes a POST call to apiB and sends with it a somewhat large payload in its body parameter. This POST request times out if the payload in its body is more than 30kb.</p>
<p>We have checked the server logs and that POST request is not seen.</p>
<p>The error prompt shows <em>connection timeout to 20.xx.xx.xx</em> which is the public ip address of xyz.com</p>
<p>I'm new to Kubernetes and would appreciate your help.</p>
<p>So far have tried <a href="https://kubernetes.io/blog/2019/03/29/kube-proxy-subtleties-debugging-an-intermittent-connection-reset/" rel="nofollow noreferrer">this</a>, but it didn't help.</p>
<p>Please let me know if more information is needed.</p>
<p>Edit: kubectl client and server version is 1.22.0</p>

ams
<p>To update the kind folks who took time to understand the problem and suggest solutions - The issue was due to bad routing. Internal APIs (apiB in the example above) should not be called using full domain name of xyz.com/apiB, rather they can be directly referenced using pod name as</p>
<p><code>http://pod_name.namespace.svc.local/apiB</code>.</p>
<p>This will ensure internal calls are routed through Kubernetes DNS and don't have to go through load balancer and nginx, thus improving response time heavily.</p>
<p>Every call made to apiA was creating a domino effect by populating hundreds of calls to apiB and overloading the server, which caused it to fail only after a few thousand requests.</p>
<p>Lesson learned: <em>Route all internal calls using cluster's internal network</em>.</p>
