<p>So we have been trying to get a few applications running on a GKE cluster with autopilot.</p>
<p>However when we went to test we get 5XX errors when we were expecting 4XX. This is even stranger because when we receive a 2XX response the message is received accordingly.</p>
<p>Upon reviewing application logs we see that the inteded output is the 4XX however, when the response is sent to the client, it is sent as a 5XX. What could be changing the response? Where does this response come from?</p>
<pre><code>&lt;html lang=&quot;en&quot;&gt;
&lt;head&gt;
    &lt;title&gt;Unable to forward your request to a backend - Web Forwarder - Cloud Shell&lt;/title&gt;
&lt;/head&gt;
&lt;body style=&quot;font-family: monospace;&quot;&gt;
    &lt;h1 style=&quot;font-size: 1.5em;&quot;&gt;Unable to forward your request to a backend&lt;/h1&gt;
    &lt;p&gt;Couldn&amp;#39;t connect to a server on port 8080&lt;/p&gt;
&lt;/body&gt;
</code></pre>

<p>The load balancers follow this template</p>
<pre><code>apiVersion: v1
kind: Service
metadata:
  name: app-gateway
  namespace: namespace
  annotations:
    networking.gke.io/load-balancer-type: &quot;Internal&quot;
    cloud.google.com/neg: '{&quot;ingress&quot;: true}'
spec:
  type: LoadBalancer
  externalTrafficPolicy: Cluster
  selector:
    app: app-gateway
  ports:
    - port: 80
      targetPort: 80
      name: http
</code></pre>
<p>And the ingress</p>
<pre><code>apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-gateway
  namespace: namespace
  annotations:
    kubernetes.io/ingress.class: &quot;gce-internal&quot;
spec:
  rules:
    - host: app.internal
      http:
        paths:
          - pathType: Prefix
            path: &quot;/&quot;
            backend:
              service:
                name: app-gateway
                port:
                  number: 80
</code></pre>
<p>Upon request this is the architecture for the system. Names were omitted<a href="https://i.stack.imgur.com/HtNCi.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/HtNCi.png" alt="enter image description here" /></a></p>
<p>This is a very straightforward approach, just a few workloads behind a internal load balancer, connected to an on-prem Mongo and Rabbit.</p>
<p><strong>Edit</strong> - Some more details</p>
<p>The way i'm doing is by setting up a port forward from my gcp console to the pod.</p>
<p>When I go to /swagger/index.html and try to test the API it returns 503 errors when a 4XX is expected. However a 2XX is sent sucessfully.
<a href="https://i.stack.imgur.com/uYTMJ.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/uYTMJ.png" alt="Response using swagger" /></a>
When I port forward in my own console (using the same command as in GCP console) and do a <code>curl -X -I GET localhost:8080/swagger/index.html</code> I get the correct response.
<a href="https://i.stack.imgur.com/cTzJo.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/cTzJo.png" alt="enter image description here" /></a></p>
<p>Meaning it's likely something regarding the cloud shell itself.</p>

Eddoasso
<p>We need to know how your traffic routes using both sources when testing using curl. I suggest using the Connectivity Test. In this way we can identify if both devices (source) have the same route when reaching its destination.</p>
<p>To know more about my suggestion you can check this <a href="https://cloud.google.com/network-intelligence-center/docs/connectivity-tests/concepts/overview#analyze-configurations" rel="nofollow noreferrer">link</a>. From the link you will see the information where traffic might go to a different path. From here we might see another useful information why we are having these kinds of error messages and will look at a different angle on how to resolve the issue.</p>
