<p>I have an ingress providing routing for two microservices running on GKE, and intermittently when the microservice returns a 404/422, the ingress returns a 502.</p>

<p>Here is my ingress definition:</p>

<pre><code>apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: basic-ingress
  annotations:
    kubernetes.io/ingress.global-static-ip-name: develop-static-ip
    ingress.gcp.kubernetes.io/pre-shared-cert: dev-ssl-cert
spec:
  rules:
  - http:
      paths:
      - path: /*
        backend:
          serviceName: srv
          servicePort: 80
      - path: /c/*
        backend:
          serviceName: collection
          servicePort: 80
      - path: /w/*
        backend:
          serviceName: collection
          servicePort: 80
</code></pre>

<p>I run tests that hit the <code>srv</code> back-end where I expect a 404 or 422 response. I have verified when I hit the <code>srv</code> back-end directly (bypassing the ingress) that the service responds correctly with the 404/422.</p>

<p>When I issue the same requests through the ingress, the ingress will intermittently respond with a 502 instead of the 404/422 coming from the back-end.</p>

<p>How can I have the ingress just return the 404/422 response from the back-end?</p>

<p>Here's some example code to demonstrate the behavior I'm seeing (the expected status is 404):</p>

<pre class="lang-py prettyprint-override"><code>&gt;&gt;&gt; for i in range(10):
        resp = requests.get('https://&lt;server&gt;/a/v0.11/accounts/junk', cookies=&lt;token&gt;)
        print(resp.status_code)

502
502
404
502
502
404
404
502
404
404
</code></pre>

<p>And here's the same requests issued from a python prompt within the pod, i.e. bypassing the ingress:</p>

<pre class="lang-py prettyprint-override"><code>&gt;&gt;&gt; for i in range(10):
...     resp = requests.get('http://0.0.0.0/a/v0.11/accounts/junk', cookies=&lt;token&gt;)
...     print(resp.status_code)
...
404
404
404
404
404
404
404
404
404
404
</code></pre>

<p>Here's the output of the kubectl commands to demonstrate that the loadbalancer is set up correctly (I never get a 502 for a 2xx/3xx response from the microservice):</p>

<pre class="lang-sh prettyprint-override"><code>$ kubectl get pods -o wide
NAME                          READY   STATUS    RESTARTS   AGE   IP          NODE                                     NOMINATED NODE   READINESS GATES
srv-799976fbcb-4dxs7          2/2     Running   0          19m   10.24.3.8   gke-develop-default-pool-ea507abc-43h7   &lt;none&gt;           &lt;none&gt;
srv-799976fbcb-5lh9m          2/2     Running   0          19m   10.24.1.7   gke-develop-default-pool-ea507abc-q0j3   &lt;none&gt;           &lt;none&gt;
srv-799976fbcb-5zvmv          2/2     Running   0          19m   10.24.2.9   gke-develop-default-pool-ea507abc-jjzg   &lt;none&gt;           &lt;none&gt;
collection-5d9f8586d8-4zngz   2/2     Running   0          19m   10.24.1.6   gke-develop-default-pool-ea507abc-q0j3   &lt;none&gt;           &lt;none&gt;
collection-5d9f8586d8-cxvgb   2/2     Running   0          19m   10.24.2.7   gke-develop-default-pool-ea507abc-jjzg   &lt;none&gt;           &lt;none&gt;
collection-5d9f8586d8-tzwjc   2/2     Running   0          19m   10.24.2.8   gke-develop-default-pool-ea507abc-jjzg   &lt;none&gt;           &lt;none&gt;
parser-7df86f57bb-9qzpn       1/1     Running   0          19m   10.24.0.8   gke-develop-parser-pool-5931b06f-6mcq    &lt;none&gt;           &lt;none&gt;
parser-7df86f57bb-g6d4q       1/1     Running   0          19m   10.24.5.5   gke-develop-parser-pool-5931b06f-9xd5    &lt;none&gt;           &lt;none&gt;
parser-7df86f57bb-jchjv       1/1     Running   0          19m   10.24.0.9   gke-develop-parser-pool-5931b06f-6mcq    &lt;none&gt;           &lt;none&gt;

$ kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)        AGE
srv          NodePort    10.0.2.110   &lt;none&gt;        80:30141/TCP   129d
collection   NodePort    10.0.4.237   &lt;none&gt;        80:30270/TCP   129d
kubernetes   ClusterIP   10.0.0.1     &lt;none&gt;        443/TCP        130d

$ kubectl get endpoints
NAME         ENDPOINTS                                AGE
srv          10.24.1.7:80,10.24.2.9:80,10.24.3.8:80   129d
collection   10.24.1.6:80,10.24.2.7:80,10.24.2.8:80   129d
kubernetes   35.237.239.186:443                       130d
</code></pre>

<p>tl;dr: GCP LoadBalancer/GKE Ingress will 502 if 404/422s from the back-ends don't have response bodies.</p>

<p>Looking at the LoadBalancer logs, I would see the following errors:</p>

<pre><code>502: backend_connection_closed_before_data_sent_to_client
404: backend_connection_closed_after_partial_response_sent
</code></pre>

<p>Since everything was configured correctly (even the LoadBalancer said the backends were healthy)--backend was working as expected and no failed health checks--I experimented with a few things and noticed that all of my 404 responses had empty bodies.</p>

<p>Sooo, I added a body to my 404 and 422 responses and lo and behold no more 502s!</p>
