<p>I have deployed my Kubernetes cluster on EKS. I have an ingress-nginx which is exposed via load balancer to route traffic to different services. In ingress-nginx first request goes to auth service for authentication and if it is a valid request then I allow it to move forward. This is done using ingress-nginx annotation <strong>nginx.ingress.kubernetes.io/auth-url</strong>.
Auth service is developed using FastAPI. In case of <strong>401</strong> response from fastAPI look like this
<a href="https://i.stack.imgur.com/uI2Za.png" rel="nofollow noreferrer">FASTAPI</a></p>
<p>But when I use ingress-nginx the response look like this
<a href="https://i.stack.imgur.com/wt5UA.png" rel="nofollow noreferrer">INGRESS_NGINX</a></p>
<p>Is there a way to get JSON respone from Ingress-nginx?
Ingress File</p>
<pre><code>apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: ingress-service
  annotations:
    kubernetes.io/ingress.class: 'nginx'
    nginx.ingress.kubernetes.io/use-regex: 'true'
    nginx.ingress.kubernetes.io/rewrite-target: /$1
    nginx.ingress.kubernetes.io/auth-response-headers: item_id
    nginx.ingress.kubernetes.io/auth-method: POST
    nginx.ingress.kubernetes.io/auth-url: http://pth-auth.default.svc.cluster.local:8000/item/1
    # UPDATE THIS LINE ABOVE
spec:
  rules:
    - http:
        paths:
          - path: /?(.*)
            # UPDATE THIS LINE ABOVE
            backend:
              serviceName: client-cluster-ip-service
              servicePort: 3000
          - path: /api/?(.*)
            # UPDATE THIS LINE ABOVE
            backend:
              serviceName: server-cluster-ip-service
              servicePort: 5000
          - path: /pth-auth/?(.*)
            # UPDATE THIS LINE ABOVE
            backend:
              serviceName: pth-auth
              servicePort: 8000
</code></pre>

<p>Here's a solution that worked for me. It allows the auth service to return a custom error message for each request.</p>
<p>The caveat is that because nginx can't access auth response body, the <code>pth-auth</code> service needs to put the data in <code>Pth-Auth-Error</code> header (base64-encoded).</p>
<p>This example handles 401, 500, and a special case when <code>pth-auth</code> service is unavailable.</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: ingress-service
  annotations:
    kubernetes.io/ingress.class: 'nginx'
    nginx.ingress.kubernetes.io/use-regex: 'true'
    nginx.ingress.kubernetes.io/rewrite-target: /$1
    nginx.ingress.kubernetes.io/auth-response-headers: item_id
    nginx.ingress.kubernetes.io/auth-method: POST
    nginx.ingress.kubernetes.io/auth-url: http://pth-auth.default.svc.cluster.local:8000/item/1
    # UPDATE THIS LINE ABOVE
    nginx.ingress.kubernetes.io/configuration-snippet: |
      # Redirect auth errors to custom named locations
      error_page 401 = @ingress_service_custom_error_401;
      error_page 500 = @ingress_service_custom_error_500;

      # Grab data from auth error response
      auth_request_set $pth_auth_error $upstream_http_pth_auth_error;
      auth_request_set $pth_auth_error_content_type $upstream_http_content_type;
      auth_request_set $pth_auth_status $upstream_status;

    nginx.ingress.kubernetes.io/server-snippet: |
      location @ingress_service_custom_error_401 {
        internal;

        # Decode auth response header
        set_decode_base64 $pth_auth_error_decoded $pth_auth_error;

        # Return the error from pth-auth service if any
        if ($pth_auth_error_decoded != &quot;&quot;){
          add_header Content-Type $pth_auth_error_content_type always;
          return 401 $pth_auth_error_decoded;
        }

        # Fall back to default nginx response
        return 401;
      }

      location @ingress_service_custom_error_500 {
        internal;

        # Decode auth response header
        set_decode_base64 $pth_auth_error_decoded $pth_auth_error;

        # Return the error from pth-auth service if any
        if ($pth_auth_error_decoded != &quot;&quot;){
          add_header Content-Type $pth_auth_error_content_type always;
          return 500 $pth_auth_error_decoded;
        }

        # Return a hardcoded error in case no pth-auth pods are available
        if ($pth_auth_status = 503){
          add_header Content-Type application/json always;
          return 503 &quot;{\&quot;msg\&quot;:\&quot;pth-auth service is unavailable\&quot;}&quot;;
        }

        # Fall back to default nginx response
        return 500;
      }
spec:
  rules:
    - http:
        paths:
          - path: /?(.*)
            # UPDATE THIS LINE ABOVE
            backend:
              serviceName: client-cluster-ip-service
              servicePort: 3000
          - path: /api/?(.*)
            # UPDATE THIS LINE ABOVE
            backend:
              serviceName: server-cluster-ip-service
              servicePort: 5000
          - path: /pth-auth/?(.*)
            # UPDATE THIS LINE ABOVE
            backend:
              serviceName: pth-auth
              servicePort: 8000
</code></pre>
<p>Inspired by: <a href="https://stackoverflow.com/a/31485557/99237">https://stackoverflow.com/a/31485557/99237</a></p>
<h2>Troubleshooting tips:</h2>
<ul>
<li><a href="https://github.com/kubernetes/ingress-nginx/blob/main/rootfs/etc/nginx/template/nginx.tmpl" rel="nofollow noreferrer">Here's the template nginx ingress uses when transforming the ingress annotations into nginx config file.</a></li>
<li>Connect to the ingress controller pod and look at <code>/etc/nginx/nginx.conf</code> to view the generated nginx config.</li>
</ul>
