<p>We have secured some of our services on the K8S cluster using the approach described on <a href="https://kubernetes.github.io/ingress-nginx/examples/auth/oauth-external-auth/" rel="nofollow noreferrer">this page</a>. Concretely, we have:</p>

<pre><code>  nginx.ingress.kubernetes.io/auth-url: "https://oauth2.${var.hosted_zone}/oauth2/auth"
  nginx.ingress.kubernetes.io/auth-signin: "https://oauth2.${var.hosted_zone}/oauth2/start?rd=/redirect/$http_host$escaped_request_uri"
</code></pre>

<p>set on the service to be secured and we have followed <a href="https://www.callumpember.com/Kubernetes-A-Single-OAuth2-Proxy-For-Multiple-Ingresses/" rel="nofollow noreferrer">this tutorial</a> to only have one deployment of oauth2_proxy per cluster. We have 2 proxies set up, both with affinity to be placed on the same node as the nginx ingress. </p>

<pre><code>$ kubectl get pods -o wide -A | egrep "nginx|oauth"                                                                    
infra-system   wer-exp-nginx-ingress-exp-controller-696f5fbd8c-bm5ld        1/1     Running   0          3h24m   10.76.11.65    ip-10-76-9-52.eu-central-1.compute.internal     &lt;none&gt;           &lt;none&gt;
infra-system   wer-exp-nginx-ingress-exp-controller-696f5fbd8c-ldwb8        1/1     Running   0          3h24m   10.76.14.42    ip-10-76-15-164.eu-central-1.compute.internal   &lt;none&gt;           &lt;none&gt;
infra-system   wer-exp-nginx-ingress-exp-default-backend-7d69cc6868-wttss   1/1     Running   0          3h24m   10.76.15.52    ip-10-76-15-164.eu-central-1.compute.internal   &lt;none&gt;           &lt;none&gt;
infra-system   wer-exp-nginx-ingress-exp-default-backend-7d69cc6868-z998v   1/1     Running   0          3h24m   10.76.11.213   ip-10-76-9-52.eu-central-1.compute.internal     &lt;none&gt;           &lt;none&gt;
infra-system   oauth2-proxy-68bf786866-vcdns                                 2/2     Running   0          14s     10.76.10.106   ip-10-76-9-52.eu-central-1.compute.internal     &lt;none&gt;           &lt;none&gt;
infra-system   oauth2-proxy-68bf786866-wx62c                                 2/2     Running   0          14s     10.76.12.107   ip-10-76-15-164.eu-central-1.compute.internal   &lt;none&gt;           &lt;none&gt;
</code></pre>

<p>However, a simple website load usually takes around 10 seconds, compared to 2-3 seconds with the proxy annotations not being present on the secured service. </p>

<p>We added a <code>proxy_cache</code> to the <code>auth.domain.com</code> service which hosts our proxy by adding</p>

<pre><code>        "nginx.ingress.kubernetes.io/server-snippet": &lt;&lt;EOF
          proxy_cache auth_cache;
          proxy_cache_lock on;
          proxy_ignore_headers Cache-Control;
          proxy_cache_valid any 30m;
          add_header X-Cache-Status $upstream_cache_status;
        EOF
</code></pre>

<p>but this didn't improve the latency either. We still see all HTTP requests triggering a log line in our proxy. Oddly, only some of the requests take 5 seconds. 
<a href="https://i.stack.imgur.com/IcJZO.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/IcJZO.png" alt="enter image description here"></a></p>

<p>We are unsure if:
- the proxy forwards each request to the oauth provider (github) or
- caches the authentications</p>

<p>We use cookie authentication, therefore, in theory, the oauth2_proxy <em>should</em> just decrypt the cookie and then return a 200 to the nginx ingress. Since they are both on the same node it <em>should</em> be fast. But it's not. Any ideas?  </p>

<h2>Edit 1</h2>

<p>I have analyzed the situation further. Visiting my auth server with <code>https://oauth2.domain.com/auth</code> in the browser and copying the request <code>copy for curl</code> I found that:</p>

<ol>
<li>running 10.000 queries against my oauth server from my local machine (via curl) is very fast</li>
<li>running 100 requests on the nginx ingress with the same curl is slow</li>
<li>replacing the host name in the curl with the cluster IP of the auth service makes the performance increase drastically</li>
<li>setting the annotation to <code>nginx.ingress.kubernetes.io/auth-url: http://172.20.95.17/oauth2/auth</code> (e.g. setting the host == cluster IP) makes the GUI load as expected (fast)</li>
<li>it doesn't matter if the curl is run on the nginx-ingress or on any other pod (e.g. a test debian), the result is the same</li>
</ol>

<h2>Edit 2</h2>

<p>A better fix I found was to set the annotation to the following</p>

<pre><code>  nginx.ingress.kubernetes.io/auth-url: "http://oauth2.infra-system.svc.cluster.local/oauth2/auth"
  nginx.ingress.kubernetes.io/auth-signin: "https://oauth2.domain.com/oauth2/start?rd=/redirect/$http_host$escaped_request_uri"
</code></pre>

<p>The <code>auth-url</code> is what the ingress queries with the cookie of the user. Hence, a local DNS of the oauth2 service is the same as the external dns name, but without the SSL communication and since it's DNS, it's permanent (while the cluster IP is not)</p>

<p>Given that it's unlikely that someone comes up with the <em>why</em> this happens, I'll answer my workaround.</p>

<p>A fix I found was to set the annotation to the following</p>

<pre><code>  nginx.ingress.kubernetes.io/auth-url: "http://oauth2.infra-system.svc.cluster.local/oauth2/auth"
  nginx.ingress.kubernetes.io/auth-signin: "https://oauth2.domain.com/oauth2/start?rd=/redirect/$http_host$escaped_request_uri"
</code></pre>

<p>The <code>auth-url</code> is what the ingress queries with the cookie of the user. Hence, a local DNS of the oauth2 service is the same as the external dns name, but without the SSL communication and since it's DNS, it's permanent (while the cluster IP is not)</p>
