<h2>The problem</h2>

<p>I'm getting 403 SSL required from Spring when trying to route through my ELB to Kubernetes Nginx ingress controller. </p>

<h2>Setup</h2>

<p>My set up is as follows:</p>

<ol>
<li>I've got an ELB (AWS) with ACM for my Kubernetes cluster (created by kops) which routes all requests to the</li>
<li>Nginx Ingress Controller which in turn routes all requests according to the rules dictated in the</li>
<li>Ingress that passes the traffic unto the</li>
<li>Service that exposes port 80 and routes in to port 8080 in the </li>
<li>Pods selected with labels "app=foobar" (which are described in a Deployment)</li>
<li>Pods are running a Spring Boot Web App v2.1.3</li>
</ol>

<p>So basically:
<code>https://foo.bar.com(:443) -&gt; ingress -&gt; http://foo.bar.svc.cluster.local:80</code></p>

<p><strong>This works like a charm for everything.</strong> Except SprintBoot.
For some reason, I keep getting <code>403 - SSL required</code> from Spring</p>

<p><strong>One note to keep in mind here:</strong>  my Spring application does not have anything to do with SSL. I don't want it to do anything in that nature. For this example's purposes, this should be a regular REST API requests, with the SSL termination happening outside the container.</p>

<h2>What I tried so far</h2>

<ul>
<li>Port-forwarding to the service itself and requesting - it works fine.</li>
<li>Disabling CSRF in <code>WebSecurityConfigurerAdapter</code></li>
<li>Putting ingress annotation <code>nginx.ingress.kubernetes.io/force-ssl-redirect=true</code> - it gives out TOO_MANY_REDIRECTS error when I try it (instead of the 403)</li>
<li>Putting ingress annotation <code>nginx.ingress.kubernetes.io/ssl-redirect=true</code> - doesn't do anything</li>
<li>Putting ingress annotation <code>nginx.ingress.kubernetes.io/enable-cors: "true"</code> - doesn't do anything</li>
<li>Also <code>nginx.ingress.kubernetes.io/ssl-passthrough: "true"</code></li>
<li>Also <code>nginx.ingress.kubernetes.io/secure-backends: "true"</code></li>
<li>Also <code>kubernetes.io/tls-acme: "true"</code></li>
<li>I tried a whole bunch of other stuff that I can't really remember right now</li>
</ul>

<h2>How it all looks like in my cluster</h2>

<p>Nginx ingress controller annotations look like this (I'm using the official nginx ingress controller helm chart, with very little modifications other than this thing):</p>

<pre><code>service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "aws_acm_certificate_arn"
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"
</code></pre>

<p>Ingress looks like this:</p>

<pre><code>apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: foobar
  namespace: api
spec:
  rules:
  - host: foo.bar.com
    http:
      paths:
      - backend:
          serviceName: foobar
          servicePort: http
        path: /
</code></pre>

<p>Service looks like this:</p>

<pre><code>apiVersion: v1
kind: Service
metadata:
  name: foobar
  namespace: api
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 8080
  selector:
    app: foobar
</code></pre>

<h2>What I think the problem is</h2>

<p>My hunch is that it's something with X-Forwarded headers, and Spring doing its magic behind the scenes, trying to be all smart like and deciding that I need SSL based on some headers without me explicitly asking for it. But I didn't figure it out yet.</p>

<hr>

<p>I searched far and wide for a solution, but I couldn't find anything to ease my pain... hope you'll be able to help!</p>

<h2>Edit</h2>

<p>I found out that my current setup (without k8s and nginx) works fine and ELB passes <code>X-Forwarded-Port: 443</code> and <code>X-Forwarded-Proto: https</code>, and it seems to work, but on my k8s cluster with nginx, I put in a listener client that spits out all the headers, and my headers seem to be <code>X-Forwarded-Port: 80</code> and <code>X-Forwarded-Proto: http</code></p>

Dor Munis
<p>Thanks for all the people that helped out, I actually found the answer.
Within the code there were some validations that all requests should come from a secure source, and Nginx Ingress Controller changed these headers (X-Forwarded-Proto and X-Forwarded-Port) because SSL was terminated within ELB and handed to the ingress controller as HTTP</p>

<p>To fix that I did the following:</p>

<ol>
<li><p>Added <code>use-proxy-protocol: true</code> to the config map - which passed the correct headers, but got errors regarding broken connection (which I don't really remember the actual error right now, I'll edit this answer later if there will be any requests for it)</p></li>
<li><p>To fix these errors I added the following the the nginx ingress controller annotations configuration:</p></li>
</ol>

<pre><code>service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "tcp"
</code></pre>

<p>This made sure that all traffic will use the proxy protocol, and I also had to change the <code>backend-protocol</code> from <code>HTTP</code> to <code>TCP</code>.</p>

<p>Doing this made sure all requests routed ELB reserve their original X headers, and are passed unto Nginx Ingress Controller, which in turn passed unto my applications that require these headers to be passed.</p>
