title: Applying TLS to a working ingress results in default backend - 404
body: <p>I have installed the Nginx Ingress Controller through helm in the ingress namespace.</p>
  
  <pre><code>helm ls --namespace ingress
  NAME            NAMESPACE   REVISION    UPDATED                                 STATUS      CHART                   APP VERSION
  nginx-ingress   ingress     1           2020-03-15 10:47:51.143159 +0530 IST    deployed    nginx-ingress-1.34.2    0.30.0  
  </code></pre>
  
  <p>The Service and Deployment is as follows</p>
  
  <pre><code>apiVersion: v1
  kind: Service
  metadata:
    name: test-service
    labels:
      app.kubernetes.io/name: test-service
      helm.sh/chart: test-service-0.1.0
      app.kubernetes.io/instance: test-service
      app.kubernetes.io/managed-by: Helm
    annotations:
      service.beta.kubernetes.io/azure-load-balancer-internal: '"true"'
  spec:
    type: LoadBalancer
    ports:
      - port: 8080
        targetPort: http
        protocol: TCP
        name: http
    selector:
      app.kubernetes.io/name: test-service
      app.kubernetes.io/instance: test-service
  ---
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: test-service
    labels:
      app.kubernetes.io/name: test-service
      helm.sh/chart: test-service-0.1.0
      app.kubernetes.io/instance: test-service
      app.kubernetes.io/managed-by: Helm
  spec:
    replicas: 1
    selector:
      matchLabels:
        app.kubernetes.io/name: test-service
        app.kubernetes.io/instance: test-service
    template:
      metadata:
        labels:
          app.kubernetes.io/name: test-service
          app.kubernetes.io/instance: test-service
      spec:
        containers:
          - name: test-service
            image: "&lt;acr-url&gt;/test-service:c93c58c0bd4918de06d46381a89b293087262cf9"
            imagePullPolicy: IfNotPresent
            ports:
              - name: http
                containerPort: 8080
                protocol: TCP
            livenessProbe:
              httpGet:
                path: /devops/health/liveness
                port: 8080
              initialDelaySeconds: 60
              periodSeconds: 10
              successThreshold: 1
              timeoutSeconds: 1
            readinessProbe:
              httpGet:
                path: /devops/health/readiness
                port: 8080
              periodSeconds: 10
              successThreshold: 1
              timeoutSeconds: 1
            env:
              - name: test-serviceClientId
                valueFrom:
                  secretKeyRef:
                    key: test-serviceClientId
                    name: test-service-133
              - name: test-serviceClientSecret
                valueFrom:
                  secretKeyRef:
                    key: test-serviceClientSecret
                    name: test-service-133
              - name: test-serviceTenantClientId
                valueFrom:
                  secretKeyRef:
                    key: test-serviceTenantClientId
                    name: test-service-133
              - name: test-serviceTenantClientSecret
                valueFrom:
                  secretKeyRef:
                    key: test-serviceTenantClientSecret
                    name: test-service-133
            resources:
              limits:
                cpu: 800m
              requests:
                cpu: 300m
  </code></pre>
  
  <p>Configured the ingress on a service with rewrite as follows</p>
  
  <pre><code>apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    name: test-service
    labels:
      app.kubernetes.io/name: test-service
      helm.sh/chart: test-service-0.1.0
      app.kubernetes.io/instance: test-service
      app.kubernetes.io/managed-by: Helm
    annotations:
      nginx.ingress.kubernetes.io/rewrite-target: /$2
  spec:
    tls:
    - hosts:
      - apiexample.centralus.cloudapp.azure.com
      secretName: tls-secret
    rules:
      - host: "apiexample.centralus.cloudapp.azure.com"
        http:
          paths:
            - path: /testservice(/|$)(.*)
              backend:
                serviceName: test-service
                servicePort: 8080
  </code></pre>
  
  <p>The tls-secret has been generated using</p>
  
  <pre><code>$ openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=apiexample.centralus.cloudapp.azure.com/O=apiexample.centralus.cloudapp.azure.com"
  
  $ kubectl create secret tls tls-secret --key tls.key --cert tls.crt
  </code></pre>
  
  <p>Before applying the tls configuration in the ingress, I was able to get response from the api endpoint. The api endpoint is secured with oauth.</p>
  
  <p>API Endpoint:</p>
  
  <pre><code>http://apiexample.centralus.cloudapp.azure.com/testservice/tenant/api/v1/endpoint
  </code></pre>
  
  <p>After applying the TLS config on the ingress, and hitting</p>
  
  <pre><code>https://apiexample.centralus.cloudapp.azure.com/testservice/tenant/api/v1/endpoint
  </code></pre>
  
  <p>I am getting default backend 404.</p>
  
  <p>I have tested the TLS with ingress using another sample service (which is not secured with oauth) and it seems to be working for that service.
  Here's the configuration for the other services</p>
  
  <pre><code>apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: tea
  spec:
    replicas: 3
    selector:
      matchLabels:
        app: tea
    template:
      metadata:
        labels:
          app: tea
      spec:
        containers:
        - name: tea
          image: nginxdemos/nginx-hello:plain-text
          ports:
          - containerPort: 8080
  ---
  apiVersion: v1
  kind: Service
  metadata:
    name: tea-svc
    labels:
  spec:
    ports:
    - port: 80
      targetPort: 8080
      protocol: TCP
      name: http
    selector:
      app: tea
  </code></pre>
  
  <p>The ingress for the service is configured as follows</p>
  
  <pre><code>apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    name: cafe-ingress
    annotations:
      nginx.ingress.kubernetes.io/rewrite-target: /$2
  spec:
    tls:
    - hosts:
      - apiexample.centralus.cloudapp.azure.com
      secretName: tls-secret
    rules:
    - host: apiexample.centralus.cloudapp.azure.com
      http:
        paths:
        - path: /teaprefix(/|$)(.*)
          backend:
            serviceName: tea-svc
            servicePort: 80
  </code></pre>
  
  <p>The endpoint</p>
  
  <pre><code>https://apiexample.centralus.cloudapp.azure.com/teaprefix/someurl
  </code></pre>
  
  <p>works fine.</p>
  
  <p>Please let me know if there's anything missing in my configuration and any potential issues that i may have ignored.</p>
  
  <p><strong>Note: The Service and the Ingress are deployed in the default namespace and the Ingress Controller is running in the ingress namespace
  The Nginx Ingress Controller is running in 2 Pods</strong></p>
  
  <p><strong>Logs from Ingress Controller with TLS configuration</strong></p>
  
  <pre><code>Pod1
  
   10.244.0.1 - - [22/Mar/2020:06:57:12 +0000] "GET /testservice/tenant/api/v1/endpoint HTTP/1.1" 302 0 "-" "PostmanRuntime/7.23.0" 1495 0.004 [default-test-service-8080] [] 10.244.0.7:8080 0 0.004 302 f4671ede2f95148220c21fe44de6fdad
   10.244.0.1 - - [22/Mar/2020:06:57:13 +0000] "GET /tenant/api/v1/endpoint HTTP/1.1" 404 21 "http://apiexample.centralus.cloudapp.azure.com/tenant/api/v1/endpoint" "PostmanRuntime/7.23.0" 1563 0.001 [upstream-default-backend] [] 10.244.0.225:8080 21 0.004 404 ed41b36bc6b89b60bc3f208539a0d44c
  
  Pod2
  
      10.244.0.1 - - [22/Mar/2020:06:57:12 +0000] "GET /tenant/api/v1/endpoint HTTP/1.1" 308 171 "https://apiexample.centralus.cloudapp.azure.com/testservice/tenant/api/v1/endpoint" "PostmanRuntime/7.23.0" 1580 0.000 [upstream-default-backend] [] - - - - ce955b7bb5118169e99dd4051060c897
  </code></pre>
  
  <p><strong>Logs from Ingress Controller without TLS configuration</strong></p>
  
  <pre><code>10.244.0.1 - - [22/Mar/2020:07:04:34 +0000] "GET /testservice/tenant/api/v1/endpoint HTTP/1.1" 200 276 "-" "PostmanRuntime/7.23.0" 1495 2.165 [default-test-service-8080] [] 10.244.0.4:8080 548 2.168 200 e866f277def90c398df4e509e45718b2
  </code></pre>
  
  <p><strong>UPDATE</strong></p>
  
  <p>Disabling the authentication on the backend service (test-service) also results in the same behavior.
  Without applying TLS, able to hit the endpoint using http without any Bearer Token.
  After applying TLS, get a default backend - 404 when i hit the endpoint with https/http</p>
  
  <p><strong>UPDATE</strong></p>
  
  <p>Exposing the Service via ClusterIP without the </p>
  
  <pre><code>service.beta.kubernetes.io/azure-load-balancer-internal: '"true"'
  </code></pre>
  
  <p>annotation instead of LoadBalancer also does not seem to be helping. The endpoint works without TLS and with TLS applied, get a default backend - 404</p>
  
  <p><strong>UPDATE</strong></p>
  
  <p>The test-service is a Spring Boot Application with the following WebSecurityConfiguration</p>
  
  <pre><code>@Component
  @EnableResourceServer
  public class WebSecurityConfiguration extends ResourceServerConfigurerAdapter {
  
    private static final Logger LOGGER = LoggerFactory.getLogger(WebSecurityConfiguration.class);
  
    private final HealthCheckWebSecurity healthCheckWebSecurity = new HealthCheckWebSecurity();
  
    private final Oauth2Settings oauth2Settings;
    private final JwtTokenStore jwtTokenStore;
    private final TenantService tenantService;
    private final TransportGuaranteeWebSecurity transportGuaranteeWebSecurity;
  
    @Autowired
    public WebSecurityConfiguration(
        Oauth2Settings oauth2Settings,
        JwtTokenStore jwtTokenStore,
        TenantService tenantService,
        TransportGuaranteeWebSecurity transportGuaranteeWebSecurity) {
      this.oauth2Settings = oauth2Settings;
      this.jwtTokenStore = jwtTokenStore;
      this.tenantService = tenantService;
      this.transportGuaranteeWebSecurity = transportGuaranteeWebSecurity;
    }
  
    @Override
    public void configure(ResourceServerSecurityConfigurer resources) throws Exception {
      String resourceId = oauth2Settings.getResource("default").getResourceId();
      LOGGER.info("Resource service id: {}", resourceId);
  
      resources.resourceId(resourceId).tokenStore(jwtTokenStore);
    }
  
    @Override
    public void configure(HttpSecurity http) throws Exception {
      http.requestMatchers().anyRequest();
      http.sessionManagement().sessionCreationPolicy(SessionCreationPolicy.STATELESS);
      http.csrf().disable();
  
      healthCheckWebSecurity.configure(http);
      transportGuaranteeWebSecurity.configure(http);
  
      http.authorizeRequests().anyRequest().permitAll();
      http.addFilterAfter(buildTenancyContextFilter(), ChannelProcessingFilter.class);
      http.addFilterAfter(buildLongUrlFilter(), ChannelProcessingFilter.class);
    }
  
    private TenancyContextFilter buildTenancyContextFilter() {
      return new TenancyContextFilter(tenantService,
          new PathVariableTenantExtractor(Arrays.asList("/{tenantAlias}/api/**")));
    }
  
    private LongRequestHttpFilter buildLongUrlFilter() {
      return new LongRequestHttpFilter();
    }
  }
  
  public final class TransportGuaranteeWebSecurity {
  
    private TransportGuaranteeSettings transportGuaranteeSettings;
  
    TransportGuaranteeWebSecurity(TransportGuaranteeSettings transportGuaranteeSettings) {
      this.transportGuaranteeSettings = transportGuaranteeSettings;
    }
  
  
    public void configure(HttpSecurity httpSecurity) throws Exception {
      if (httpsRequired()) {
        httpSecurity.requiresChannel().anyRequest().requiresSecure();
      } else {
        httpSecurity.requiresChannel().anyRequest().requiresInsecure();
      }
    }
  
    private boolean httpsRequired() {
      final String transportGuarantee = transportGuaranteeSettings.getTransportGuarantee();
      return !TransportGuaranteeSettings.TRANSPORT_GUARANTEE_NONE.equalsIgnoreCase(transportGuarantee);
    }
  
  }
  
  
  @ConfigurationProperties(prefix = "web.security")
  public class TransportGuaranteeSettings {
    static final String TRANSPORT_GUARANTEE_NONE = "NONE";
    static final String TRANSPORT_GUARANTEE_CONFIDENTIAL = "CONFIDENTIAL";
  
    private static final Logger LOGGER = LoggerFactory.getLogger(TransportGuaranteeSettings.class);
  
    private static final String TRANSPORT_GUARANTEE_PROPERTY = "web.security.transportGuarantee";
  
    private String transportGuarantee;
  
    public String getTransportGuarantee() {
      return transportGuarantee;
    }
  
    public void setTransportGuarantee(String transportGuarantee) {
      this.transportGuarantee = transportGuarantee.trim();
      logUnexpectedValue();
    }
  
    private void logUnexpectedValue() {
      if (!TRANSPORT_GUARANTEE_NONE.equalsIgnoreCase(transportGuarantee)
          &amp;&amp; !TRANSPORT_GUARANTEE_CONFIDENTIAL.equalsIgnoreCase(transportGuarantee)) {
        LOGGER.debug(
            "Unknown value '{}' for property '{}' (expected '{}' or '{}'). Defaulted to '{}'.",
            transportGuarantee, TRANSPORT_GUARANTEE_PROPERTY, TRANSPORT_GUARANTEE_NONE, TRANSPORT_GUARANTEE_CONFIDENTIAL,
            TRANSPORT_GUARANTEE_CONFIDENTIAL);
      }
    }
  }
  </code></pre>
  
  <p>In my application.yaml, </p>
  
  <pre><code>web.security.transportGuarantee: NONE
  </code></pre>
  
  <p>The tenancy context filter extracts the Tenant information from the URL and sets a ThreadLocal. There should not be any issue with that since I am able to hit the endpoint without the TLS configuration. I also do not see any issue with the TransportGuaranteeWebSecurity for the same reason.</p>
  
  <p><strong>Some more logs for Debug</strong></p>
  
  <p><strong>kubectl get pods -owide --namespace ingress</strong></p>
  
  <pre><code>NAME                                             READY   STATUS    RESTARTS   AGE   IP             NODE                                NOMINATED NODE   READINESS GATES
  nginx-ingress-controller-5fcbccd545-bdh25        1/1     Running   1          15d   10.244.0.22    aks-agentpool-44086776-vmss000000   &lt;none&gt;           &lt;none&gt;
  nginx-ingress-controller-5fcbccd545-ptx6j        1/1     Running   0          15d   10.244.0.21    aks-agentpool-44086776-vmss000000   &lt;none&gt;           &lt;none&gt;
  nginx-ingress-default-backend-554d7bd77c-zxzlf   1/1     Running   0          15d   10.244.0.225   aks-agentpool-44086776-vmss000000   &lt;none&gt;           &lt;none&gt;
  </code></pre>
  
  <p><strong>kubectl get svc</strong></p>
  
  <pre><code>test-service                          LoadBalancer   10.0.231.35    13.89.111.39    8080:31534/TCP               14d
  tea-svc                                   ClusterIP      10.0.12.216    &lt;none&gt;          80/TCP                       17d
  </code></pre>
  
  <p><strong>kubectl get ing</strong></p>
  
  <pre><code>test-service         apiexample.centralus.cloudapp.azure.com   10.240.0.4   80, 443      15d
  </code></pre>