<p>I'm struggling to expose a service in an AWS cluster to outside and access it via a browser. Since my <a href="https://stackoverflow.com/q/69801758/3995261">previous question</a> haven't drawn any answers, I decided to simplify the issue in several aspects.</p>
<p>First, I've created a deployment which should work without any configuration. Based on <a href="https://www.digitalocean.com/community/tutorials/how-to-set-up-an-nginx-ingress-on-digitalocean-kubernetes-using-helm" rel="nofollow noreferrer">this article</a>, I did</p>
<ol>
<li><p><code>kubectl create namespace tests</code></p>
</li>
<li><p>created file <code>probe-service.yaml</code> based on <code>paulbouwer/hello-kubernetes:1.8</code> and deployed it <code>kubectl create -f probe-service.yaml -n tests</code>:</p>
<pre><code>apiVersion: v1
kind: Service
metadata:
  name: hello-kubernetes-first
spec:
  type: ClusterIP
  ports:
  - port: 80
    targetPort: 8080
  selector:
    app: hello-kubernetes-first
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-kubernetes-first
spec:
  replicas: 3
  selector:
    matchLabels:
      app: hello-kubernetes-first
  template:
    metadata:
      labels:
        app: hello-kubernetes-first
    spec:
      containers:
      - name: hello-kubernetes
        image: paulbouwer/hello-kubernetes:1.8
        ports:
        - containerPort: 8080
        env:
        - name: MESSAGE
          value: Hello from the first deployment!
</code></pre>
</li>
<li><p>created <code>ingress.yaml</code> and applied it (<code>kubectl apply -f .\probes\ingress.yaml -n tests</code>)</p>
<pre><code>apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: hello-kubernetes-ingress
spec:
  rules:
  - host: test.projectname.org
    http:
      paths:
      - pathType: Prefix
        path: &quot;/test&quot;
        backend:
          service:
            name: hello-kubernetes-first
            port:
              number: 80
  - host: test2.projectname.org
    http:
      paths:
      - pathType: Prefix
        path: &quot;/test2&quot;
        backend:
          service:
            name: hello-kubernetes-first
            port:
              number: 80
  ingressClassName: nginx
</code></pre>
</li>
</ol>
<p>Second, I can see that DNS actually point to the cluster and ingress rules are applied:</p>
<ul>
<li>if I open <code>http://test.projectname.org/test</code> or any irrelevant path (<code>http://test.projectname.org/test3</code>), I'm shown <code>NET::ERR_CERT_AUTHORITY_INVALID</code>, but</li>
<li>if I use &quot;open anyway&quot; in browser, irrelevant paths give <code>ERR_TOO_MANY_REDIRECTS</code> while <code>http://test.projectname.org/test</code> gives <code>Cannot GET /test</code></li>
</ul>
<p>Now, TLS issues aside (those deserve a separate question), why can I get <code>Cannot GET /test</code>? It looks like ingress controller (ingress-nginx) got the rules (otherwise it wouldn't descriminate paths; that's why I don't show DNS settings, although they are described in the previous question) but instead of showing the simple hello-kubernetes page at <code>/test</code> it returns this simple 404 message. <strong>Why is that?</strong> What could possibly go wrong? <strong>How to debug this?</strong></p>
<p>Some debug info:</p>
<ul>
<li><p><code>kubectl version --short</code> tells Kubernetes Client Version is v1.21.5 and Server Version is v1.20.7-eks-d88609</p>
</li>
<li><p><code>kubectl get ingress -n tests</code> shows that <code>hello-kubernetes-ingress</code> exists indeed, with <code>nginx</code> class, 2 expected hosts, address equal to that shown for load balancer in AWS console</p>
</li>
<li><p><code>kubectl get all -n tests</code> shows</p>
<pre><code>NAME                                          READY   STATUS    RESTARTS   AGE
pod/hello-kubernetes-first-6f77d8ff99-gjw5d   1/1     Running   0          5h4m
pod/hello-kubernetes-first-6f77d8ff99-ptwsn   1/1     Running   0          5h4m
pod/hello-kubernetes-first-6f77d8ff99-x8w87   1/1     Running   0          5h4m

NAME                             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/hello-kubernetes-first   ClusterIP   10.100.18.189   &lt;none&gt;        80/TCP    5h4m

NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/hello-kubernetes-first   3/3     3            3           5h4m

NAME                                                DESIRED   CURRENT   READY   AGE
replicaset.apps/hello-kubernetes-first-6f77d8ff99   3         3         3       5h4m
</code></pre>
</li>
<li><p><code>ingress-nginx</code> was installed before me via the following chart:</p>
<pre><code>apiVersion: v2
name: nginx
description: A Helm chart for Kubernetes
type: application
version: 4.0.6
appVersion: &quot;1.0.4&quot;
dependencies:
- name: ingress-nginx
  version: 4.0.6
  repository: https://kubernetes.github.io/ingress-nginx
</code></pre>
<p>and the values overwrites applied with the chart differ from the <a href="https://github.com/kubernetes/ingress-nginx/blob/main/charts/ingress-nginx/values.yaml" rel="nofollow noreferrer">original ones</a> mostly (well, those got updated since the installation) in <code>extraArgs</code>: <code>default-ssl-certificate: &quot;nginx-ingress/dragon-family-com&quot;</code> is uncommneted</p>
</li>
</ul>
<p>PS To answer Andrew, I indeed tried to setup HTTPS but it seemingly didn't help, so I haven't included what I tried into the initial question. Yet, here's what I did:</p>
<ol>
<li><p>installed cert-manager, currently without a custom chart: <code>kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.5.4/cert-manager.yaml</code></p>
</li>
<li><p>based on <a href="https://cert-manager.io/docs/configuration/acme/dns01/route53/#creating-an-issuer-or-clusterissuer" rel="nofollow noreferrer">cert-manager's tutorial</a> and <a href="https://stackoverflow.com/q/58553510/3995261">SO question</a> created a ClusterIssuer with the following config:</p>
<pre><code>apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-backoffice

spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    # use https://acme-v02.api.letsencrypt.org/directory after everything is fixed and works
    privateKeySecretRef: # this secret will be created in the namespace of cert-manager
      name: letsencrypt-backoffice-private-key
    # email: &lt;will be used for urgent alerts about expiration etc&gt;

    solvers:
    # TODO: add for each domain/second-level domain/*.projectname.org
    - selector:
        dnsZones:
          - test.projectname.org
          - test2.projectname.org
      # haven't made it to work yet, so switched to the simpler to configure http01 challenge
      # dns01:
      #   route53:
      #     region: ... # that of load balancer (but we also have ...)
      #     accessKeyID: &lt;of IAM user with access to Route53&gt;
      #     secretAccessKeySecretRef: # created that
      #       name: route53-credentials-secret
      #       key: secret-access-key
      #     role: arn:aws:iam::645730347045:role/cert-manager
      http01:
        ingress:
          class: nginx
</code></pre>
<p>and applied it via <code>kubectl apply -f issuer.yaml</code></p>
</li>
<li><p>created 2 certificates in the same file and applied it again:</p>
<pre><code>---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: letsencrypt-certificate
spec:
  secretName: tls-secret
  issuerRef:
    kind: ClusterIssuer
    name: letsencrypt-backoffice
  commonName: test.projectname.org
  dnsNames:
  - test.projectname.org
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: letsencrypt-certificate-2
spec:
  secretName: tls-secret-2
  issuerRef:
    kind: ClusterIssuer
    name: letsencrypt-backoffice
  commonName: test2.projectname.org
  dnsNames:
  - test2.projectname.org
</code></pre>
</li>
<li><p>made sure that the certificates are issued correctly (skipping the pain part, the result is: <code>kubectl get certificates</code> shows that both certificates have <code>READY</code> = <code>true</code> and both tls secrets are created)</p>
</li>
<li><p>figured that my ingress is in another namespace and secrets for tls in ingress spec <a href="https://github.com/kubernetes/ingress-nginx/issues/2170" rel="nofollow noreferrer">can only</a> be referred in the same namespace (haven't tried the wildcard certificate and <code>--default-ssl-certificate</code> option yet), so for each one copied them to <code>tests</code> namespace:</p>
<ol>
<li>opened existing secret, like <code>kubectl edit secret tls-secret-2</code>, copied data and annotations</li>
<li>created an empty (Opaque) secret in tests: <code>kubectl create secret generic tls-secret-2-copy -n tests</code></li>
<li>opened it (<code>kubectl edit secret tls-secret-2-copy -n tests</code>) and inserted data an annotations</li>
</ol>
</li>
<li><p>in ingress <code>spec</code>, added the tls bit:</p>
<pre><code>tls:
- hosts:
  - test.projectname.org
  secretName: tls-secret-copy
- hosts:
  - test2.projectname.org
  secretName: tls-secret-2-copy
</code></pre>
</li>
<li><p>I hoped that this will help, but actually it made no difference (I get <code>ERR_TOO_MANY_REDIRECTS</code> for irrelevant paths, redirect from http to https, <code>NET::ERR_CERT_AUTHORITY_INVALID</code> at https and <code>Cannot GET /test</code> if I insist on getting to the page)</p>
</li>
</ol>

<p>Since you've used your own answer to complement the question, I'll kind of answer all the things you asked, while providing a divide and conquer strategy to troubleshooting kubernetes networking.</p>
<p>At the end I'll give you some nginx and IP answers</p>
<p>This is correct</p>
<pre><code>- host: test3.projectname.org
http:
  paths:
  - pathType: Prefix
    path: &quot;/&quot;
    backend:
      service:
        name: hello-kubernetes-first
        port:
          number: 80
</code></pre>
<hr />
<p>Breaking down troubleshooting with Ingress</p>
<ol>
<li>DNS</li>
<li>Ingress</li>
<li>Service</li>
<li>Pod</li>
<li>Certificate</li>
</ol>
<hr />
<p>1.DNS</p>
<p>you can use the command dig to query the DNS</p>
<p>dig google.com
<a href="https://i.stack.imgur.com/hBwrC.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/hBwrC.png" alt="dig result" /></a></p>
<ol start="2">
<li>Ingress</li>
</ol>
<p>the ingress controller doesn't look for the IP, it just looks for the headers</p>
<p>you can force a host using any tool that lets you change the headers, like curl</p>
<p><code>curl --header 'Host: test3.projectname.com' http://123.123.123.123</code> (your public IP)</p>
<ol start="3">
<li>Service</li>
</ol>
<p>you can be sure that your service is working by creating ubuntu/centos pod, using <code>kubectl exec -it podname -- bash</code> and trying to curl your service form withing the cluster</p>
<ol start="4">
<li>Pod</li>
</ol>
<p>You're getting this</p>
<pre><code>192.168.14.57 - - [14/Nov/2021:12:02:58 +0000] &quot;GET /test2 HTTP/2.0&quot; 404 144
 &quot;-&quot; &quot;&lt;browser's user-agent header value&gt;&quot; 448 0.002
</code></pre>
<p>This part <code>GET /test2</code> means that the request got the address from the DNS, went all the way from the internet, found your clusters, found your ingress controller, got through the service and reached your pod. Congratz! Your ingress is working!</p>
<p>But why is it returning 404?</p>
<p>The path that was passed to the service and from the service to the pod is /test2</p>
<p>Do you have a file called test2 that nginx can serve? Do you have an upstream config in nginx that has a test2 prefix?</p>
<p>That's why, you're getting a 404 from nginx, not from the ingress controller.</p>
<p>Those IPs are internal, remember, the internet traffic ended at the cluster border, now you're in an internal network. Here's a rough sketch of what's happening</p>
<p>Let's say that you're accessing it from your laptop. Your laptop has the IP 192.168.123.123, but your home has the address 7.8.9.1, so when your request hits the cluster, the cluster sees 7.8.9.1 requesting test3.projectname.com.</p>
<p>The cluster looks for the ingress controller, which finds a suitable configuration and passed the request down to the service, which passes the request down to the pod.</p>
<p>So,</p>
<pre><code>your router can see your private IP (192.168.123.123)
Your cluster(ingress) can see your router's IP (7.8.9.1)
Your service can see the ingress's IP (192.168.?.?)
Your pod can see the service's IP (192.168.14.57)
</code></pre>
<p>It's a game of pass around.
If you want to see the public IP in your nginx logs, you need to customize it to get the X-Real-IP header, which is usually where load-balancers/ingresses/ambassador/proxies put the actual requester public IP.</p>
