<p>I have a service (installed as Helm subchart of another app) that is correctly reachable on a Minikube environment, but that is not reachable when deployed on a remote Kubernetes cluster.</p>
<p>The Service should be reachable via an ingress.</p>
<h3>Ingress:</h3>
<pre><code>apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: helper-service
  labels:
    helm.sh/chart: &quot;helper-service-0.2.3&quot;
    app.kubernetes.io/name: &quot;helper-service&quot;
    app.kubernetes.io/version: &quot;2.1&quot;
    app.kubernetes.io/managed-by: &quot;Helm&quot;
  annotations:
    # These annotations don't seem to make any difference
    # traefik.ingress.kubernetes.io/ingress.class: traefik
    # traefik.ingress.kubernetes.io/ssl-proxy-headers: X-Forwarded-Proto:https
    # traefik.ingress.kubernetes.io/ssl-redirect: &quot;true&quot;
spec:
  rules:
    - host: &quot;helper-service.example.com&quot;
      http:
        paths:
          - path: /
            backend:
              serviceName: helper-service
              servicePort: service-port
</code></pre>
<h3>Service</h3>
<pre><code>apiVersion: v1
kind: Service
metadata:
  name: helper-service
  labels:
    helm.sh/chart: &quot;helper-service-1.0.0&quot;
    app.kubernetes.io/name: &quot;helper-service&quot;
    app.kubernetes.io/version: &quot;2.13.1&quot;
    app.kubernetes.io/managed-by: &quot;Helm&quot;
spec:
  selector:
    app.kubernetes.io/name: &quot;helper-service&quot;
  type: ClusterIP
  ports:
    - port: 8080
      name: service-port
      targetPort: 8080
      protocol: TCP
</code></pre>
<h3>Deployment</h3>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: helper-service
  labels:
    helm.sh/chart: &quot;helper-service-1.0.0&quot;
    app.kubernetes.io/name: &quot;helper-service&quot;
    app.kubernetes.io/version: &quot;2.1&quot;
    app.kubernetes.io/managed-by: &quot;Helm&quot;
spec:
  replicas: 1
  revisionHistoryLimit: 3
  strategy:
    rollingUpdate:
      maxSurge: 2
      maxUnavailable: 0
    type: RollingUpdate
  selector:
    matchLabels:
      app.kubernetes.io/name: &quot;helper-service&quot;
  template:
    metadata:
      annotations: {}
      labels:
        app.kubernetes.io/name: &quot;helper-service&quot;
    spec:
      containers:
      - name: helper-service
        image: &quot;myregistry/myimage:myversion&quot;
        env:
        - name: HELPER-SERVICE_BIND
          value: &quot;:8080&quot;
        ports:
        - containerPort: 8080
</code></pre>
<hr />
<p>Sending requests to <code>helper-service.example.com</code> always return a <code>404 not found</code> error from <em>Nginx</em>. Given the response, seems like Nginx is reachable but isn't able to provide access to the service specified in the Ingress.</p>
<p>However, the parent-chart Helm app, on the same cluster/namespace, is reachable via Ingress on a NodePort service and has no problems in being accessed.</p>
<p>Both services are running correctly:</p>
<pre><code>$ k get all
NAME                                         READY   STATUS             RESTARTS   AGE
pod/myapp-645cd66d9-g84vm                    2/2     Running            0          51m
pod/myapp-helper-service-fbc6bb4d8-zklf2     1/1     Running            0          4d19h

NAME                            TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)             AGE
service/myapp                   NodePort    10.10.10.11    &lt;none&gt;        8080:31033/TCP      110d
service/myapp-helper-service    ClusterIP   10.10.10.61    &lt;none&gt;        8080/TCP            110d

NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/myapp                   1/1     1            1           110d
deployment.apps/myapp-helper-service    1/1     1            1           110d
</code></pre>
<p>and the ingresses looks to be there correctly:</p>
<pre><code>$ k get ingresses
NAME                    HOSTS                                             ADDRESS       PORTS     AGE
myapp                   myapp.example.com                                 37.44.1.172   80, 443   116d
myapp-helper-service    myapp-helper-service.example.com                                80        116d
</code></pre>
<p>with the ingress details being:</p>
<pre><code>$ k describe ingress/myapp-helper-service
Name:             myapp-helper-service
Namespace:        my-namespace
Address:
Default backend:  default-http-backend:80 (&lt;error: endpoints &quot;default-http-backend&quot; is forbidden: User cannot get resource &quot;endpoints&quot; in API group &quot;&quot; in the namespace &quot;kube-system&quot;&gt;)
Rules:
  Host                                             Path  Backends
  ----                                             ----  --------
  myapp-helper-service.example.com
                                                   /   myapp-helper-service:service-port (172.25.0.207:8080)
                                                   Events:                                            &lt;none&gt;
</code></pre>
<p>and the deployed service being:</p>
<pre><code>$ k describe service/myapp-helper-service
Name:              myapp-helper-service
Namespace:         mynamespace
Labels:            app.kubernetes.io/instance=myapp
                   app.kubernetes.io/managed-by=Helm
                   app.kubernetes.io/name=helper-service
                   app.kubernetes.io/version=2.1
                   helm.sh/chart=helper-service-0.2.3
Annotations:       meta.helm.sh/release-name: myapp
                   meta.helm.sh/release-namespace: mynamespace
Selector:          app.kubernetes.io/instance=myapp,app.kubernetes.io/name=helper-service
Type:              ClusterIP
IP:                10.10.10.61
Port:              service-port  8080/TCP
TargetPort:        8080/TCP
Endpoints:         172.25.0.207:8080
Port:              metrics-port  8081/TCP
TargetPort:        8081/TCP
Endpoints:         172.25.0.207:8081
Session Affinity:  None
Events:            &lt;none&gt;
</code></pre>
<hr />
<p>I tried to enable/disable the add Traefik <code>annotations</code> to see if anything changed, but the result is the same (still 404 from Nginx 🤷‍♂️).</p>
<ol>
<li>Why the same configuration works correctly on Minikube?</li>
<li>What does the 404 http status suggest here? (ingress reachable but service not found?)</li>
<li>How do I debug the reachability from inside the cluster?</li>
<li>How do I debug the reachability from outside the cluster/ingress?</li>
<li>If the setup is correct here, what else should I check?</li>
</ol>

Kamafeather
<p>I figured the solution out.</p>
<p>Even though the ingress was getting created, apparently it didn't actually work (or was getting some non-better-specified other <em>default</em> class).</p>
<p>The ingress <strong>needed a missing explicit annotation</strong>, specifying the ingress <em>class</em>:</p>
<pre><code>annotations:
    kubernetes.io/ingress.class: nginx-external
</code></pre>
<p>After adding that and redeploying, the ingress started to correctly catch my requests and redirect them to the <code>helper-service</code>!</p>
<h3>Note</h3>
<p>please, write a comment if you know:</p>
<ul>
<li><p><strong>what is the default class when none is specified</strong></p>
</li>
<li><p><strong>how to figure out what class is used by an ingress already running on a cluster</strong></p>
</li>
</ul>
