title: Microk8s ingress with nginx - why this setup doesn't work?
body: <p>I'd like to put together a dev environment where there is a kubernetes cluster (I intend to use Microk8s with multiple nodes at the end). The reason is that I'll have a prod system running on this cluster with test environments, and eventually when a new PR is created based on the PR id a totally new system will be created and the url will be different. Something like this: prod system: <a href="http://my-system.com" rel="nofollow noreferrer">http://my-system.com</a>, test: <a href="http://test-pr-63.my-system.com" rel="nofollow noreferrer">http://test-pr-63.my-system.com</a>.</p>
  <p>But, first I need a kubernetes with a minimal ingress listening on an IP address and able to route traffic to services/pods, based on URL. I'm not a well versed in kubernetes space.</p>
  <p>The end result is always <code>connection refused</code> when I call the IP via curl, and I don't know why.</p>
  <p>I do the following steps to install the system on a newly created microk8s environment on m Macbook Pro.</p>
  <h1>Status</h1>
  <pre><code>microk8s is running
  high-availability: no
    datastore master nodes: 127.0.0.1:19001
    datastore standby nodes: none
  addons:
    enabled:
      ha-cluster           # Configure high availability on the current node
    disabled:
  </code></pre>
  <h1>Enable services</h1>
  <pre><code>~/: microk8s enable ingress dns storage
  Enabling Ingress
  ingressclass.networking.k8s.io/public created
  namespace/ingress created
  serviceaccount/nginx-ingress-microk8s-serviceaccount created
  clusterrole.rbac.authorization.k8s.io/nginx-ingress-microk8s-clusterrole created
  role.rbac.authorization.k8s.io/nginx-ingress-microk8s-role created
  clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress-microk8s created
  rolebinding.rbac.authorization.k8s.io/nginx-ingress-microk8s created
  configmap/nginx-load-balancer-microk8s-conf created
  configmap/nginx-ingress-tcp-microk8s-conf created
  configmap/nginx-ingress-udp-microk8s-conf created
  daemonset.apps/nginx-ingress-microk8s-controller created
  Ingress is enabled
  Enabling DNS
  Applying manifest
  serviceaccount/coredns created
  configmap/coredns created
  Warning: spec.template.metadata.annotations[scheduler.alpha.kubernetes.io/critical-pod]: non-functional in v1.16+; use the &quot;priorityClassName&quot; field instead
  deployment.apps/coredns created
  service/kube-dns created
  clusterrole.rbac.authorization.k8s.io/coredns created
  clusterrolebinding.rbac.authorization.k8s.io/coredns created
  Restarting kubelet
  DNS is enabled
  Enabling default storage class
  deployment.apps/hostpath-provisioner created
  storageclass.storage.k8s.io/microk8s-hostpath created
  serviceaccount/microk8s-hostpath created
  clusterrole.rbac.authorization.k8s.io/microk8s-hostpath created
  clusterrolebinding.rbac.authorization.k8s.io/microk8s-hostpath created
  Storage will be available soon
  </code></pre>
  <p>The cluster looks like this:</p>
  <pre><code> ~/: kubectl get all --all-namespaces
  NAMESPACE     NAME                                           READY   STATUS    RESTARTS   AGE
  ingress       pod/nginx-ingress-microk8s-controller-5w7tw    1/1     Running   0          92s
  kube-system   pod/coredns-7f9c69c78c-94nbl                   1/1     Running   0          91s
  kube-system   pod/calico-kube-controllers-69d7f794d9-wz79l   1/1     Running   0          4m10s
  kube-system   pod/calico-node-2r6bv                          1/1     Running   0          4m11s
  kube-system   pod/hostpath-provisioner-566686b959-8rwkv      1/1     Running   0          14s
  
  NAMESPACE     NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                  AGE
  default       service/kubernetes   ClusterIP   10.152.183.1    &lt;none&gt;        443/TCP                  7m59s
  kube-system   service/kube-dns     ClusterIP   10.152.183.10   &lt;none&gt;        53/UDP,53/TCP,9153/TCP   91s
  
  NAMESPACE     NAME                                               DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
  kube-system   daemonset.apps/calico-node                         1         1         1       1            1           kubernetes.io/os=linux   4m48s
  ingress       daemonset.apps/nginx-ingress-microk8s-controller   1         1         1       1            1           &lt;none&gt;                   92s
  
  NAMESPACE     NAME                                      READY   UP-TO-DATE   AVAILABLE   AGE
  kube-system   deployment.apps/calico-kube-controllers   1/1     1            1           4m48s
  kube-system   deployment.apps/coredns                   1/1     1            1           92s
  kube-system   deployment.apps/hostpath-provisioner      1/1     1            1           79s
  
  NAMESPACE     NAME                                                 DESIRED   CURRENT   READY   AGE
  kube-system   replicaset.apps/calico-kube-controllers-69d7f794d9   1         1         1       4m11s
  kube-system   replicaset.apps/coredns-7f9c69c78c                   1         1         1       92s
  kube-system   replicaset.apps/hostpath-provisioner-566686b959      1         1         1       14s
  </code></pre>
  <h1>Install kafka by helm using bitnamy's chart</h1>
  <pre><code>NAMESPACE     NAME                                           READY   STATUS    RESTARTS      AGE
  ingress       pod/nginx-ingress-microk8s-controller-5w7tw    1/1     Running   0             3m55s
  kube-system   pod/coredns-7f9c69c78c-94nbl                   1/1     Running   0             3m54s
  kube-system   pod/calico-kube-controllers-69d7f794d9-wz79l   1/1     Running   0             6m33s
  kube-system   pod/calico-node-2r6bv                          1/1     Running   0             6m34s
  kube-system   pod/hostpath-provisioner-566686b959-8rwkv      1/1     Running   0             2m37s
  eg            pod/eg-kafka-zookeeper-0                       1/1     Running   0             48s
  eg            pod/eg-kafka-zookeeper-1                       1/1     Running   0             48s
  eg            pod/eg-kafka-zookeeper-2                       1/1     Running   0             48s
  eg            pod/eg-kafka-0                                 1/1     Running   1 (32s ago)   48s
  eg            pod/eg-kafka-1                                 1/1     Running   1 (31s ago)   48s
  eg            pod/eg-kafka-2                                 1/1     Running   1 (30s ago)   48s
  
  NAMESPACE     NAME                                  TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
  default       service/kubernetes                    ClusterIP   10.152.183.1     &lt;none&gt;        443/TCP                      10m
  kube-system   service/kube-dns                      ClusterIP   10.152.183.10    &lt;none&gt;        53/UDP,53/TCP,9153/TCP       3m54s
  eg            service/eg-kafka-zookeeper-headless   ClusterIP   None             &lt;none&gt;        2181/TCP,2888/TCP,3888/TCP   48s
  eg            service/eg-kafka                      ClusterIP   10.152.183.18    &lt;none&gt;        9092/TCP                     48s
  eg            service/eg-kafka-zookeeper            ClusterIP   10.152.183.140   &lt;none&gt;        2181/TCP,2888/TCP,3888/TCP   48s
  eg            service/eg-kafka-headless             ClusterIP   None             &lt;none&gt;        9092/TCP,9093/TCP            48s
  
  NAMESPACE     NAME                                               DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
  kube-system   daemonset.apps/calico-node                         1         1         1       1            1           kubernetes.io/os=linux   7m11s
  ingress       daemonset.apps/nginx-ingress-microk8s-controller   1         1         1       1            1           &lt;none&gt;                   3m55s
  
  NAMESPACE     NAME                                      READY   UP-TO-DATE   AVAILABLE   AGE
  kube-system   deployment.apps/calico-kube-controllers   1/1     1            1           7m11s
  kube-system   deployment.apps/coredns                   1/1     1            1           3m55s
  kube-system   deployment.apps/hostpath-provisioner      1/1     1            1           3m42s
  
  NAMESPACE     NAME                                                 DESIRED   CURRENT   READY   AGE
  kube-system   replicaset.apps/calico-kube-controllers-69d7f794d9   1         1         1       6m34s
  kube-system   replicaset.apps/coredns-7f9c69c78c                   1         1         1       3m55s
  kube-system   replicaset.apps/hostpath-provisioner-566686b959      1         1         1       2m37s
  
  NAMESPACE   NAME                                  READY   AGE
  eg          statefulset.apps/eg-kafka-zookeeper   3/3     48s
  eg          statefulset.apps/eg-kafka             3/3     48s
  </code></pre>
  <h1>Deploy the single service where to the traffic will be routed</h1>
  <p>It is a simple webapi using Spring, the image works and spins up without any problem.</p>
  <pre class="lang-yaml prettyprint-override"><code>    apiVersion: v1
      kind: Service
      metadata:
        name: webapi
      spec:
        selector:
          app: webapi
        ports:
        - port: 80
          targetPort: 80
      
      ---
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: webapi
      spec:
        selector:
          matchLabels:
            app: webapi
        
        template:
          metadata:
            labels:
              app: webapi
          
          spec:
            containers:
            - name: webapi
        
              image: ghcr.io/encyclopediagalactica/sourceformats.api.rest/sourceformat-api-rest:latest
              resources:
                limits:
                  memory: &quot;1Gi&quot;
                  cpu: &quot;500m&quot;
              ports:
              - containerPort: 80
  </code></pre>
  <p><code>kubectl get all --all-namespaces</code> the newly added stuff is marked <code>====&gt;</code></p>
  <pre><code>NAMESPACE     NAME                                           READY   STATUS    RESTARTS        AGE
  ingress       pod/nginx-ingress-microk8s-controller-5w7tw    1/1     Running   0               8m45s
  kube-system   pod/coredns-7f9c69c78c-94nbl                   1/1     Running   0               8m44s
  kube-system   pod/calico-kube-controllers-69d7f794d9-wz79l   1/1     Running   0               11m
  kube-system   pod/calico-node-2r6bv                          1/1     Running   0               11m
  kube-system   pod/hostpath-provisioner-566686b959-8rwkv      1/1     Running   0               7m27s
  eg            pod/eg-kafka-zookeeper-0                       1/1     Running   0               5m38s
  eg            pod/eg-kafka-zookeeper-1                       1/1     Running   0               5m38s
  eg            pod/eg-kafka-zookeeper-2                       1/1     Running   0               5m38s
  eg            pod/eg-kafka-0                                 1/1     Running   1 (5m22s ago)   5m38s
  eg            pod/eg-kafka-1                                 1/1     Running   1 (5m21s ago)   5m38s
  eg            pod/eg-kafka-2                                 1/1     Running   1 (5m20s ago)   5m38s
  ====&gt; eg            pod/webapi-7755b88f98-kwsmx                    1/1     Running   0               52s
  
  NAMESPACE     NAME                                  TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
  default       service/kubernetes                    ClusterIP   10.152.183.1     &lt;none&gt;        443/TCP                      15m
  kube-system   service/kube-dns                      ClusterIP   10.152.183.10    &lt;none&gt;        53/UDP,53/TCP,9153/TCP       8m44s
  eg            service/eg-kafka-zookeeper-headless   ClusterIP   None             &lt;none&gt;        2181/TCP,2888/TCP,3888/TCP   5m38s
  eg            service/eg-kafka                      ClusterIP   10.152.183.18    &lt;none&gt;        9092/TCP                     5m38s
  eg            service/eg-kafka-zookeeper            ClusterIP   10.152.183.140   &lt;none&gt;        2181/TCP,2888/TCP,3888/TCP   5m38s
  eg            service/eg-kafka-headless             ClusterIP   None             &lt;none&gt;        9092/TCP,9093/TCP            5m38s
  ====&gt; eg            service/webapi                        ClusterIP   10.152.183.96    &lt;none&gt;        80/TCP                       52s
  
  NAMESPACE     NAME                                               DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
  kube-system   daemonset.apps/calico-node                         1         1         1       1            1           kubernetes.io/os=linux   12m
  ingress       daemonset.apps/nginx-ingress-microk8s-controller   1         1         1       1            1           &lt;none&gt;                   8m45s
  
  NAMESPACE     NAME                                      READY   UP-TO-DATE   AVAILABLE   AGE
  kube-system   deployment.apps/calico-kube-controllers   1/1     1            1           12m
  kube-system   deployment.apps/coredns                   1/1     1            1           8m45s
  kube-system   deployment.apps/hostpath-provisioner      1/1     1            1           8m32s
  ====&gt; eg            deployment.apps/webapi                    1/1     1            1           52s
  
  NAMESPACE     NAME                                                 DESIRED   CURRENT   READY   AGE
  kube-system   replicaset.apps/calico-kube-controllers-69d7f794d9   1         1         1       11m
  kube-system   replicaset.apps/coredns-7f9c69c78c                   1         1         1       8m45s
  kube-system   replicaset.apps/hostpath-provisioner-566686b959      1         1         1       7m27s
  eg            replicaset.apps/webapi-7755b88f98                    1         1         1       52s
  
  NAMESPACE   NAME                                  READY   AGE
  eg          statefulset.apps/eg-kafka-zookeeper   3/3     5m38s
  eg          statefulset.apps/eg-kafka             3/3     5m38s
  </code></pre>
  <h1>Ingress rule</h1>
  <pre class="lang-yaml prettyprint-override"><code>    apiVersion: networking.k8s.io/v1
      kind: Ingress
      metadata:
        name: webapi-ingress
        #annotations:
          #kubernetes.io/ingress.class: public
      spec:
        rules:
        - host: blabla.com
          http:
            paths:
            - pathType: Prefix
              path: &quot;/&quot;
              backend:
                service:
                  name: webapi
                  port: 
                    number: 80
        ingressClassName: public
  </code></pre>
  <p>result:</p>
  <pre><code> ~/: kubectl get ingress -n eg
  NAME             CLASS    HOSTS        ADDRESS     PORTS   AGE
  webapi-ingress   public   blabla.com   127.0.0.1   80      48s
  </code></pre>
  <p><strong>Additional info</strong>:</p>
  <p>Regarding the commented out lines. There this this <a href="https://stackoverflow.com/questions/54506269/simple-ingress-from-host-with-microk8s/67041204#67041204">post</a> which points out that the annotations should match. If I follow what is described in the answer the result is the same.
  There is <a href="https://stackoverflow.com/a/69588941/825968">this</a> post which emphasizes using the <code>ingressClassName</code> property. If I follow this the end result is the same. The two suggestion cannot be combined because kubernetes throws an error.</p>
  <h1>Call the endpoint</h1>
  <pre><code>~/: curl http://blabla.com/get
  curl: (7) Failed to connect to blabla.com port 80: Connection refused
  </code></pre>
  <p>My <code>/etc/hosts</code> looks like the following:</p>
  <pre><code>    ~/: cat /etc/hosts
      ##
      # Host Database
      #
      # localhost is used to configure the loopback interface
      # when the system is booting.  Do not change this entry.
      ##
      127.0.0.1       localhost
      255.255.255.255 broadcasthost
      ::1             localhost
      127.0.0.1       blabla.com
      # Added by Docker Desktop
      # To allow the same kube context to work on the host and the container:
      127.0.0.1 kubernetes.docker.internal
      # End of section
  </code></pre>
  <p>Why <code>127.0.0.1</code>? I watched <a href="https://youtu.be/80Ew_fsV4rM?t=860" rel="nofollow noreferrer">this video</a> where Nana says that to whatever IP the ingress rule bound to in the <code>/etc/hosts</code> file I have to map it with the domain in order to route the traffic there. It makes sense to me.</p>
  <p>I also followed <a href="https://stackoverflow.com/a/64522733/825968">this</a> answer's suggestions and the result is the same.</p>
  <p>How about debugging you might ask... So, nginx logs don't say a word. Seems like the traffic doesn't get to nginx.</p>
  <p>I can't decide whether I need metallb... but, discouraging that Microk8s docs mentions that some part of the metallb won't work due to Macbook network traffic manipulation.</p>
  <p>So, I have tried at least 4-5 scenarios and the result is the same. I assume 1, I either miss something fundamental, or 2, my Macbook has some magic which doesn't let the traffic go the kubernetes..., or, 3, the both, which wouldn't not be a big surprise. :)</p>
  <p>Have you tried minikube, you might ask... Yes, I tried. It can't deal with setting up the Kafka instances. Not an option for me.</p>
  <p>So, what I'm doing wrong here? Is there a tutorial which helps me to setup this cluster?</p>