<p>I am learning about microk8s and how ingress works.</p>
<p>I have a single node microk8s (v1.18.4) with the following add-ons: DNS, ingress, RBAC, storage</p>
<p>I am trying to get it working with the microbot example. I've read (and reread) through the tutorial but, the address, once the ingress manifest is applied results with the microbot service, routed to <code>127.0.0.1</code> (and not the internal pod IP).</p>
<p>I am attempting to access the app: <code>http://192.168.91.166/microbot</code> via web external to the vm that it's running inside (and have tried curl while logged into the vm) it would result in an error page being returned. <code>192.168.91.166</code> is the vm's ip.</p>
<pre class="lang-sh prettyprint-override"><code># kubectl get all -A
NAMESPACE     NAME                                          READY   STATUS    RESTARTS   AGE
development   pod/microbot-7dcc49575f-fqttp                 1/1     Running   1          22h
ingress       pod/nginx-ingress-microk8s-controller-phjgl   1/1     Running   1          21h
kube-system   pod/coredns-588fd544bf-lb8mx                  1/1     Running   3          37h
kube-system   pod/hostpath-provisioner-75fdc8fccd-4klwh     1/1     Running   5          37h

NAMESPACE     NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                  AGE
default       service/kubernetes   ClusterIP   10.152.183.1    &lt;none&gt;        443/TCP                  37h
development   service/microbot     ClusterIP   10.152.183.19   &lt;none&gt;        80/TCP                   22h
kube-system   service/kube-dns     ClusterIP   10.152.183.10   &lt;none&gt;        53/UDP,53/TCP,9153/TCP   37h

NAMESPACE   NAME                                               DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
ingress     daemonset.apps/nginx-ingress-microk8s-controller   1         1         1       1            1           &lt;none&gt;          21h

NAMESPACE     NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
development   deployment.apps/microbot               1/1     1            1           22h
kube-system   deployment.apps/coredns                1/1     1            1           37h
kube-system   deployment.apps/hostpath-provisioner   1/1     1            1           37h

NAMESPACE     NAME                                              DESIRED   CURRENT   READY   AGE
development   replicaset.apps/microbot-7dcc49575f               1         1         1       22h
kube-system   replicaset.apps/coredns-588fd544bf                1         1         1       37h
kube-system   replicaset.apps/hostpath-provisioner-75fdc8fccd   1         1         1       37h

</code></pre>
<p><strong>microbot svc description</strong></p>
<pre class="lang-sh prettyprint-override"><code># kubectl describe svc microbot -n development
Name:              microbot
Namespace:         development
Labels:            &lt;none&gt;
Annotations:       Selector:  name=microbot
Type:              ClusterIP
IP:                10.152.183.19
Port:              &lt;unset&gt;  80/TCP
TargetPort:        8080/TCP
Endpoints:         10.1.62.21:8080
Session Affinity:  None
Events:            &lt;none&gt;

</code></pre>
<p><strong>ingress description</strong></p>
<pre class="lang-sh prettyprint-override"><code># kubectl describe ingress microbot -n development
Name:             microbot
Namespace:        development
Address:          127.0.0.1
Default backend:  default-http-backend:80 (&lt;error: endpoints &quot;default-http-backend&quot; not found&gt;)
Rules:
  Host        Path  Backends
  ----        ----  --------
  *           
              /microbot   microbot:80 (10.1.62.21:8080)
Annotations:  kubernetes.io/ingress.class: nginx
              nginx.ingress.kubernetes.io/rewrite-target: /
Events:
  Type    Reason  Age   From                      Message
  ----    ------  ----  ----                      -------
  Normal  UPDATE  14s   nginx-ingress-controller  Ingress development/microbot
</code></pre>
<p>ingress pod log</p>
<pre class="lang-sh prettyprint-override"><code># kubectl logs -n ingress nginx-ingress-microk8s-controller-phjgl
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:       0.33.0
  Build:         git-589187c35
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.19.0

-------------------------------------------------------------------------------

W0715 13:52:22.191579       6 flags.go:249] SSL certificate chain completion is disabled (--enable-ssl-chain-completion=false)
W0715 13:52:22.192057       6 client_config.go:552] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0715 13:52:22.194678       6 main.go:218] Creating API client for https://10.152.183.1:443
W0715 13:52:53.301344       6 main.go:259] Initial connection to the Kubernetes API server was retried 1 times.
I0715 13:52:53.301361       6 main.go:262] Running in Kubernetes cluster version v1.18+ (v1.18.4-1+6f17be3f1fd54a) - git (clean) commit 6f17be3f1fd54a88681869d1cf8bedd5a2174504 - platform linux/amd64
I0715 13:52:53.538979       6 main.go:103] SSL fake certificate created /etc/ingress-controller/ssl/default-fake-certificate.pem
I0715 13:52:53.539788       6 main.go:111] Enabling new Ingress features available since Kubernetes v1.18
E0715 13:52:53.542526       6 main.go:120] Unexpected error searching IngressClass: ingressclasses.networking.k8s.io &quot;nginx&quot; is forbidden: User &quot;system:serviceaccount:ingress:nginx-ingress-microk8s-serviceaccount&quot; cannot get resource &quot;ingressclasses&quot; in API group &quot;networking.k8s.io&quot; at the cluster scope
W0715 13:52:53.542574       6 main.go:123] No IngressClass resource with name nginx found. Only annotation will be used.
I0715 13:52:53.590982       6 nginx.go:263] Starting NGINX Ingress controller
I0715 13:52:53.604026       6 event.go:278] Event(v1.ObjectReference{Kind:&quot;ConfigMap&quot;, Namespace:&quot;ingress&quot;, Name:&quot;nginx-ingress-tcp-microk8s-conf&quot;, UID:&quot;5f1b23d0-297c-4b2f-ac43-f3dc7256f684&quot;, APIVersion:&quot;v1&quot;, ResourceVersion:&quot;14859&quot;, FieldPath:&quot;&quot;}): type: 'Normal' reason: 'CREATE' ConfigMap ingress/nginx-ingress-tcp-microk8s-conf
I0715 13:52:53.604099       6 event.go:278] Event(v1.ObjectReference{Kind:&quot;ConfigMap&quot;, Namespace:&quot;ingress&quot;, Name:&quot;nginx-ingress-udp-microk8s-conf&quot;, UID:&quot;abbfd3c7-c38c-43df-ae0c-7c26abcdb9cc&quot;, APIVersion:&quot;v1&quot;, ResourceVersion:&quot;14860&quot;, FieldPath:&quot;&quot;}): type: 'Normal' reason: 'CREATE' ConfigMap ingress/nginx-ingress-udp-microk8s-conf
I0715 13:52:53.604190       6 event.go:278] Event(v1.ObjectReference{Kind:&quot;ConfigMap&quot;, Namespace:&quot;ingress&quot;, Name:&quot;nginx-load-balancer-microk8s-conf&quot;, UID:&quot;bc525770-be0a-4303-b26d-7e1c1d424d39&quot;, APIVersion:&quot;v1&quot;, ResourceVersion:&quot;14858&quot;, FieldPath:&quot;&quot;}): type: 'Normal' reason: 'CREATE' ConfigMap ingress/nginx-load-balancer-microk8s-conf
I0715 13:52:54.694319       6 event.go:278] Event(v1.ObjectReference{Kind:&quot;Ingress&quot;, Namespace:&quot;development&quot;, Name:&quot;microbot&quot;, UID:&quot;58d26f3e-d19a-4c53-8a47-1223b3ccc300&quot;, APIVersion:&quot;networking.k8s.io/v1beta1&quot;, ResourceVersion:&quot;22517&quot;, FieldPath:&quot;&quot;}): type: 'Normal' reason: 'CREATE' Ingress development/microbot
I0715 13:52:54.791730       6 nginx.go:307] Starting NGINX process
I0715 13:52:54.791929       6 leaderelection.go:242] attempting to acquire leader lease  ingress/ingress-controller-leader-nginx...
I0715 13:52:54.793717       6 controller.go:139] Configuration changes detected, backend reload required.
I0715 13:52:54.805276       6 leaderelection.go:252] successfully acquired lease ingress/ingress-controller-leader-nginx
I0715 13:52:54.805481       6 status.go:86] new leader elected: nginx-ingress-microk8s-controller-phjgl
I0715 13:52:55.235056       6 controller.go:155] Backend successfully reloaded.
I0715 13:52:55.235148       6 controller.go:164] Initial sync, sleeping for 1 second.
2020/07/15 15:49:54 [error] 38#38: *57190 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: _, request: &quot;GET /microbot HTTP/1.1&quot;, upstream: &quot;http://10.1.62.21:8080/&quot;, host: &quot;127.0.0.1&quot;
2020/07/15 15:49:54 [error] 38#38: *57190 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: _, request: &quot;GET /microbot HTTP/1.1&quot;, upstream: &quot;http://10.1.62.21:8080/&quot;, host: &quot;127.0.0.1&quot;
2020/07/15 15:49:54 [error] 38#38: *57190 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: _, request: &quot;GET /microbot HTTP/1.1&quot;, upstream: &quot;http://10.1.62.21:8080/&quot;, host: &quot;127.0.0.1&quot;
127.0.0.1 - - [15/Jul/2020:15:49:54 +0000] &quot;GET /microbot HTTP/1.1&quot; 502 157 &quot;-&quot; &quot;curl/7.68.0&quot; 81 0.000 [development-microbot-80] [] 10.1.62.21:8080, 10.1.62.21:8080, 10.1.62.21:8080 0, 0, 0 0.000, 0.000, 0.000 502, 502, 502 f4a04290c54d7562d01c2b2ef7b68d6b
2020/07/15 15:50:03 [error] 36#36: *57305 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: _, request: &quot;GET /microbot HTTP/1.1&quot;, upstream: &quot;http://10.1.62.21:8080/&quot;, host: &quot;127.0.0.1&quot;
2020/07/15 15:50:03 [error] 36#36: *57305 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: _, request: &quot;GET /microbot HTTP/1.1&quot;, upstream: &quot;http://10.1.62.21:8080/&quot;, host: &quot;127.0.0.1&quot;
2020/07/15 15:50:03 [error] 36#36: *57305 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: _, request: &quot;GET /microbot HTTP/1.1&quot;, upstream: &quot;http://10.1.62.21:8080/&quot;, host: &quot;127.0.0.1&quot;
127.0.0.1 - - [15/Jul/2020:15:50:03 +0000] &quot;GET /microbot HTTP/1.1&quot; 502 157 &quot;-&quot; &quot;curl/7.68.0&quot; 81 0.000 [development-microbot-80] [] 10.1.62.21:8080, 10.1.62.21:8080, 10.1.62.21:8080 0, 0, 0 0.000, 0.000, 0.000 502, 502, 502 58813a2861ae850b96d10ab9482aaaae
2020/07/15 15:54:44 [error] 37#37: *60738 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: _, request: &quot;GET /microbot HTTP/1.1&quot;, upstream: &quot;http://10.1.62.21:8080/&quot;, host: &quot;127.0.0.1&quot;
2020/07/15 15:54:44 [error] 37#37: *60738 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: _, request: &quot;GET /microbot HTTP/1.1&quot;, upstream: &quot;http://10.1.62.21:8080/&quot;, host: &quot;127.0.0.1&quot;
2020/07/15 15:54:44 [error] 37#37: *60738 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: _, request: &quot;GET /microbot HTTP/1.1&quot;, upstream: &quot;http://10.1.62.21:8080/&quot;, host: &quot;127.0.0.1&quot;
127.0.0.1 - - [15/Jul/2020:15:54:44 +0000] &quot;GET /microbot HTTP/1.1&quot; 502 157 &quot;-&quot; &quot;curl/7.68.0&quot; 81 0.000 [development-microbot-80] [] 10.1.62.21:8080, 10.1.62.21:8080, 10.1.62.21:8080 0, 0, 0 0.000, 0.000, 0.000 502, 502, 502 b2bc837c1eb56d5e598f8160e4f2e3cb
I0715 16:36:08.010736       6 event.go:278] Event(v1.ObjectReference{Kind:&quot;Ingress&quot;, Namespace:&quot;development&quot;, Name:&quot;microbot&quot;, UID:&quot;58d26f3e-d19a-4c53-8a47-1223b3ccc300&quot;, APIVersion:&quot;networking.k8s.io/v1beta1&quot;, ResourceVersion:&quot;94444&quot;, FieldPath:&quot;&quot;}): type: 'Normal' reason: 'UPDATE' Ingress development/microbot
I0715 16:36:08.011329       6 controller.go:139] Configuration changes detected, backend reload required.
I0715 16:36:08.116361       6 controller.go:155] Backend successfully reloaded.
I0715 16:41:19.262503       6 store.go:383] removing ingress microbot based on annotation kubernetes.io/ingress.class
I0715 16:41:19.262648       6 controller.go:139] Configuration changes detected, backend reload required.
I0715 16:41:19.262688       6 event.go:278] Event(v1.ObjectReference{Kind:&quot;Ingress&quot;, Namespace:&quot;development&quot;, Name:&quot;microbot&quot;, UID:&quot;58d26f3e-d19a-4c53-8a47-1223b3ccc300&quot;, APIVersion:&quot;networking.k8s.io/v1beta1&quot;, ResourceVersion:&quot;94444&quot;, FieldPath:&quot;&quot;}): type: 'Normal' reason: 'DELETE' Ingress development/microbot
I0715 16:41:19.343023       6 controller.go:155] Backend successfully reloaded.
I0715 16:42:53.063307       6 store.go:324] ignoring delete for ingress microbot based on annotation kubernetes.io/ingress.class
I0715 16:43:26.671104       6 store.go:349] ignoring add for ingress microbot based on annotation kubernetes.io/ingress.class with value 
</code></pre>
<p>deployment manifest:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: Namespace
metadata:
  name: development
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: microbot
  namespace: development
spec:
  strategy:
    type: Recreate
  replicas: 1
  selector:
    matchLabels:
      name: microbot
  template:
    metadata:
      labels:
        name: microbot
    spec:
      containers:
      - name:  microbot
        image: dontrebootme/microbot:v1
        imagePullPolicy: Always
        envFrom:
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: microbot
  namespace: development
spec:
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP 
  selector:
    name: microbot
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: microbot
  namespace: development
  annotations: 
    kubernetes.io/ingress.class: &quot;nginx&quot;
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - http:
      paths:
      - path: /microbot
        backend:
          serviceName: microbot
          servicePort: 80
</code></pre>
<p>I'm wondering if there's something I am not doing correctly. Any help would be much appreciated.</p>

<p>In microK8s you should be using <code>http://127.0.0.1/microbot</code> to access a pod via ingress from outside the cluster i.e a browser. This is giving you <code>502</code> error in nginx ingress controller log. Few things to check</p>
<ol>
<li><p>Check the service has got <code>Endpoints</code> reflecting correct POD IP using <code>kubectl describe svc microbot -n development</code></p>
</li>
<li><p>Check if container inside pod is listening on port <code>8080</code>. Maybe it's <code>80</code> or something else.</p>
</li>
<li><p>The application running as a container in the pod need to listen on <code>0.0.0.0</code> instead of <code>127.0.0.1</code></p>
</li>
</ol>
