<p>Kubernetes/Ubuntu newbie here!</p>
<p>I'm setting up a k8s cluster using a single Raspberry Pi (hoping to have more in the future). I'm using microk8s <code>v1.18.8</code> and Ubuntu Server <code>20.04.1 LTS (GNU/Linux 5.4.0-1018-raspi aarch64)</code>.</p>
<p>I'm trying to access one of my k8s services on port <code>80</code>, but I haven't been able to set it up correctly. I've also set a static IP address for accessing the service, and I'm routing traffic from the router to the service's IP address.</p>
<p>I would like to know what I'm doing wrong, or if there's a better approach for what I'm trying to do!</p>
<p>The steps I'm following:</p>
<ol>
<li>I've run <code>microk8s enable dns metallb</code>. I've given MetalLB IP addresses not being handled by the DHPC server (<code>192.168.0.90-192.168.0.99</code>).</li>
<li>I've installed <code>ingress-nginx</code> by running <code>kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.35.0/deploy/static/provider/baremetal/deploy.yaml</code>. This creates a <code>NodePort</code> service for the <code>ingress-nginx-controller</code>, which doesn't work with MetalLB. As mentioned <a href="https://medium.com/better-programming/how-to-expose-your-services-with-kubernetes-ingress-7f34eb6c9b5a" rel="noreferrer">here</a>, I edit the <code>spec.type</code> of the service from <code>NodePort</code> to <code>LoadBalancer</code> by running <code>kubectl edit service ingress-nginx-controller -n ingress-nginx</code>. MetalLB then assigns IP <code>192.168.0.90</code> to the service.</li>
<li>Then I apply the following configuration file:</li>
</ol>
<pre><code>apiVersion: v1
kind: Service
metadata:
  name: wow-ah-api-service
  namespace: develop
spec:
  selector:
    app: wow-ah-api
  ports:
    - protocol: TCP
      port: 80
      targetPort: 3000
---
apiVersion: apps/v1
kind: Deployment
metadata:
  # Unique key of the Deployment instance
  name: wow-ah-api
  namespace: develop
spec:
  # 3 Pods should exist at all times.
  replicas: 3
  selector:
    matchLabels:
      app: wow-ah-api
  template:
    metadata:
      namespace: develop
      labels:
        # Apply this label to pods and default
        # the Deployment label selector to this value
        app: wow-ah-api
    spec:
      imagePullSecrets:
        - name: some-secret
      containers:
        - name: wow-ah-api
          # Run this image
          image: some-image
          imagePullPolicy: Always
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: wow-ah-api-ingress
  namespace: develop
spec:
  backend:
    serviceName: wow-ah-api-service
    servicePort: 3000
</code></pre>
<p>These are some outputs I'm seeing:</p>
<p><code>microk8s kubectl get all --all-namespaces</code></p>
<pre><code>NAMESPACE        NAME                                            READY   STATUS      RESTARTS   AGE
develop          pod/wow-ah-api-6c4bff88f9-2x48v                 1/1     Running     4          4h21m
develop          pod/wow-ah-api-6c4bff88f9-ccw9z                 1/1     Running     4          4h21m
develop          pod/wow-ah-api-6c4bff88f9-rd6lp                 1/1     Running     4          4h21m
ingress-nginx    pod/ingress-nginx-admission-create-mnn8g        0/1     Completed   0          4h27m
ingress-nginx    pod/ingress-nginx-admission-patch-x5r6d         0/1     Completed   1          4h27m
ingress-nginx    pod/ingress-nginx-controller-7896b4fbd4-nglsd   1/1     Running     4          4h27m
kube-system      pod/coredns-588fd544bf-576x5                    1/1     Running     4          4h26m
metallb-system   pod/controller-5f98465b6b-hcj9g                 1/1     Running     4          4h23m
metallb-system   pod/speaker-qc9pc                               1/1     Running     4          4h23m

NAMESPACE       NAME                                         TYPE           CLUSTER-IP       EXTERNAL-IP    PORT(S)                      AGE
default         service/kubernetes                           ClusterIP      10.152.183.1     &lt;none&gt;         443/TCP                      21h
develop         service/wow-ah-api-service                   ClusterIP      10.152.183.88    &lt;none&gt;         80/TCP                       4h21m
ingress-nginx   service/ingress-nginx-controller             LoadBalancer   10.152.183.216   192.168.0.90   80:32151/TCP,443:30892/TCP   4h27m
ingress-nginx   service/ingress-nginx-controller-admission   ClusterIP      10.152.183.41    &lt;none&gt;         443/TCP                      4h27m
kube-system     service/kube-dns                             ClusterIP      10.152.183.10    &lt;none&gt;         53/UDP,53/TCP,9153/TCP       4h26m

NAMESPACE        NAME                     DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                 AGE
metallb-system   daemonset.apps/speaker   1         1         1       1            1           beta.kubernetes.io/os=linux   4h23m

NAMESPACE        NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
develop          deployment.apps/wow-ah-api                 3/3     3            3           4h21m
ingress-nginx    deployment.apps/ingress-nginx-controller   1/1     1            1           4h27m
kube-system      deployment.apps/coredns                    1/1     1            1           4h26m
metallb-system   deployment.apps/controller                 1/1     1            1           4h23m

NAMESPACE        NAME                                                  DESIRED   CURRENT   READY   AGE
develop          replicaset.apps/wow-ah-api-6c4bff88f9                 3         3         3       4h21m
ingress-nginx    replicaset.apps/ingress-nginx-controller-7896b4fbd4   1         1         1       4h27m
kube-system      replicaset.apps/coredns-588fd544bf                    1         1         1       4h26m
metallb-system   replicaset.apps/controller-5f98465b6b                 1         1         1       4h23m

NAMESPACE       NAME                                       COMPLETIONS   DURATION   AGE
ingress-nginx   job.batch/ingress-nginx-admission-create   1/1           27s        4h27m
ingress-nginx   job.batch/ingress-nginx-admission-patch    1/1           29s        4h27m
</code></pre>
<p><code>microk8s kubectl get ingress --all-namespaces</code></p>
<pre><code>NAMESPACE   NAME                 CLASS    HOSTS   ADDRESS         PORTS   AGE
develop     wow-ah-api-ingress   &lt;none&gt;   *       192.168.0.236   80      4h23m
</code></pre>
<p>I have been thinking it could be related to my iptables configuration, but I'm not sure how to configure them to work with microk8s.</p>
<p><code>sudo iptables -L</code></p>
<pre><code>Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
KUBE-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes service portals */
KUBE-EXTERNAL-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes externally-visible service portals */
KUBE-FIREWALL  all  --  anywhere             anywhere            

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
KUBE-FORWARD  all  --  anywhere             anywhere             /* kubernetes forwarding rules */
KUBE-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes service portals */
ACCEPT     all  --  10.1.0.0/16          anywhere             /* generated for MicroK8s pods */
ACCEPT     all  --  anywhere             10.1.0.0/16          /* generated for MicroK8s pods */
ACCEPT     all  --  10.1.0.0/16          anywhere            
ACCEPT     all  --  anywhere             10.1.0.0/16         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
KUBE-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes service portals */
KUBE-FIREWALL  all  --  anywhere             anywhere            

Chain KUBE-EXTERNAL-SERVICES (1 references)
target     prot opt source               destination         

Chain KUBE-FIREWALL (2 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere             /* kubernetes firewall for dropping marked packets */ mark match 0x8000/0x8000
DROP       all  -- !localhost/8          localhost/8          /* block incoming localnet connections */ ! ctstate RELATED,ESTABLISHED,DNAT

Chain KUBE-FORWARD (1 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere             ctstate INVALID
ACCEPT     all  --  anywhere             anywhere             /* kubernetes forwarding rules */ mark match 0x4000/0x4000
ACCEPT     all  --  anywhere             anywhere             /* kubernetes forwarding conntrack pod source rule */ ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere             /* kubernetes forwarding conntrack pod destination rule */ ctstate RELATED,ESTABLISHED

Chain KUBE-KUBELET-CANARY (0 references)
target     prot opt source               destination         

Chain KUBE-PROXY-CANARY (0 references)
target     prot opt source               destination         

Chain KUBE-SERVICES (3 references)
target     prot opt source               destination 
</code></pre>
<p><strong>UPDATE #1</strong></p>
<p><code>metallb ConfigMap</code> (<code>microk8s kubectl edit ConfigMap/config -n metallb-system</code>)</p>
<pre><code>apiVersion: v1
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 192.168.0.90-192.168.0.99
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {&quot;apiVersion&quot;:&quot;v1&quot;,&quot;data&quot;:{&quot;config&quot;:&quot;address-pools:\n- name: default\n  protocol: layer2\n  addresses:\n  - 192.168.0.90-192.168.0.99\n&quot;},&quot;kind&quot;:&quot;ConfigMap&quot;,&quot;metadata&quot;:{&quot;annotations&quot;:{},&quot;name&quot;:&quot;config&quot;,&quot;namespace&quot;:&quot;metallb-system&quot;}}
  creationTimestamp: &quot;2020-09-19T21:18:45Z&quot;
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:data:
        .: {}
        f:config: {}
      f:metadata:
        f:annotations:
          .: {}
          f:kubectl.kubernetes.io/last-applied-configuration: {}
    manager: kubectl
    operation: Update
    time: &quot;2020-09-19T21:18:45Z&quot;
  name: config
  namespace: metallb-system
  resourceVersion: &quot;133422&quot;
  selfLink: /api/v1/namespaces/metallb-system/configmaps/config
  uid: 774f6a73-b1e1-4e26-ba73-ef71bc2e1060
</code></pre>
<p>I'd appreciate any help you could give me!</p>

<h2>Short answer:</h2>
<ol>
<li>You only need (and probably have) one IP address. You must can ping it from outside Microk8s machine.</li>
<li>Here's the error. Remove this step</li>
</ol>
<h2>Long answer by example:</h2>
<p>Clean Microk8s. Only one public IP (or local machine IP. In your use case, I'll use 192.168.0.90).</p>
<p>How do you test? For example</p>
<pre><code>curl -H &quot;Host: blue.nginx.example.com&quot; http://PUBLIC_IP
</code></pre>
<p>from outside the machine.</p>
<p>Run the test. It must fail.</p>
<p>Enable microk8s dns and ingress</p>
<pre><code>microk8s.enable dns ingress
</code></pre>
<p>Run the test. Fails?</p>
<p><strong>If it's the same error then: You need metallb</strong></p>
<ul>
<li><p>With Internet public IP</p>
<p><strong>microk8s.enable metallb:$(curl ipinfo.io/ip)-$(curl ipinfo.io/ip)</strong></p>
</li>
<li><p>With LAN IP 192.168.0.90</p>
<p><strong>microk8s.enable metallb:192.168.0.90-192.168.0.90</strong></p>
</li>
</ul>
<p>Run the test again</p>
<p>If Test NOT return 503 or 404 then: You can't do next steps. Perhaps you have a network problem or firewall filter.</p>
<h3>The Ingress layer</h3>
<p>Our test arrived to the Microk8s Ingress controller. He doesn't know what to do and return a 404 error (sometimes 503).</p>
<p>That's ok. Go next!</p>
<p>I'll use an example from <a href="https://youtu.be/A_PjjCM1eLA?t=984" rel="noreferrer">https://youtu.be/A_PjjCM1eLA?t=984</a> 16:24</p>
<p>[ Kube 32 ] Set up Traefik Ingress on kubernetes Bare Metal Cluster</p>
<p>set kubectl alias</p>
<pre><code>alias kubectl=microk8s.kubectl
</code></pre>
Deploy apps
<pre><code>kubectl create -f https://raw.githubusercontent.com/justmeandopensource/kubernetes/master/yamls/ingress-demo/nginx-deploy-main.yaml
kubectl create -f https://raw.githubusercontent.com/justmeandopensource/kubernetes/master/yamls/ingress-demo/nginx-deploy-blue.yaml
kubectl create -f https://raw.githubusercontent.com/justmeandopensource/kubernetes/master/yamls/ingress-demo/nginx-deploy-green.yaml
</code></pre>
Expose apps in the internal cluster network. ClusterIP by default.
<pre><code>kubectl expose deploy nginx-deploy-main --port 80
kubectl expose deploy nginx-deploy-blue --port 80
kubectl expose deploy nginx-deploy-green --port 80
</code></pre>
<p>Run Test. It doesn't work... yet.</p>
Ingress rule example: how to delivery by host name
<p>Configure the hosts nginx.example.com, blue.nginx.example.com, and green.nginx.example.com and distribute requests to exposed deployments:</p>
<pre><code>kubectl create -f https://raw.githubusercontent.com/justmeandopensource/kubernetes/master/yamls/ingress-demo/ingress-resource-2.yaml
</code></pre>
<p>Run this tests:</p>
<pre><code>curl -H &quot;Host: blue.nginx.example.com&quot; http://PUBLIC_IP
</code></pre>
<p>Now you'll have a response like</p>
<pre><code>&lt;h1&gt;I am &lt;font color=blue&gt;BLUE&lt;/font&gt;&lt;/h1&gt;
</code></pre>
<p>You can play with</p>
<pre><code>curl -H &quot;Host: nginx.example.com&quot; http://PUBLIC_IP
curl -H &quot;Host: blue.nginx.example.com&quot; http://PUBLIC_IP
curl -H &quot;Host: green.nginx.example.com&quot; http://PUBLIC_IP
</code></pre>
<h2>Conclusion:</h2>
<ul>
<li>We only have 1 IP address and multiple hosts.</li>
<li>We have 3 different services using the same port.</li>
<li>The requests distribution is done using Ingress.</li>
</ul>
