<p>Currently I have a cluster with single controller and single worker, also a nginx reverse-proxy (hhtp only) outside cluster.</p>
<p>Controller is at 192.168.1.65
worker is at 192.168.1.61
reverse proxy at 192.168.1.93 and public ip</p>
<p>here is my ingress-nginx services</p>
<pre><code> bino@corobalap  ~/k0s-sriwijaya/ingress-nginx/testapp  kubectl -n ingress-nginx get services        
NAME                                 TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                      AGE
ingress-nginx-controller             LoadBalancer   10.102.58.7     192.168.1.186   80:31097/TCP,443:31116/TCP   56m
ingress-nginx-controller-admission   ClusterIP      10.108.233.49   &lt;none&gt;          443/TCP                      56m

 bino@corobalap  ~/k0s-sriwijaya/ingress-nginx/testapp  kubectl -n ingress-nginx describe svc ingress-nginx-controller 
Name:                     ingress-nginx-controller
Namespace:                ingress-nginx
Labels:                   app.kubernetes.io/component=controller
                          app.kubernetes.io/instance=ingress-nginx
                          app.kubernetes.io/name=ingress-nginx
                          app.kubernetes.io/part-of=ingress-nginx
                          app.kubernetes.io/version=1.3.0
Annotations:              &lt;none&gt;
Selector:                 app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.102.58.7
IPs:                      10.102.58.7
LoadBalancer Ingress:     192.168.1.186
Port:                     http  80/TCP
TargetPort:               http/TCP
NodePort:                 http  31097/TCP
Endpoints:                10.244.0.23:80
Port:                     https  443/TCP
TargetPort:               https/TCP
NodePort:                 https  31116/TCP
Endpoints:                10.244.0.23:443
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   &lt;none&gt;
</code></pre>
<p>that 192.168.1.186 is assigned by MetalLB.</p>
<pre><code> bino@corobalap  ~/k0s-sriwijaya/ingress-nginx/testapp  kubectl get IPAddressPools -A
NAMESPACE        NAME     AGE
metallb-system   pool01   99m
 bino@corobalap  ~/k0s-sriwijaya/ingress-nginx/testapp  kubectl -n metallb-system describe IPAddressPool pool01
Name:         pool01
Namespace:    metallb-system
Labels:       &lt;none&gt;
Annotations:  &lt;none&gt;
API Version:  metallb.io/v1beta1
Kind:         IPAddressPool
Metadata:
  Creation Timestamp:  2022-07-26T09:08:10Z
  Generation:          1
  Managed Fields:
    API Version:  metallb.io/v1beta1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:addresses:
        f:autoAssign:
        f:avoidBuggyIPs:
    Manager:         kubectl-client-side-apply
    Operation:       Update
    Time:            2022-07-26T09:08:10Z
  Resource Version:  41021
  UID:               2a0dcfb2-bf8f-4b1a-b459-380e78959586
Spec:
  Addresses:
    192.168.1.186 - 192.168.1.191
  Auto Assign:       true
  Avoid Buggy I Ps:  false
Events:              &lt;none&gt;
</code></pre>
<p>I deploy hello-app at namespace : 'dev'</p>
<pre><code> bino@corobalap  ~/k0s-sriwijaya/ingress-nginx/testapp  kubectl -n dev get all
NAME                             READY   STATUS    RESTARTS      AGE
pod/hello-app-5c554f556c-v2gx9   1/1     Running   1 (20m ago)   63m

NAME                    TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
service/hello-service   ClusterIP   10.111.161.2   &lt;none&gt;        8081/TCP   62m

NAME                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/hello-app   1/1     1            1           63m

NAME                                   DESIRED   CURRENT   READY   AGE
replicaset.apps/hello-app-5c554f556c   1         1         1       63m


 bino@corobalap  ~/k0s-sriwijaya/ingress-nginx/testapp  kubectl -n dev describe service hello-service
Name:              hello-service
Namespace:         dev
Labels:            app=hello
Annotations:       &lt;none&gt;
Selector:          app=hello
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.111.161.2
IPs:               10.111.161.2
Port:              &lt;unset&gt;  8081/TCP
TargetPort:        8080/TCP
Endpoints:         10.244.0.22:8080
Session Affinity:  None
Events:            &lt;none&gt;
</code></pre>
<p>Local tests of that service:</p>
<pre><code>bino@k8s-worker-1:~$ curl http://10.111.161.2:8081
Hello, world!
Version: 2.0.0
Hostname: hello-app-5c554f556c-v2gx9
bino@k8s-worker-1:~$ curl http://10.244.0.22:8080
Hello, world!
Version: 2.0.0
Hostname: hello-app-5c554f556c-v2gx9
</code></pre>
<p>and the ingress resource of that service:</p>
<pre><code> bino@corobalap  ~/k0s-sriwijaya/ingress-nginx/testapp  kubectl -n dev describe  ingress hello-app-ingress
Name:             hello-app-ingress
Labels:           &lt;none&gt;
Namespace:        dev
Address:          192.168.1.61
Ingress Class:    nginx
Default backend:  &lt;default&gt;
Rules:
  Host                  Path  Backends
  ----                  ----  --------
  bino.k8s.jcamp.cloud  
                        /   hello-service:8081 (10.244.0.22:8080)
Annotations:            ingress.kubernetes.io/rewrite-target: /
Events:
  Type    Reason  Age                From                      Message
  ----    ------  ----               ----                      -------
  Normal  Sync    23m (x3 over 24m)  nginx-ingress-controller  Scheduled for sync
</code></pre>
<p>When I open <a href="http://bino.k8s.jcamp.cloud" rel="nofollow noreferrer">http://bino.k8s.jcamp.cloud</a> I got 502</p>
<p>my nginx reverse proxy conf :</p>
<pre><code>server {
    listen 80 default_server;
    location / {
        proxy_set_header Host            $host;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_pass http://192.168.1.186;
    }
}
</code></pre>
<p>The nginx error log say</p>
<pre><code>2022/07/26 06:24:21 [error] 1593#1593: *6 connect() failed (113: No route to host) while connecting to upstream, client: 203.161.185.210, server: , request: &quot;GET /favicon.ico HTTP/1.1&quot;, upstream: &quot;http://192.168.1.186:80/favicon.ico&quot;, host: &quot;bino.k8s.jcamp.cloud&quot;, referrer: &quot;http://bino.k8s.jcamp.cloud/&quot;
</code></pre>
<p>from describe ingress-nginx-controller pod</p>
<pre><code> bino@corobalap  ~/k0s-sriwijaya/ingress-nginx/testapp  kubectl -n ingress-nginx describe pod ingress-nginx-controller-6dc865cd86-9fmsk 
Name:         ingress-nginx-controller-6dc865cd86-9fmsk
Namespace:    ingress-nginx
Priority:     0
Node:         k8s-worker-1/192.168.1.61
Start Time:   Tue, 26 Jul 2022 16:11:05 +0700
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=ingress-nginx
              app.kubernetes.io/name=ingress-nginx
              pod-template-hash=6dc865cd86
Annotations:  kubernetes.io/psp: 00-k0s-privileged
Status:       Running
IP:           10.244.0.23
IPs:
  IP:           10.244.0.23
Controlled By:  ReplicaSet/ingress-nginx-controller-6dc865cd86
Containers:
  controller:
    Container ID:  containerd://541446c98b55312376aba4744891baa325dca26410abe5f94707d270d378d881
    Image:         registry.k8s.io/ingress-nginx/controller:v1.3.0@sha256:d1707ca76d3b044ab8a28277a2466a02100ee9f58a86af1535a3edf9323ea1b5
    Image ID:      registry.k8s.io/ingress-nginx/controller@sha256:d1707ca76d3b044ab8a28277a2466a02100ee9f58a86af1535a3edf9323ea1b5
    Ports:         80/TCP, 443/TCP, 8443/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP
    Args:
      /nginx-ingress-controller
      --election-id=ingress-controller-leader
      --controller-class=k8s.io/ingress-nginx
      --ingress-class=nginx
      --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
      --validating-webhook=:8443
      --validating-webhook-certificate=/usr/local/certificates/cert
      --validating-webhook-key=/usr/local/certificates/key
    State:          Running
      Started:      Tue, 26 Jul 2022 16:56:40 +0700
    Last State:     Terminated
      Reason:       Unknown
      Exit Code:    255
      Started:      Tue, 26 Jul 2022 16:11:09 +0700
      Finished:     Tue, 26 Jul 2022 16:56:26 +0700
    Ready:          True
    Restart Count:  1
    Requests:
      cpu:      100m
      memory:   90Mi
    Liveness:   http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
    Readiness:  http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:       ingress-nginx-controller-6dc865cd86-9fmsk (v1:metadata.name)
      POD_NAMESPACE:  ingress-nginx (v1:metadata.namespace)
      LD_PRELOAD:     /usr/local/lib/libmimalloc.so
    Mounts:
      /usr/local/certificates/ from webhook-cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-nfmrc (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  webhook-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ingress-nginx-admission
    Optional:    false
  kube-api-access-nfmrc:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       &lt;nil&gt;
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason          Age                From                      Message
  ----     ------          ----               ----                      -------
  Warning  NodeNotReady    44m                node-controller           Node is not ready
  Warning  FailedMount     43m                kubelet                   MountVolume.SetUp failed for volume &quot;webhook-cert&quot; : object &quot;ingress-nginx&quot;/&quot;ingress-nginx-admission&quot; not registered
  Warning  FailedMount     43m                kubelet                   MountVolume.SetUp failed for volume &quot;webhook-cert&quot; : failed to sync secret cache: timed out waiting for the condition
  Warning  FailedMount     43m                kubelet                   MountVolume.SetUp failed for volume &quot;kube-api-access-nfmrc&quot; : failed to sync configmap cache: timed out waiting for the condition
  Normal   SandboxChanged  43m                kubelet                   Pod sandbox changed, it will be killed and re-created.
  Normal   Pulled          43m                kubelet                   Container image &quot;registry.k8s.io/ingress-nginx/controller:v1.3.0@sha256:d1707ca76d3b044ab8a28277a2466a02100ee9f58a86af1535a3edf9323ea1b5&quot; already present on machine
  Normal   Created         43m                kubelet                   Created container controller
  Normal   Started         43m                kubelet                   Started container controller
  Warning  Unhealthy       42m (x2 over 42m)  kubelet                   Liveness probe failed: Get &quot;http://10.244.0.23:10254/healthz&quot;: dial tcp 10.244.0.23:10254: connect: connection refused
  Warning  Unhealthy       42m (x3 over 43m)  kubelet                   Readiness probe failed: Get &quot;http://10.244.0.23:10254/healthz&quot;: dial tcp 10.244.0.23:10254: connect: connection refused
  Normal   RELOAD          42m                nginx-ingress-controller  NGINX reload triggered due to a change in configuration
</code></pre>
<p>and here is the nft ruleset</p>
<pre><code>bino@k8s-worker-1:~$ su -
Password: 
root@k8s-worker-1:~# systemctl status nftables.service 
● nftables.service - nftables
     Loaded: loaded (/lib/systemd/system/nftables.service; enabled; vendor preset: enabled)
     Active: active (exited) since Tue 2022-07-26 05:56:17 EDT; 46min ago
       Docs: man:nft(8)
             http://wiki.nftables.org
    Process: 186 ExecStart=/usr/sbin/nft -f /etc/nftables.conf (code=exited, status=0/SUCCESS)
   Main PID: 186 (code=exited, status=0/SUCCESS)
        CPU: 34ms

Warning: journal has been rotated since unit was started, output may be incomplete.
</code></pre>
<p>[]</p>
<p>Complete ruleset is at <a href="https://pastebin.com/xd58rcQp" rel="nofollow noreferrer">https://pastebin.com/xd58rcQp</a></p>
<p>Kindly please tell me what to do, to check, or to learn for fixing this problem</p>
<p>Sincerely
-bino-</p>

Bino Oetomo
<p>my bad ...
There is a name mismatch between ip pool devinition yaml and the l2 advertisement yaml.</p>
