<p>We have configured MetalLB since our K8s cluster is hosted on bare metal infrastructure. It seems to be running fine with all pods up and running.</p>
<pre><code>[~]# kubectl get all -n metallb-system
NAME                             READY   STATUS    RESTARTS   AGE
pod/controller-b78574c59-47qfv   1/1     Running   0          24h
pod/speaker-4q2vm                1/1     Running   0          24h
pod/speaker-m8kwk                1/1     Running   0          24h
pod/speaker-t4rvs                1/1     Running   0          24h

NAME                     DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
daemonset.apps/speaker   3         3         3       3            3           kubernetes.io/os=linux   24h

NAME                         READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/controller   1/1     1            1           24h

NAME                                   DESIRED   CURRENT   READY   AGE
replicaset.apps/controller-b78574c59   1         1         1       24h
</code></pre>
<p>We have configured ingress controller via helm from  <a href="https://github.com/kubernetes/ingress-nginx/releases/tag/helm-chart-3.29.0" rel="nofollow noreferrer">https://github.com/kubernetes/ingress-nginx/releases/tag/helm-chart-3.29.0</a> and updating <code>hostNetwork</code>,<code>ingressClass</code>,<code>kind</code> to <code>true</code>,<code>ingress-nginx</code>,<code>DaemonSet</code> respectively in file <code>values.yaml</code>. The helm installation seems to have worked fine with all daemonset pods running and an LB ip provided to created ingress controller service.</p>
<pre><code>[~]# kubectl get all -n ingress-nginx
NAME                                            READY   STATUS    RESTARTS   AGE
pod/devingress-ingress-nginx-controller-c2x42   1/1     Running   0          18h
pod/devingress-ingress-nginx-controller-wtmgw   1/1     Running   0          18h

NAME                                                    TYPE           CLUSTER-IP       EXTERNAL-IP      PORT(S)                      AGE
service/devingress-ingress-nginx-controller             LoadBalancer     x.x.x.x         1.2.3.40     80:32386/TCP,443:30020/TCP   18h
service/devingress-ingress-nginx-controller-admission   ClusterIP        x.x.x.x           &lt;none&gt;        443/TCP                      18h

NAME                                                 DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
daemonset.apps/devingress-ingress-nginx-controller   2         2         2       2            2           kubernetes.io/os=linux   18h
</code></pre>
<p>Now we have deployed two pods namely <code>nginx</code> with <code>LoadBalancer</code> service type &amp; <code>nginx-deploy-main</code> with <code>ClusterIP</code> service type.</p>
<pre><code>[~]# kubectl get all -n default
NAME                                     READY   STATUS    RESTARTS   AGE
pod/nginx-854cf6b4d7-lv5ss               1/1     Running   0          18h
pod/nginx-deploy-main-6b5457fbb5-7tg9z   1/1     Running   0          18h

NAME                           TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)          AGE
service/nginx                  LoadBalancer   x.x.x.x        1.2.3.41       8080:31101/TCP    18h
service/nginx-deploy-main      ClusterIP      x.x.x.x          &lt;none&gt;           80/TCP          18h

NAME                                READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx               1/1     1            1           18h
deployment.apps/nginx-deploy-main   1/1     1            1           18h

NAME                                           DESIRED   CURRENT   READY   AGE
replicaset.apps/nginx-854cf6b4d7               1         1         1       18h
replicaset.apps/nginx-deploy-main-6b5457fbb5   1         1         1       18h
</code></pre>
<p>Below is the ingress resource setup to access <code>nginx-deploy-main</code>.</p>
<pre><code>apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: ingress-resource
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  ingressClassName: nginx
  rules:
  - host: nginx-main.int.org.com
    http:
      paths:
      - path: /
        backend:
          serviceName: nginx-deploy-main
          servicePort: 80
</code></pre>
<p>And the ingress resource seems to be created correctly pointing to <code>nginx-deploy-main</code> service.</p>
<pre><code>[~]# kubectl get ing -n default
NAME                 CLASS   HOSTS                           ADDRESS   PORTS   AGE
ingress-resource     nginx   nginx-main.int.org.com                    80      19h

[~]# kubectl describe ing/ingress-resource -n default
Name:             ingress-resource
Namespace:        default
Address:
Default backend:  default-http-backend:80 (&lt;none&gt;)
Rules:
  Host                           Path  Backends
  ----                           ----  --------
  nginx-main.int.org.com
                                 /   nginx-deploy-main:80 (x.x.x.x:80)
Annotations:                     kubernetes.io/ingress.class: nginx
Events:                          &lt;none&gt;
</code></pre>
<p>Outside of K8s cluster, we have nginx set up serving as reverse proxy with domain int.org.com resolution.</p>
<p>Below is the nginx configuration which should help me hit url <code>http://nginx-main.int.org.com</code> and get response but the response returned is 404.</p>
<pre><code>upstream nginx-main.int.org.com {
  server 1.2.3.40:80;     ## Ingress Controller Service IP
}

server {
  listen 80;
  server_name nginx-main.int.org.com;
  location / {
    proxy_pass http://nginx-main.int.org.com;
  }
}
</code></pre>
<p>Now when I try to access nginx pod (<strong>not nginx-main</strong>) using its <code>LoadBalancer</code> Service IP with below configuration , its able to provide response and works just fine</p>
<pre><code>upstream nginx.int.org.com {
  server 1.2.3.41:8080;
}

server {
  listen 80;
  server_name nginx.int.org.com;
  location / {
    proxy_pass http://nginx.int.org.com;
  }
}
</code></pre>
<p>Am I missing something here with regards to Ingress Controller or Resource. Port Forwarding works fine and am able to access services using the same.</p>
<p>This really is a blocker and any help or documentation reference would be really useful .</p>

Alim Azad
<p>We tried with another Ingress Controller i.e. <a href="https://github.com/nginxinc/kubernetes-ingress" rel="nofollow noreferrer">https://github.com/nginxinc/kubernetes-ingress</a> and were able to make it work .</p>
<p>Below were the steps done .</p>
<pre><code>[~] git clone https://github.com/nginxinc/kubernetes-ingress/
[~] cd kubernetes-ingress/deployments
[~] git checkout v1.11.1
[~] kubectl apply -f common/ns-and-sa.yaml
[~] kubectl apply -f rbac/rbac.yaml
[~] kubectl apply -f common/default-server-secret.yaml
[~] kubectl apply -f common/nginx-config.yaml
[~] kubectl apply -f common/ingress-class.yaml
</code></pre>
<p>Created daemon-set pods with  extra environment argument i.e. <code>--enable-custom-resources=false</code> added in yaml due to below issue in controller logs</p>
<p>Refer : <a href="https://stackoverflow.com/questions/61086542/kubernetes-cluster-working-but-getting-this-error-from-the-nginx-controller">Kubernetes cluster working but getting this error from the NGINX controller</a></p>
<pre><code>[~] kubectl apply -f daemon-set/nginx-ingress.yaml
[~] kubectl get all -n nginx-ingress -o wide
NAME                      READY   STATUS    RESTARTS   AGE     IP            NODE         NOMINATED NODE   READINESS GATES
pod/nginx-ingress-gd8gw   1/1     Running   0          3h55m   x.x.x.x      worker1          &lt;none&gt;           &lt;none&gt;
pod/nginx-ingress-kr9lx   1/1     Running   0          3h55m   x.x.x.x      worker2          &lt;none&gt;           &lt;none&gt;
 
NAME                           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE     CONTAINERS     IMAGES                                                  SELECTOR
daemonset.apps/nginx-ingress   2         2         2       2            2           &lt;none&gt;          5h14m   nginx-ingress   nginx/nginx-ingress:1.11.1   app=nginx-ingress
</code></pre>
<p>Hit respective worker nodes at port 80 and a 404 response means its working fine.</p>
<p>Deployed a sample application using github link <a href="https://github.com/vipin-k/Ingress-Controller-v1.9.0/blob/main/hotel.yml" rel="nofollow noreferrer">https://github.com/vipin-k/Ingress-Controller-v1.9.0/blob/main/hotel.yml</a> and updated host entry within Ingress object to <code>hotel.int.org.com</code></p>
<pre><code>[~] kubectl create -f hotel.yaml
[~] kubectl get all -n hotel -o wide
NAME                         READY   STATUS    RESTARTS   AGE     IP            NODE         NOMINATED NODE   READINESS GATES
pod/hotel-65d644c8f7-bj597   1/1     Running   0          3h51m   x.x.x.x     worker1          &lt;none&gt;           &lt;none&gt;
pod/hotel-65d644c8f7-csvgp   1/1     Running   0          3h51m   x.x.x.x     worker2          &lt;none&gt;           &lt;none&gt;
 
NAME                TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE     SELECTOR
service/hotel-svc   ClusterIP   x.x.x.x   &lt;none&gt;        80/TCP    3h51m   app=hotel
 
NAME                    READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS   IMAGES                                                    SELECTOR
deployment.apps/hotel   2/2     2            2           3h51m   hotel        nginxdemos/hello:plain-text   app=hotel
 
NAME                               DESIRED   CURRENT   READY   AGE     CONTAINERS   IMAGES                                                    SELECTOR
replicaset.apps/hotel-65d644c8f7   2         2         2       3h51m   hotel        nginxdemos/hello:plain-text   app=hotel,pod-template-hash=65d644c8f7

[~] kubectl get ing -n hotel
NAME            CLASS   HOSTS                       ADDRESS   PORTS   AGE
hotel-ingress   nginx   hotel.int.org.com            80      3h52m
[~] kubectl describe ing hotel-ingress -n hotel
Name:             hotel-ingress
Namespace:        hotel
Address:
Default backend:  default-http-backend:80 (&lt;error: endpoints &quot;default-http-backend&quot; not found&gt;) 
Rules:
  Host                       Path  Backends
  ----                       ----  --------
  hotel.int.org.com
                             /        hotel-svc:80 (x.x.x.x:80,x.x.x.x:80)
Annotations:                 Events:  &lt;none&gt;
</code></pre>
<p>Updated external nginx configuration with enabled domain resolution .</p>
<pre><code>upstream hotel.int.org.com {
  server 1.2.3.41:80;  #worker1
  server 1.2.3.42:80;  #worker2
}

server {
  listen 80;
  server_name hotel.int.org.com;
  location / {
    proxy_pass http://hotel.int.org.com;
  }
}
</code></pre>
<p>Restart nginx and verify able to access via browser its serving response from respective running hotel namespace daemonset pods.</p>
<pre><code>[~]# curl hotel.int.org.com
Server address: x.x.x.x:80
Server name: hotel-65d644c8f7-bj597
Date: 28/Apr/2021:05:47:15 +0000
URI: /
Request ID: 28d5cfab4ea28beea49879422b7e8f4c

[~]# curl hotel.int.org.com
Server address: x.x.x.x:80
Server name: hotel-65d644c8f7-csvgp
Date: 28/Apr/2021:05:52:06 +0000
URI: /
Request ID: 4135cacf83f8bf41c9677104500e610b
</code></pre>
<p>Exploring with MetalLB too and will post solution once its works</p>
