<p>ENV: <br />
k8s: v1.20.5 <br />
ingress-nginx: v1.6.4</p>
<p>I created ingress-nginx-controller from offical yaml: <a href="https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.6.4/deploy/static/provider/baremetal/deploy.yaml" rel="nofollow noreferrer">https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.6.4/deploy/static/provider/baremetal/deploy.yaml</a></p>
<p>and I changed the network type to hostnetwork:</p>
<pre class="lang-yaml prettyprint-override"><code>hostNetwork: true
</code></pre>
<p>Then I created a deployment to create a backend server.Below is the yaml file:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: Service
metadata:
  name: http-svc
spec:
  selector:
    app: http
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: http-deployment
  labels:
    app: http
spec:
  replicas: 2
  selector:
    matchLabels:
      app: http
  template:
    metadata:
      labels:
        app: http
    spec:
      containers:
      - name: http
        image: hashicorp/http-echo:alpine
        args: [&quot;-text&quot;, &quot;hello&quot;, &quot;-listen=:80&quot;]
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: test-ingress
spec:
  rules:
  - host: &quot;test.com&quot;
    http:
      paths:
      - pathType: Prefix
        path: &quot;/test&quot;
        backend:
          service:
            name: http-svc
            port:
              number: 80
</code></pre>
<p>Every things looks running fine, but I still got &quot;500 Internal Server Error&quot; when I access the web server via ingress-nginx.</p>
<p>Below is the info for the resources:</p>
<pre class="lang-bash prettyprint-override"><code>#kubectl describe ingress
Name:             test-ingress
Namespace:        default
Address:          
Default backend:  default-http-backend:80 (&lt;error: endpoints &quot;default-http-backend&quot; not found&gt;)
Rules:
  Host        Path  Backends
  ----        ----  --------
  test.com    
              /test   http-svc:80 (192.168.107.203:80,192.168.122.81:80)
Annotations:  &lt;none&gt;
Events:       &lt;none&gt;

#kubectl get svc
NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
http-svc     ClusterIP   10.100.58.107   &lt;none&gt;        80/TCP    48m
kubernetes   ClusterIP   10.96.0.1       &lt;none&gt;        443/TCP   283d

#kubectl get pods -owide
NAME                               READY   STATUS    RESTARTS   AGE   IP                NODE        NOMINATED NODE   READINESS GATES
http-deployment-764c4597c5-rdks7   1/1     Running   0          48m   192.168.122.81    k8s-node4   &lt;none&gt;           &lt;none&gt;
http-deployment-764c4597c5-rf99t   1/1     Running   0          48m   192.168.107.203   k8s-node3   &lt;none&gt;           &lt;none&gt;

#kubectl get pods -n ingress-nginx -owide
NAME                                        READY   STATUS      RESTARTS   AGE   IP                NODE        NOMINATED NODE   READINESS GATES
ingress-nginx-admission-create-kb64z        0/1     Completed   0          44m   192.168.107.204   k8s-node3   &lt;none&gt;           &lt;none&gt;
ingress-nginx-admission-patch-xmswb         0/1     Completed   1          44m   192.168.122.82    k8s-node4   &lt;none&gt;           &lt;none&gt;
ingress-nginx-controller-69695968f9-7dtxf   1/1     Running     0          44m   10.1.1.12         k8s-node2   &lt;none&gt;           &lt;none&gt;

</code></pre>
<p>I can directly acces my backend server via service ip, so I think the problem maybe is ingress-nginx can not forward the request to the backend?</p>
<p>Below is the log in ingress-nginx-controller:</p>
<pre><code>2023/03/19 07:31:54 [error] 25#25: *25199 could not find named location &quot;@custom_upstream-default-backend_404&quot;, client: 127.0.0.1, server: , request: &quot;GET /test HTTP/1.1&quot;, host: &quot;test.com&quot;
2023/03/19 07:31:54 [error] 26#26: *25201 could not find named location &quot;@custom_upstream-default-backend_404&quot;, client: 127.0.0.1, server: , request: &quot;GET / HTTP/1.0&quot;, host: &quot;test.com&quot;
10.1.1.11 - - [19/Mar/2023:07:31:54 +0000] &quot;GET /test HTTP/1.1&quot; 500 170 &quot;-&quot; &quot;curl/7.29.0&quot; 76 0.000 [upstream-default-backend] [] 127.0.0.1:8181 : 127.0.0.1:8181 0 : 170 0.000 : 0.000 500 : 500 28acaef695f43cac09e7dfc932511c92
</code></pre>
<p>Looks like it forwarded the request to localhost,but why? Did I miss something?</p>
<p>Below is the different respones from different way to access:</p>
<pre><code>#curl http://10.100.58.107
hello
#curl http://test.com/test
&lt;html&gt;
&lt;head&gt;&lt;title&gt;500 Internal Server Error&lt;/title&gt;&lt;/head&gt;
&lt;body&gt;
&lt;center&gt;&lt;h1&gt;500 Internal Server Error&lt;/h1&gt;&lt;/center&gt;
&lt;hr&gt;&lt;center&gt;nginx&lt;/center&gt;
&lt;/body&gt;
&lt;/html&gt;
</code></pre>
<p>I want to access the backend server via ingress-nginx successfully.</p>

<p><strong>Debug pointers:</strong></p>
<ul>
<li>Application Level
<ul>
<li>Check if you have a <code>/test</code> endpoint in your service what does it respond with. Because based on your test requests, with the IP you made request to <code>/</code> and with DNS you made a request to <code>/test</code></li>
</ul>
</li>
<li>K8s Service Level
<ul>
<li>Port forward the service and try to run it on your local machine.
<pre><code>kubectl port-forward svc/http-svc 80:80
</code></pre>
</li>
</ul>
</li>
<li>Ingress Level
<ul>
<li>Is the error thrown by Nginx? If yes, then nginx ingress controller is throwing the error.</li>
<li>Check if you have used the proper ingress class name in your ingress object based on what you configured while deploying your nginx ingress controller.</li>
<li>As visible in your logs it says <code>[upstream-default-backend]</code> , that I feel based on my understanding once the controller receives the request it doesn't have any configured server configuration for this host.
<ul>
<li>This happens because nginx ingress controller only adds the server configuration of the hosts that happens to have an ingress class in their ingress object.</li>
</ul>
</li>
</ul>
</li>
</ul>
