<p>i have a web application running inside cluster ip on worker node on port 5001,i'm also using k3s for cluster deployment, i checked the cluster connection it's running fine</p>
<p><a href="https://i.stack.imgur.com/7OK5O.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/7OK5O.png" alt="enter image description here" /></a></p>
<p>the deployment has the container port set to 5001:</p>
<pre><code>ports:
  - containerPort:5001
</code></pre>
<p>Here is the service file:</p>
<pre><code>apiVersion: v1
kind: Service
metadata:
  labels:
    io.kompose.service: user-ms
  name: user-ms
spec:
  ports:
    - name: http
      port: 80
      targetPort: 5001
  selector:
    io.kompose.service: user-ms
status:
  loadBalancer: {}
</code></pre>
<p>and here is the ingress file:</p>
<pre><code>apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
   name: user-ms-ingress
spec:
   rules:
   - http:
      paths:
      - path: / 
        pathType: Prefix
        backend:
           service:
             name: user-ms
             port: 
               number: 80
</code></pre>
<p>i'm getting <code>502 Bad Gateway</code> error whenever i type in my worker or master ip address</p>
<p>expected: it should return the web application page</p>
<p>i looked online and most of them mention wrong port for service and ingress, but my ports are correct yes i triple check it:</p>
<blockquote>
<p>try calling user-ms service on port 80 from another pod -&gt; worked try
calling cluster ip on worker node on port 5001 -&gt; worked</p>
</blockquote>
<p>the ports are running correct, why is the ingress returning 502?</p>
<p>here is the ingress describe:</p>
<p><a href="https://i.stack.imgur.com/68dzX.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/68dzX.png" alt="enter image description here" /></a></p>
<p>and here is the describe of nginx ingress controller pod:</p>
<p><a href="https://i.stack.imgur.com/wbbqS.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/wbbqS.png" alt="enter image description here" /></a>
<a href="https://i.stack.imgur.com/WDVRN.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/WDVRN.png" alt="enter image description here" /></a></p>
<p>the nginx ingress pod running normally:</p>
<p><a href="https://i.stack.imgur.com/m39g7.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/m39g7.png" alt="enter image description here" /></a></p>
<p>here is the logs of the nginx ingress pod:</p>
<p><a href="https://i.stack.imgur.com/Xdu7s.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Xdu7s.jpg" alt="enter image description here" /></a></p>
<p>sorry for the images, but i'm using a streaming machine to access the terminal so i can't copy paste</p>
<p>How should i go with debugging this error?</p>

Linh Nguyen
<p>ok i managed to figure out this, in the default setting of K3S it uses traefik as it default ingress, so that why my nginx ingress log doesn't show anything from 502 Bad GateWay</p>
<p>I decided to tear down my cluster and set it up again, now with suggestion from this issue <a href="https://github.com/k3s-io/k3s/issues/1160#issuecomment-1058846505" rel="nofollow noreferrer">https://github.com/k3s-io/k3s/issues/1160#issuecomment-1058846505</a> to create cluster without traefik:</p>
<pre><code>curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC=&quot;--disable=traefik&quot; sh -
</code></pre>
<p>now when i call <code>kubectl get pods --all-namespaces</code> i no longer see traefik pod running, previously it had traefik pods runining.</p>
<p>once i done all of it, run apply on ingress once again -&gt; get 404 error, i checked in the nginx ingress pod logs now it's showing new error of missing Ingress class, i add the following to my ingress configuration file under <code>metadata</code>:</p>
<pre><code>metadata:
  name: user-ms-ingress
  annotitations:
    kubernetes.io/ingress.class: &quot;nginx&quot;
</code></pre>
<p>now i once more go to the ip of the worker node -&gt; 404 error gone but got 502 bad gateway error, i checked the logs get connection refused errors:</p>
<p><a href="https://i.stack.imgur.com/5b5sB.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/5b5sB.png" alt="enter image description here" /></a></p>
<p>i figured out that i was setting a <code>network policy</code> for all of my micro services, i delete the <code>network policy</code> and remove it's setting from all my deployment files.</p>
<p>Finally check once more and i can see that i can access my api and swagger page normally.</p>
<p>TLDR:</p>
<blockquote>
<ol>
<li>If you are using nginx ingress on K3S, remember to disable traefik first when created the cluster</li>
<li>don't forget to set ingress class inside your ingress configuration</li>
<li>don't setup network policy because nginx pod won't be able to call the other pods in that network</li>
</ol>
</blockquote>
