<p>I'm on MacOS and using <code>kind v0.10.0</code>. I've followed the instructions given at <a href="https://kind.sigs.k8s.io/docs/user/ingress/" rel="nofollow noreferrer">https://kind.sigs.k8s.io/docs/user/ingress/</a> for setting up <code>kind</code> with an <code>nginx</code> ingress controller (and a private docker registry). Unfortunately, I am unable to connect to be ingress controller on <code>127.0.0.1:80</code>. I've even tried moving my service, deployment, and ingress into the same namespace as the nginx ingress controller, i.e. <code>namespace=ingress-nginx</code>, but even that didn't fix anything.</p>
<p><strong>Here's what I'm trying to do:</strong> route incoming gRPC messages to two separate backend services based on the gRPC method name.</p>
<p><strong>Here's what I'm observing:</strong></p>
<ul>
<li><p>If I open <a href="http://127.0.0.1" rel="nofollow noreferrer">http://127.0.0.1</a> in my browser I see a <strong>502</strong> page served by <code>nginx</code>.</p>
</li>
<li><p>I have even confirmed that the process listening to port 80 is <code>com.docker.backend</code>.</p>
</li>
<li><p>However, when I try to connect to this port using a gRPC client, it just hangs/timesout:</p>
<pre><code>% ~/go/bin/grpcurl -plaintext localhost:80 list
Failed to dial target host &quot;localhost:80&quot;: context deadline exceeded
</code></pre>
</li>
</ul>
<p>Can anyone help me debug this?</p>
<h2>Pods</h2>
<pre><code>% kubectl get pods -n ingress-nginx                         
NAME                                       READY   STATUS      RESTARTS   AGE
ingress-nginx-admission-create-b8tkt       0/1     Completed   0          13h
ingress-nginx-admission-patch-xddd7        0/1     Completed   2          13h
ingress-nginx-controller-9ddc777c5-bwnkx   1/1     Running     0          13h
route-guide-a-6fc997cbd-7zgbk              1/1     Running     0          10m
route-guide-a-6fc997cbd-hgpjv              1/1     Running     0          10m
route-guide-b-7b954b77bb-2p8vm             1/1     Running     0          10m
route-guide-b-7b954b77bb-nmdnt             1/1     Running     0          10m
</code></pre>
<h2>Deployments</h2>
<pre><code>% kubectl get deployment -n ingress-nginx
NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
ingress-nginx-controller   1/1     1            1           13h
route-guide-a              2/2     2            2           10m
route-guide-b              2/2     2            2           10m
</code></pre>
<h2>Services</h2>
<pre><code>% kubectl get svc -n ingress-nginx
NAME                                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx-controller             NodePort    10.96.230.48    &lt;none&gt;        80:32236/TCP,443:32573/TCP   13h
ingress-nginx-controller-admission   ClusterIP   10.96.79.228    &lt;none&gt;        443/TCP                      13h
route-guide-a                        ClusterIP   10.96.249.171   &lt;none&gt;        10000/TCP                    11m
route-guide-b                        ClusterIP   10.96.143.194   &lt;none&gt;        10000/TCP                    11m
</code></pre>
<h2>Ingress</h2>
<pre><code>% kubectl get ingress -n ingress-nginx
NAME                  CLASS    HOSTS   ADDRESS     PORTS   AGE
route-guide-ingress   &lt;none&gt;   *       localhost   80      12m
</code></pre>
<h2>Config file</h2>
<pre><code>---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: route-guide-a
  namespace: ingress-nginx
  labels:
    app: route-guide-a
spec:
  replicas: 2
  selector:
    matchLabels:
      app: route-guide-a
  template:
    metadata:
      labels:
        app: route-guide-a

    spec:
      containers:
        - name: route-guide-a
          image: localhost:5000/route_guide:v4
          imagePullPolicy: IfNotPresent
          command: [&quot;/usr/local/go/bin/go&quot;, &quot;run&quot;, &quot;server/server.go&quot;]
          ports:
            - containerPort: 10000
              name: grpc-port
          env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: route-guide-b
  namespace: ingress-nginx
  labels:
    app: route-guide-b
spec:
  replicas: 2
  selector:
    matchLabels:
      app: route-guide-b
  template:
    metadata:
      labels:
        app: route-guide-b

    spec:
      containers:
        - name: route-guide-b
          image: localhost:5000/route_guide:v4
          imagePullPolicy: IfNotPresent
          command: [&quot;/usr/local/go/bin/go&quot;, &quot;run&quot;, &quot;server/server.go&quot;]
          ports:
            - containerPort: 10000
              name: grpc-port
          env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name

---
apiVersion: v1
kind: Service
metadata:
  name: route-guide-a
  namespace: ingress-nginx
  labels:
    app: route-guide-a
spec:
  # type: NodePort
  selector:
    app: route-guide-a
  ports:
    - port: 10000
      # nodePort: 31397
      targetPort: 10000
      name: grpc-port
      protocol: TCP

---
apiVersion: v1
kind: Service
metadata:
  name: route-guide-b
  namespace: ingress-nginx
  labels:
    app: route-guide-b
spec:
  # type: NodePort
  selector:
    app: route-guide-b
  ports:
    - port: 10000
      # nodePort: 31398
      targetPort: 10000
      name: grpc-port
      protocol: TCP

---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: &quot;nginx&quot;
    # nginx.ingress.kubernetes.io/ssl-redirect: &quot;true&quot;
    nginx.ingress.kubernetes.io/backend-protocol: &quot;GRPC&quot;
  name: route-guide-ingress
  namespace: ingress-nginx
spec:
  rules:
  - http:
      paths: 
        - path: /routeguide.RouteGuide/GetFeature
          backend:
            serviceName: route-guide-a
            servicePort: grpc-port

        - path: /routeguide.RouteGuide/ListFeatures
          backend:
            serviceName: route-guide-a
            servicePort: grpc-port

        - path: /routeguide.RouteGuide/RecordRoute
          backend:
            serviceName: route-guide-b
            servicePort: grpc-port
        
        - path: /routeguide.RouteGuide/RouteChat
          backend:
            serviceName: route-guide-b
            servicePort: grpc-port

        # catch-all route
        - backend:
            serviceName: route-guide-b
            servicePort: grpc-port

</code></pre>
<h2>Ingress config in the running cluster</h2>
<p>Output of the following command: <code>kubectl get ingress -n ingress-nginx route-guide-ingress -o yaml</code></p>
<pre><code>apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {&quot;apiVersion&quot;:&quot;networking.k8s.io/v1beta1&quot;,&quot;kind&quot;:&quot;Ingress&quot;,&quot;metadata&quot;:{&quot;annotations&quot;:{&quot;kubernetes.io/ingress.class&quot;:&quot;nginx&quot;,&quot;nginx.ingress.kubernetes.io/backend-protocol&quot;:&quot;GRPC&quot;},&quot;name&quot;:&quot;route-guide-ingress&quot;,&quot;namespace&quot;:&quot;ingress-nginx&quot;},&quot;spec&quot;:{&quot;rules&quot;:[{&quot;http&quot;:{&quot;paths&quot;:[{&quot;backend&quot;:{&quot;serviceName&quot;:&quot;route-guide-a&quot;,&quot;servicePort&quot;:&quot;grpc-port&quot;},&quot;path&quot;:&quot;/routeguide.RouteGuide/GetFeature&quot;},{&quot;backend&quot;:{&quot;serviceName&quot;:&quot;route-guide-a&quot;,&quot;servicePort&quot;:&quot;grpc-port&quot;},&quot;path&quot;:&quot;/routeguide.RouteGuide/ListFeatures&quot;},{&quot;backend&quot;:{&quot;serviceName&quot;:&quot;route-guide-b&quot;,&quot;servicePort&quot;:&quot;grpc-port&quot;},&quot;path&quot;:&quot;/routeguide.RouteGuide/RecordRoute&quot;},{&quot;backend&quot;:{&quot;serviceName&quot;:&quot;route-guide-b&quot;,&quot;servicePort&quot;:&quot;grpc-port&quot;},&quot;path&quot;:&quot;/routeguide.RouteGuide/RouteChat&quot;},{&quot;backend&quot;:{&quot;serviceName&quot;:&quot;route-guide-a&quot;,&quot;servicePort&quot;:&quot;grpc-port&quot;}}]}}]}}
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/backend-protocol: GRPC
  creationTimestamp: &quot;2021-05-02T02:57:24Z&quot;
  generation: 2
  managedFields:
  - apiVersion: networking.k8s.io/v1beta1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:kubectl.kubernetes.io/last-applied-configuration: {}
          f:kubernetes.io/ingress.class: {}
          f:nginx.ingress.kubernetes.io/backend-protocol: {}
      f:spec:
        f:rules: {}
    manager: kubectl-client-side-apply
    operation: Update
    time: &quot;2021-05-02T02:57:24Z&quot;
  - apiVersion: networking.k8s.io/v1beta1
    fieldsType: FieldsV1
    fieldsV1:
      f:status:
        f:loadBalancer:
          f:ingress: {}
    manager: nginx-ingress-controller
    operation: Update
    time: &quot;2021-05-02T02:57:44Z&quot;
  name: route-guide-ingress
  namespace: ingress-nginx
  resourceVersion: &quot;12115&quot;
  uid: be3862b4-13a7-4b96-b5f8-399e28bb373c
spec:
  rules:
  - http:
      paths:
      - backend:
          service:
            name: route-guide-a
            port:
              name: grpc-port
        path: /routeguide.RouteGuide/GetFeature
        pathType: ImplementationSpecific
      - backend:
          service:
            name: route-guide-a
            port:
              name: grpc-port
        path: /routeguide.RouteGuide/ListFeatures
        pathType: ImplementationSpecific
      - backend:
          service:
            name: route-guide-b
            port:
              name: grpc-port
        path: /routeguide.RouteGuide/RecordRoute
        pathType: ImplementationSpecific
      - backend:
          service:
            name: route-guide-b
            port:
              name: grpc-port
        path: /routeguide.RouteGuide/RouteChat
        pathType: ImplementationSpecific
      - backend:
          service:
            name: route-guide-a
            port:
              name: grpc-port
        pathType: ImplementationSpecific
status:
  loadBalancer:
    ingress:
    - hostname: localhost
</code></pre>

<p>According to <a href="https://github.com/kubernetes/ingress-nginx/issues/2444" rel="nofollow noreferrer">kubernetes/ingress-nginx#2444</a> and <a href="https://github.com/kubernetes/ingress-nginx/issues/6313" rel="nofollow noreferrer">kubernetes/ingress-nginx#6313</a>, Nginx cannot multiplex HTTP/1.1 and HTTP/2 in Port 80. As shown below, http2 is not enabled in Port 80 by default.</p>
<pre class="lang-sh prettyprint-override"><code>$ kubectl exec -it -n $INGRESS_CONTROLLER_NS $INGRESS_CONTROLLER_POD -- cat /etc/nginx/nginx.conf
...
server {
    server_name your.domain ;

    listen 80  ;
    listen [::]:80  ;
    listen 443  ssl http2 ;
    listen [::]:443  ssl http2 ;
}
...
</code></pre>
<p>As a result, some probable solutions could be:</p>
<ul>
<li>Use SSL instead.</li>
<li>Use a custom template as mentioned in the <a href="https://github.com/kubernetes/ingress-nginx/issues/2444#issuecomment-385864000" rel="nofollow noreferrer">comment</a>.</li>
</ul>
