<p>I deployed a gRPC service (spring boot docker image) in my on-premise kubernetes cluster. I followed this <a href="https://github.com/kubernetes/ingress-nginx/tree/master/docs/examples/grpc" rel="nofollow noreferrer">documentation</a> to configure correctly deployment, service and ingress kubernetes manifests.</p>

<p>I tried to test my service using grpcurl : </p>

<pre><code>grpcurl -insecure fortune-teller.mydomain.cloud:443 build.stack.fortune.FortuneTeller/Predict
</code></pre>

<p>and the request still stuck for minutes. </p>

<p>In the ingress logs (debug enabled), I see client timeout :</p>

<pre><code>client timed out (110: Connection timed out), client: 1.2.3.4, server: _, request: "POST /grpc.reflection.v1alpha.ServerReflection/ServerReflectionInfo HTTP/2.0", host: "fortune-teller.mydomain.cloud:443"
</code></pre>

<p>After 4 timeouts in the ingress logs, the command ends at the client side with :</p>

<pre><code>Error invoking method "build.stack.fortune.FortuneTeller/Predict": failed to query for service descriptor "build.stack.fortune.FortuneTeller": rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: PROTOCOL_ERROR
</code></pre>

<p>I use <code>nginx/1.13.12</code> with <code>rancher/nginx-ingress-controller:0.16.2-rancher1</code> image.</p>

<p>At the annotation level, I tested :</p>

<pre><code>kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
</code></pre>

<p>and</p>

<pre><code>nginx.ingress.kubernetes.io/grpc-backend: "true"
</code></pre>

<p>and both have the same result (timeout).</p>

<p><strong>Note</strong> : I bypassed the ingress testing another gRPC service directly using the kubernetes service DNS and it works.</p>

<p>Any idea of what's going wrong ?</p>

<p>I found a problem in my ingress resource. </p>

<p>Despite the <code>tls</code> configuration, the nginx config didn't listen on 443. Something wrong with my <code>secret</code></p>

<p>I fixed this and I retested my different configuration.</p>

<p>The working annotations are :</p>

<pre><code>metadata:
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/grpc-backend: "true"
</code></pre>

<p>Which are not the annotations used in the kubernetes <a href="https://github.com/kubernetes/ingress-nginx/tree/master/docs/examples/grpc" rel="nofollow noreferrer">documentation</a>.</p>
