<p>I'm trying to communicate via grpc between two microservices internally on kubernetes, but I'm getting a connection refused error.</p>
<p>These are the yaml files of the services that are trying to communicate.</p>
<pre class="lang-yaml prettyprint-override"><code>---
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: source-api
    app.kubernetes.io/part-of: liop
    app.kubernetes.io/version: latest
  name: source-api
spec:
  ports:
  - name: grpc-server
    port: 8081
    protocol: TCP
    targetPort: 8081
  - name: http
    port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    app.kubernetes.io/name: source-api
    app.kubernetes.io/part-of: liop
    app.kubernetes.io/version: latest
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/name: source-api
    app.kubernetes.io/part-of: liop
    app.kubernetes.io/version: latest
  name: source-api
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: source-api
      app.kubernetes.io/part-of: liop
      app.kubernetes.io/version: latest
  template:
    metadata:
      labels:
        app.kubernetes.io/name: source-api
        app.kubernetes.io/part-of: liop
        app.kubernetes.io/version: latest
    spec:
      containers:
      - env:
        - name: QUARKUS_GRPC_CLIENTS_STORE_PORT
          value: &quot;8081&quot;
        - name: QUARKUS_DATASOURCE_PASSWORD
          valueFrom:
            secretKeyRef:
              key: datasourcePassword
              name: liop
        - name: KAFKA_BOOTSTRAP_SERVERS
          value: kafka-service:9092
        - name: QUARKUS_DATASOURCE_USERNAME
          valueFrom:
            secretKeyRef:
              key: datasourceUsername
              name: liop
        - name: QUARKUS_HTTP_PORT
          value: &quot;8080&quot;
        - name: QUARKUS_GRPC_SERVER_PORT
          value: &quot;8081&quot;
        - name: QUARKUS_GRPC_SERVER_HOST
          value: localhost
        - name: QUARKUS_DATASOURCE_JDBC_URL
          value: jdbc:mysql://mysql:3306/product
        - name: QUARKUS_GRPC_CLIENTS_STORE_HOST
          value: store-api
        image: tools_source-api:latest
        imagePullPolicy: Never
        name: source-api
        ports:
        - containerPort: 8081
          name: grpc-server
          protocol: TCP
        - containerPort: 8080
          name: http
          protocol: TCP
      imagePullSecrets:
      - name: gitlab-registry

---
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: product-api
    app.kubernetes.io/part-of: liop
    app.kubernetes.io/version: latest
  name: product-api
spec:
  ports:
  - name: http
    port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    app.kubernetes.io/name: product-api
    app.kubernetes.io/part-of: liop
    app.kubernetes.io/version: latest
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/name: product-api
    app.kubernetes.io/part-of: liop
    app.kubernetes.io/version: latest
  name: product-api
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: product-api
      app.kubernetes.io/part-of: liop
      app.kubernetes.io/version: latest
  template:
    metadata:
      labels:
        app.kubernetes.io/name: product-api
        app.kubernetes.io/part-of: liop
        app.kubernetes.io/version: latest
    spec:
      containers:
      - env:
        - name: KAFKA_BOOTSTRAP_SERVERS
          value: kafka-service:9092
        - name: QUARKUS_DATASOURCE_JDBC_URL
          value: jdbc:mysql://mysql:3306/product
        - name: QUARKUS_GRPC_CLIENTS_IMAGE_PORT
          value: &quot;8081&quot;
        - name: QUARKUS_GRPC_CLIENTS_SOURCE_HOST
          value: source-api
        - name: QUARKUS_DATASOURCE_PASSWORD
          valueFrom:
            secretKeyRef:
              key: datasourcePassword
              name: liop
        - name: QUARKUS_DATASOURCE_USERNAME
          valueFrom:
            secretKeyRef:
              key: datasourceUsername
              name: liop
        - name: QUARKUS_GRPC_CLIENTS_SOURCE_PORT
          value: &quot;8081&quot;
        - name: QUARKUS_GRPC_CLIENTS_IMAGE_HOST
          value: media-api
        image: tools_product-api:latest
        imagePullPolicy: Always
        name: product-api
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP
      imagePullSecrets:
      - name: gitlab-registry
</code></pre>
<p>This is the yaml of my API-gateway, which does correctly communicate via HTTP with the microservices:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/version: latest
    app.kubernetes.io/part-of: liop
    app.kubernetes.io/name: api-gateway
  name: api-gateway
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/version: latest
      app.kubernetes.io/part-of: liop
      app.kubernetes.io/name: api-gateway
  template:
    metadata:
      labels:
        app.kubernetes.io/version: latest
        app.kubernetes.io/part-of: liop
        app.kubernetes.io/name: api-gateway
    spec:
      containers:
      - env:
        - name: product_api
          value: http://product-api:8080/api/products/v1/
        - name: source_api
          value: http://source-api:8080/api/sources/v1/
        - name: store_api
          value: http://store-api:8080/api/stores/v1/
        - name: report_api
          value: http://report-api:8080/api/reports/v1/
        - name: category_api
          value: http://category-api:8080/api/categories/v1/
        - name: AUTH0_ISSUER_URL
          value: xxxx
        - name: AUTH0_AUDIENCE
          value: xxxxxxx
        - name: PORT
          value: &quot;7000&quot;
        image: tools_webgateway:latest
        imagePullPolicy: Never
        name: api-gateway
        ports:
        - containerPort: 7000
          hostPort: 7000
          name: http
          protocol: TCP
      imagePullSecrets:
      - name: gitlab-registry
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: api-gateway
    app.kubernetes.io/part-of: liop
    app.kubernetes.io/version: latest
  name: api-gateway
spec:
  ports:
  - name: http
    port: 7000
    protocol: TCP
    targetPort: 7000
  selector:
    app.kubernetes.io/name: api-gateway
    app.kubernetes.io/part-of: liop
    app.kubernetes.io/version: latest
</code></pre>
<p>Error the product-api throws:</p>
<pre><code>Caused by: java.net.ConnectException: Connection refused
        at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
        at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:834)
io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
        at io.grpc.Status.asRuntimeException(Status.java:533)
        at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:478)
        at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
        at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
        at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
        at io.quarkus.grpc.runtime.supports.IOThreadClientInterceptor$1$1.onClose(IOThreadClientInterceptor.java:68)
        at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:617)
        at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:803)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:782)
        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: source-api/10.101.237.82:8081
Caused by: java.net.ConnectException: Connection refused
        at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
        at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:834)
</code></pre>
<p>The source-api:</p>
<pre><code>source-api                ClusterIP      10.101.237.82    &lt;none&gt;          8081/TCP,8080/TCP
</code></pre>
<pre><code>2021-04-01 06:38:08,973 INFO  [io.qua.grp.run.GrpcServerRecorder] (vert.x-eventloop-thread-1) gRPC Server started on localhost:8081 [SSL enabled: false]
</code></pre>
<p>Using grpcurl internally also gives me a connection refused error. But port forwarding the source-api:8081 does allow me to do requests.</p>

MrDoekje
<p>QUARKUS_GRPC_SERVER_HOST should be 0.0.0.0 instead of localhost</p>
