<p>I'm trying to use GRPC between my services. I have just three services. Two of them running in k8s. One in GCP. 
I make calls from the service in GCP to other services. I use the same NettyChannelBuilder for my stubs. The clients have the same requests per seconds rate. I've set keepAliveWithoutCalls=true, keepAliveTime, idleTimeout in the builder. 
Unfortunately I periodically get this error from only one client? </p>

<pre><code>io.grpc.StatusRuntimeException: UNAVAILABLE: upstream connect error or disconnect/reset before headers. reset reason: connection failure
    at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233)
    at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214)
</code></pre>

<p>Could anyone please help me to find solution?
I've read many issues on github already. Now, I'm going to set retry policy for calls but it's not the best solution.  </p>

<p>From <a href="https://github.com/grpc/grpc-web/issues/528" rel="nofollow noreferrer">github</a></p>
<blockquote>
<p>We faced the same error. The reason for us was because tcp_keepalive was set too high for our upstream service. We changed keepalive_time to 300 seconds and the problem went away.</p>
</blockquote>
<blockquote>
<p>It makes sense in our case because we have Envoy pointing to a network load balancer (aws) which has a 350s idle timeout.</p>
</blockquote>
<blockquote>
<p>We added this to our envoy config.</p>
</blockquote>
<pre class="lang-yaml prettyprint-override"><code>clusters:
    - name: grpc-service
      connect_timeout: 0.25s
      http2_protocol_options: {}
      upstream_connection_options:
        tcp_keepalive:
          keepalive_time: 300
</code></pre>
