<p>I have two grpc services that communicate to each other using Istio service mesh and have envoy proxies for all the services.</p>
<p>We have an issue where during the scaling up of server pods due to high load, the client throws a few grpc UNAVAILABLE(mostly)/DEADLINE_EXCEEDED/CANCELLED errors for a while as soon as the new pod is ready.
I don't see any CPU throttling in server pods at all.</p>
<p>What else could be the issue and how can I investigate this?</p>

rishav
<p>Without the concrete status message, it's hard to say what could be the cause of the errors mentioned above.</p>
<p>To reduce UNAVAILABLE, one way to ask the RPC to wait-for-ready: <a href="https://github.com/grpc/grpc/blob/master/doc/wait-for-ready.md" rel="nofollow noreferrer">https://github.com/grpc/grpc/blob/master/doc/wait-for-ready.md</a>. This feature is available in all major gRPC languages (some may rename it to fail-fast=false).</p>
<p>DEADLINE_EXCEEDED is caused by the timeout set by your application or Envoy config, you should be able to tune it.</p>
<p>CANCELLED could mean: 1. the server is entering a graceful shutdown state; 2. the server is overloaded and rejecting new connections.</p>
