<p>I trying to debug it for few hours to no effect. I either do know how to google it or it unusual issue.</p>

<p>I have Kubernetes Engine cluster on google cloud. To test it I deploy simple app:
<a href="https://github.com/iniside/uSpark" rel="nofollow noreferrer">https://github.com/iniside/uSpark</a>
It is composed from three console apps:</p>

<ul>
<li>GreeterServer - Frontend with gRPC and Google Endpoints</li>
<li>Greeter.Backend - Backend with gRPC service.</li>
<li>uSpark - client console app.</li>
</ul>

<p>All of them are using .NET Core 2.1, gRPC 1.12, and C#. Kubernetes Engine is 1.10</p>

<p>I already tried to see if the service dns is resolved inside cluster.
I also tried to call service directly by it's IP when creating Channel.</p>

<p>Either of those end up with:</p>

<pre><code>Grpc.Core.RpcException: 'Status(StatusCode=Unavailable, Detail="Name resolution failure")'
</code></pre>

<p>GreeterServer is trying to talk to Greeter.Backend:</p>

<pre><code>public override Task&lt;HelloReply&gt; SayHelloAgain(HelloRequest request, ServerCallContext context)
{
    Channel channel = new Channel("grpc-greeter-backend.default.svc.cluster.local", 9000, ChannelCredentials.Insecure);
    var backendClient = new GreeterBackend.GreeterBackendClient(channel);

    var reply = backendClient.SayHelloFromBackend(new BackendHelloRequest { Name = "iniside" });
    channel.ShutdownAsync().Wait(); //not neeed to wait, but easier to debug now.
    return Task.FromResult&lt;HelloReply&gt;(new HelloReply { Message = "Hello Again " + request.Name + " " + reply.Message });
}
</code></pre>

<p>And this is code of my backend:</p>

<pre><code>class GreeterBackendImpl : GreeterBackend.GreeterBackendBase
{
    public override Task&lt;BackendHelloReply&gt; SayHelloFromBackend(BackendHelloRequest request, ServerCallContext context)
    {
        return Task.FromResult&lt;BackendHelloReply&gt;(new BackendHelloReply { Message = "Hello " + request.Name + " From Backend"});
    }
}

class Program
{
    static void Main(string[] args)
    {
        Console.WriteLine("Hello World!");

        Server server = new Server
        {
            Services = { GreeterBackend.BindService(new GreeterBackendImpl()) },
            Ports = { new ServerPort("127.0.0.1", 9000, ServerCredentials.Insecure) }
        };

        server.Start();

        Console.WriteLine("Gretter server is linstening on port 50051");
        Console.WriteLine("Press any key to stop server");

        int read = Console.Read();
        while (read &lt; 0)
        {

        }
    }
}
</code></pre>

<p>Configuration of Frontend:</p>

<pre><code>apiVersion: v1
kind: Service
metadata:
  name: esp-grpc-greeter
spec:
  ports:
  # Port that accepts gRPC and JSON/HTTP2 requests over HTTP.
  - port: 80
    targetPort: 9100
    protocol: TCP
    name: http2
  selector:
    app: esp-grpc-greeter
  type: LoadBalancer
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: esp-grpc-greeter
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: esp-grpc-greeter
    spec:
      containers:
      - name: esp
        image: gcr.io/endpoints-release/endpoints-runtime:1
        args: [
          "--http2_port=9100",
          "--service=hellohorld3.endpoints.sa-game-206414.cloud.goog",
          "--rollout_strategy=managed",
          "--backend=grpc://127.0.0.1:9000"
        ]
        ports:
          - containerPort: 9100
      - name: greeter
        image: eu.gcr.io/sa-game-206414/greeter-service:v1
        ports:
          - containerPort: 8000
</code></pre>

<p>Backend Configuration:</p>

<pre><code>apiVersion: v1
kind: Service
metadata:
  name: grpc-greeter-backend
spec:
  ports:
  # Port that accepts gRPC and JSON/HTTP2 requests over HTTP.
  - port: 8080
    targetPort: 9000
    protocol: TCP
    name: http2
  - port: 9000
    targetPort: 9000
    protocol: TCP
    name: http2900
  selector:
    app: grpc-greeter-backend
  type: ClusterIP
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: grpc-greeter-backend
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: grpc-greeter-backend
    spec:
      containers:
      - name: greeter-backend
        image: eu.gcr.io/sa-game-206414/greeter-backend:b1
        ports:
          - containerPort: 9000
</code></pre>

<p>I fixed issue and it was two fold. First as @spender suggested I did changed gRPC addresses to 0.0.0.0 both backend and frontend.</p>

<p>But the changes were not picked up by cluster. Because all this time it was pulling wrong images.</p>

<p>Every time I made changes, I deleted old images, with old tags (both locally and on GCR). Created new image with the same remote repository and tag and pushed them to GCR.</p>

<p>For some reason the images were not updated.</p>
