<p>I'm running istio 1.0.6 (no mTLS) on our EKS cluster and am experiencing connection issues between two pods that I can't seem to figure out. </p>

<p>I have a gRPC client (<code>abc-client</code>) and server (<code>abc-server</code>) running both written in go. The cluster has automatic sidecar injection enabled, so both have it running.<br>
Everything is working fine if I just start the two services, but whenever the server restarts, the client is not able to reconnect to it. From the restart onwards I continuously get the following error message on the client: </p>

<p><code>{"level":"error","error":"rpc error: code = Unavailable desc = upstream connect error or disconnect/reset before headers","time":"2019-03-19T12:07:52Z","message":"abc-server service unhealthy"}</code></p>

<p>Without the istio sidecar the reconnection is working immediately. 
When I look at the sidecar logs of the <code>abc-client</code> I see a lot of the following messages for the health checks the client is trying to do:</p>

<p><code>2019-03-19T12:09:41.527Z] "POST /grpc.health.v1.Health/CheckHTTP/2" 200 UF 5 0 1002 - "-" "grpc-go/1.15.0" "6a874d67-9c43-4b67-ad24-7337a26cac8d" "abc-server.abc-namespace:8086" "172.31.18.194:8086" outbound|8086||abc-server.abc-namespace.svc.cluster.local - 172.31.18.194:8086 172.31.22.131:44994</code></p>

<p>My takeaway from these logs is that the proxy is experiencing problems connecting to the upstream server that is not there anymore, hence the <code>UF</code> flag (upstream failure) but is still returning a <code>200</code> status code to the client. To me it seems that this behaviour doesn't trigger a reconnection. </p>

<p>I have not any <code>VirtualService</code>s or <code>DestinationRule</code>s set up, it's just a simple istio cluster with the sidecars enabled. </p>

<p>What am I missing? Thanks in advance for your help. </p>

Memorex42
<p>The problem is connecting to headless services. My <code>service.yaml</code> looked like this:</p>

<pre><code>apiVersion: v1
kind: Service
metadata:
  name: grpc-test-server
spec:
  type: ClusterIP
  clusterIP: None
  selector:
    app: grpc-test-server
    role: grpc-test-server
  ports:
    - name: grpc
      port: 9999
      targetPort: grpc
</code></pre>

<p>As per the issues <a href="https://github.com/istio/istio/issues/10659" rel="nofollow noreferrer">#10659</a> and <a href="https://github.com/istio/istio/issues/7495" rel="nofollow noreferrer">#7495</a> istio doesn't work yet with headless services. </p>

<p>So, if you don't need your service to be headless, you can just remove <code>clusterIP: None</code> from the yaml and everything should work as expected. </p>

<p>I still don't know though how to make statefulsets work which <a href="https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#limitations" rel="nofollow noreferrer">require</a> headless services. </p>
