<p>I have a gRPC server that works fine on my local machine. I can send grpc requests from a python app and get back the right responses.</p>
<p>I put the server into a GKE cluster (with only one node). I had a normal TCP load balancer in front of the cluster. In this setup my local client was able to get the correct response from some requests, but not others. I think it was the gRPC streaming that didn't work.</p>
<p>I assumed that this is because the streaming requires an HTTP/2 connection which requires SSL.</p>
<p>The standard load balancer I got in GKE didn't seem to support SSL, so I followed the docs to set up an ingress load balancer which does. I'm using a Lets-Encrypt certificate with it.</p>
<p>Now all gRPC requests return</p>
<blockquote>
<p>status = StatusCode.UNAVAILABLE</p>
<p>details = &quot;Socket closed&quot;</p>
<p>debug_error_string =
&quot;{&quot;created&quot;:&quot;@1556172211.931158414&quot;,&quot;description&quot;:&quot;Error received from
peer
ipv4:ip.of.ingress.service:443&quot;, &quot;file&quot;:&quot;src/core/lib/surface/call.cc&quot;, &quot;file_line&quot;:1041,&quot;grpc_message&quot;:&quot;Socket closed&quot;,&quot;grpc_status&quot;:14}&quot;</p>
</blockquote>
<p>The IP address is the external IP address of my ingress service.
The ingress yaml looks like this:</p>
<pre><code>apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: rev79-ingress
  annotations:
    kubernetes.io/ingress.global-static-ip-name: &quot;rev79-ip&quot;
    ingress.gcp.kubernetes.io/pre-shared-cert: &quot;lets-encrypt-rev79&quot;
    kubernetes.io/ingress.allow-http: &quot;false&quot; # disable HTTP
spec:
  rules:
  - host: sub-domain.domain.app
    http:
      paths:
      - path: /*
        backend:
          serviceName: sandbox-nodes
          servicePort: 60000
</code></pre>
<p>The subdomain and domain of the request from my python app match the host in the ingress rule.</p>
<p>It connects to a node-port that looks like this:</p>
<pre><code>apiVersion: v1
kind: Service
metadata:
  name: sandbox-nodes
spec:
  type: NodePort
  selector:
    app: rev79
    environment: sandbox
  ports:
  - protocol: TCP
    port: 60000
    targetPort: 9000
</code></pre>
<p>The node itself has two containers and looks like this:</p>
<pre><code>apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: rev79-sandbox
  labels:
    app: rev79
    environment: sandbox
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: rev79
        environment: sandbox
    spec:
      containers:
      - name: esp
        image: gcr.io/endpoints-release/endpoints-runtime:1.31
        args: [
          &quot;--http2_port=9000&quot;,
          &quot;--service=rev79.endpoints.rev79-232812.cloud.goog&quot;,
          &quot;--rollout_strategy=managed&quot;,
          &quot;--backend=grpc://0.0.0.0:3011&quot;
        ]
        ports:
        - containerPort: 9000
      - name: rev79-uac-sandbox
        image: gcr.io/rev79-232812/uac:latest
        imagePullPolicy: Always
        ports:
        - containerPort: 3011
        env:
        - name: RAILS_MASTER_KEY
          valueFrom:
            secretKeyRef:
              name: rev79-secrets
              key: rails-master-key
</code></pre>
<p>The target of the node port is the ESP container which connects to the gRPC service deployed in the cloud, and the backend which is a Rails app that implements the backend of the API. This rails app isn't running the rails server, but a specialised gRPC server that comes with the <code>grpc_for_rails</code> gem</p>
<p>The grpc_server in the Rails app doesn't record any action in the logs, so I don't think the request gets that far.</p>
<p><code>kubectl get ingress</code> reports this:</p>
<pre><code>NAME            HOSTS                   ADDRESS            PORTS   AGE
rev79-ingress   sub-domain.domain.app   my.static.ip.addr   80      7h
</code></pre>
<p>showing port 80, even though it's set up with SSL. That seems to be a bug. When I check with <code>curl -kv https://sub-domain.domain.app</code> the ingress server handles the request fine, and uses HTTP/2. It reurns an HTML formatted server error, but I'm not sure what generates that.</p>
<p>The API requires an API key, which the python client inserts into the metadata of each request.</p>
<p>When I go to the endpoints page of my GCP console I see that the API is not registering any requests since putting in the ingress loadbalancer, so it looks like the requests are not reaching the EPS container.</p>
<p>So why am I getting &quot;socket closed&quot; errors with gRPC?</p>

<p>I said I would come back and post an answer here once I got it working. It looks like I never did. Being a man of my word I'll post now my config files which are working for me.</p>
<p>in my deployment I've put a liveness and readiness probe for the ESP container. This made deployments happen smoothly without downtime:</p>
<pre><code>apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: rev79-sandbox
  labels:
    app: rev79
    environment: sandbox
spec:
  replicas: 3
  template:
    metadata:
      labels:
        app: rev79
        environment: sandbox
    spec:
      volumes:
      - name: nginx-ssl
        secret:
          secretName: nginx-ssl
      - name: gcs-creds
        secret:
          secretName: rev79-secrets
          items:
            - key: gcs-credentials
              path: &quot;gcs.json&quot;
      containers:
      - name: esp
        image: gcr.io/endpoints-release/endpoints-runtime:1.45
        args: [
          &quot;--http_port&quot;, &quot;8080&quot;,
          &quot;--ssl_port&quot;, &quot;443&quot;,
          &quot;--service&quot;, &quot;rev79-sandbox.endpoints.rev79-232812.cloud.goog&quot;,
          &quot;--rollout_strategy&quot;, &quot;managed&quot;,
          &quot;--backend&quot;, &quot;grpc://0.0.0.0:3011&quot;,
          &quot;--cors_preset&quot;, &quot;cors_with_regex&quot;,
          &quot;--cors_allow_origin_regex&quot;, &quot;.*&quot;,
          &quot;-z&quot;, &quot; &quot;
        ]
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 60
          timeoutSeconds: 5
          periodSeconds: 5
          failureThreshold: 3
        readinessProbe:
          httpGet:
            path: /healthz
            port: 8080
          timeoutSeconds: 5
          failureThreshold: 1
        volumeMounts:
        - name: nginx-ssl
          mountPath: /etc/nginx/ssl
          readOnly: true
        ports:
        - containerPort: 8080
        - containerPort: 443
          protocol: TCP
      - name: rev79-uac-sandbox
        image: gcr.io/rev79-232812/uac:29eff5e
        imagePullPolicy: Always
        volumeMounts:
          - name: gcs-creds
            mountPath: &quot;/app/creds&quot;
        ports:
        - containerPort: 3011
          name: end-grpc
        - containerPort: 3000
        env:
        - name: RAILS_MASTER_KEY
          valueFrom:
            secretKeyRef:
              name: rev79-secrets
              key: rails-master-key
</code></pre>
<p>This is my service config that exposes the deployment to the load balancer:</p>
<pre><code>apiVersion: v1
kind: Service
metadata:
  name: rev79-srv-ingress-sandbox
  labels:
    type: rev79-srv
  annotations:
    service.alpha.kubernetes.io/app-protocols: '{&quot;rev79&quot;:&quot;HTTP2&quot;}'
    cloud.google.com/neg: '{&quot;ingress&quot;: true}'
spec:
  type: NodePort 
  ports:
  - name: rev79
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: rev79
    environment: sandbox
</code></pre>
<p>And this is my ingress:</p>
<pre><code>apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: rev79-ingress
  annotations:
    kubernetes.io/ingress.global-static-ip-name: &quot;rev79-global-ip&quot;
spec:
  tls:
  - secretName: sandbox-api-rev79-app-tls
  rules:
  - host: sandbox-api.rev79.app
    http:
      paths:
      - backend:
          serviceName: rev79-srv-ingress-sandbox
          servicePort: 443
</code></pre>
<p>I'm using cert-manager to manage the certificates.</p>
<p>It was a long time agao now. I can't remember if there was anything else I did to solve the issue I was having</p>
