<p>I've been trying to setup an externally facing GRPC payments microservice client with automatic cert renewal with tls.</p>

<p>So far I've set up the certmanager with the certificate renewal correctly however it appears my gateway is not forwarding traffic correctly as <code>kubectl -n istio-system describe challenge payments-cert</code> shows the challenge is erroring out due to <code>HTTP 404</code> being returned.</p>

<p>I used <code>grpcurl</code> to test the liveliness of the service, this was the result:</p>

<p><code>Failed to dial target host "payments.mywebsite.com:443": read tcp 192.168.0.16:58849 
-&gt;xx.xx.xxx.xx:443: read: connection reset by peer</code></p>

<p>Which indicates the ip has been registered by the dns correctly, and the address is indeed arriving on 443, so there must be an issue with my <code>Gateway -&gt; VirtualService -&gt; Service -&gt; Deployment</code> setup.</p>

<p>When describing the istio ingress (<code>kubectl get svc -n istio-system istio-ingressgateway</code>) I get:</p>

<p><code>istio-ingressgateway   LoadBalancer   10.0.13.184   xx.xx.xx.xx ...443:31390/TCP...</code></p>

<p>Which seems like its not forwarding <code>443</code> to the intended ports?</p>

<p>I want to have traffic coming in on 'payments.mywebsite.com:443' (with automatic certificate renewal with let-encrypt) to get routed to my grpc backend on container port <code>50051</code>. I have been having a very hard time deciphering the <code>Istio</code> docs and resources to get this done!</p>

<p>Any help will be warmly welcomed!</p>

<h1>Gateway</h1>

<pre><code>apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: payments-gateway
  namespace: default
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - payments.mywebsite.com
    port:
      name: payments-gateway-https
      number: 443
      protocol: HTTPS
    tls:
      credentialName: payments-cert
      mode: SIMPLE
      privateKey: sds
      serverCertificate: sds
</code></pre>

<h1>Virtual Service</h1>

<pre><code>apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: payments-virtual
spec:
  hosts:
  - "payments.mywebsite.com"
  gateways:
  - payments-gateway
  http:
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        port:
          number: 50051
        host: payments-service
</code></pre>

<h1>Service</h1>

<pre><code>apiVersion: v1
kind: Service
metadata:
  name: payments-service
  labels:
    app: payments-service
spec:
  ports:
  - port: 50051
    name: grpc
  selector:
    app: payments-server
</code></pre>

<h1>Deployment</h1>

<pre><code>apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: payments-server
  name: payments-server
spec:
  selector:
    matchLabels:
      app: payments-server
  replicas: 1
  strategy: {}
  template:
    metadata:
      labels:
        app: payments-server
        version: v1
    spec:
      containers:
      - image: gcr.io/mywebsite/payments_server:v1
        name: payments-server
        ports:
        - containerPort: 50051
        resources: {}
      restartPolicy: Always
status: {}
</code></pre>

<h1>Edit 1: Added logs from gateway</h1>

<pre><code>2020-01-21T14:31:48.841505Z     error   k8s.io/client-go@v11.0.1-0.20190409021438-1a26190bd76a+incompatible/tools/cache/reflector.go:98: Failed to list *v1.Secret: Get https://10.0.0.1:443/api/v1/namespaces/istio-system/secrets?limit=500&amp;resourceVersion=0: net/http: TLS handshake timeout
2020-01-21T14:32:13.173045Z     info    secretFetcherLog        scrtUpdated is called on kubernetes secret payments-cert
2020-01-21T14:32:13.173124Z     warn    secretFetcherLog        failed load server cert/key pair from secret payments-cert: server cert or private key is empty
2020-01-21T14:32:13.173131Z     warn    secretFetcherLog        failed load server cert/key pair from secret payments-cert: server cert or private key is empty
2020-01-21T14:32:13.173137Z     info    secretFetcherLog        secret payments-cert does not change, skip update
2020-01-21T14:32:13.173257Z     info    Trace[1103410]: "Reflector k8s.io/client-go@v11.0.1-0.20190409021438-1a26190bd76a+incompatible/tools/cache/reflector.go:98 ListAndWatch" (started: 2020-01-21 14:31:49.841728972 +0000 UTC m=+44333.359224094) (total time: 23.331269811s):`
</code></pre>

<h1>Edit 2: Added proxy sidecar logs (bearer token omitted)</h1>

<pre><code>2020-01-21T14:47:51.784617Z     info    curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: sidecar-injector/v0.0.0 (linux/amd64) kubernetes/$Format" -H "Authorization: Bearer ***" 'https://10.0.0.1:443/apis/admissionregistration.k8s.io/v1beta1/mutatingwebhookconfigurations?fieldSelector=metadata.name%3Distio-sidecar-injector&amp;resourceVersion=299045&amp;timeoutSeconds=449&amp;watch=true'
2020-01-21T14:47:51.787107Z     info    GET https://10.0.0.1:443/apis/admissionregistration.k8s.io/v1beta1/mutatingwebhookconfigurations?fieldSelector=metadata.name%3Distio-sidecar-injector&amp;resourceVersion=299045&amp;timeoutSeconds=449&amp;watch=true 200 OK in 2 milliseconds
2020-01-21T14:47:51.787126Z     info    Response Headers:
2020-01-21T14:47:51.787132Z     info        Audit-Id: 21862701-0185-4e25-b26e-ff78ddb6de1e
2020-01-21T14:47:51.787136Z     info        Content-Type: application/json
2020-01-21T14:47:51.787140Z     info        Date: Tue, 21 Jan 2020 14:47:51 GMT
</code></pre>

<h1>Edit 3: Payments cert structure and Cluster issuer</h1>

<pre><code>apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: payments-cert
  namespace: istio-system
spec:
  commonName: payments.mywebsite.com
  dnsNames:
  - payments.mywebsite.com
  issuerRef:
    kind: ClusterIssuer
    name: letsencrypt-prod
  secretName: payments-cert

---

apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
 labels:
   name: letsencrypt-prod
 name: letsencrypt-prod
 namespace: istio-system
spec:
 acme:
   email: mywebsite@gmail.com
   solvers:
    - http01:
        ingress:
          class:  istioIngress
   privateKeySecretRef:
     name: letsencrypt-staging
   server: https://acme-v02.api.letsencrypt.org/directory
</code></pre>

<h1>Edit 4: output of describe payments-cert (+ additional logs from istio sidecar injector)</h1>

<pre><code>Name:         payments-cert
Namespace:    istio-system
Labels:       &lt;none&gt;
Annotations:  cert-manager.io/certificate-name: payments-cert
              cert-manager.io/issuer-kind: ClusterIssuer
              cert-manager.io/issuer-name: letsencrypt-prod

Type:  kubernetes.io/tls

Data
====
tls.crt:  0 bytes
tls.key:  1679 bytes
ca.crt:   0 bytes
</code></pre>

<p>I also took a look at the emitted events from the istio sidecar injector on google cloud,  these logs are recorded:</p>

<pre><code>(Message): No matching pods found   
(Reason): NoPods    
(First Seen): Jan 21, 2020, 2:32:47 PM  
(Last Seen): Jan 21, 2020, 3:28:17 PM   
(counts): 112 
</code></pre>

<p>The reason of error in istio gateway log <code>failed load server cert/key pair from secret payments-cert: server cert or private key is empty</code> is that the <code>tls.crt</code> is empty in secret  <code>payments-cert</code>. There is some issue in generating the cert by cert manager. </p>

<p>One more thing is you are mixing letsencrypt-staging and letsencrypt-prod in the cert issuer.</p>

<p>Also here is a <a href="https://medium.com/@gregoire.waymel/istio-cert-manager-lets-encrypt-demystified-c1cbed011d67" rel="nofollow noreferrer">example</a> which you can follow. </p>
