<p>I am working with cert-manager in my kubernetes cluster, in order to get certificates signed by let'sencrypt CA to my service application inside my cluster. </p>

<p>I am performing the following steps in the order presented. I've wanted provide the most details as a possible of my process in order to understand the behavior presented. </p>

<h3>Install the CustomResourceDefinition resources separately</h3>

<pre><code>⟩ kubectl apply -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.7/deploy/manifests/00-crds.yaml
customresourcedefinition.apiextensions.k8s.io/certificates.certmanager.k8s.io created
customresourcedefinition.apiextensions.k8s.io/challenges.certmanager.k8s.io created
customresourcedefinition.apiextensions.k8s.io/clusterissuers.certmanager.k8s.io created
customresourcedefinition.apiextensions.k8s.io/issuers.certmanager.k8s.io created
customresourcedefinition.apiextensions.k8s.io/orders.certmanager.k8s.io created
[I] 
</code></pre>

<h3>Label the cert-manager namespace to disable resource validation</h3>

<pre><code>⟩ kubectl label namespace kube-system certmanager.k8s.io/disable-validation=true
namespace/kube-system labeled
[I] 
</code></pre>

<h3>Install the cert-manager Helm chart</h3>

<pre><code>⟩ helm install \
    --name cert-manager \
    --namespace kube-system \
    --version v0.7.0 \
    jetstack/cert-manager
</code></pre>

<ul>
<li>I've confirmed the steps <a href="https://docs.cert-manager.io/en/latest/getting-started/troubleshooting.html" rel="nofollow noreferrer">in this guide</a>, in order to avoid possible problems, and all steps are ok ...</li>
</ul>

<h2>Creating my ingress</h2>

<ul>
<li>I am using <a href="https://github.com/Kong/kubernetes-ingress-controller/blob/master/docs/deployment/aks.md" rel="nofollow noreferrer">kong-ingress-controller</a> to manage the ingress process.</li>
</ul>

<pre><code>⟩ kubectl get pod,svc,deploy,replicaset -n kong | grep kong-ingress-controller
pod/kong-ingress-controller-667b4748d4-ccj8z   2/2     Running   14         95m
service/kong-ingress-controller   NodePort       10.0.48.131   &lt;none&gt;          8001:32257/TCP               3d19h

deployment.extensions/kong-ingress-controller   1         1         1            1           3d19h

replicaset.extensions/kong-ingress-controller-667b4748d4   1         1         1       3d19h
</code></pre>

<p>This means that my external IP addres is given by kong-proxy and is <code>52.166.60.158</code></p>

<pre><code>⟩ kubectl get svc -n kong | grep kong-proxy
kong-proxy                LoadBalancer   10.0.153.8    52.166.60.158   80:31577/TCP,443:32323/TCP   3d21h
</code></pre>

<ul>
<li>I've created the ingress for first time of this way:</li>
</ul>

<pre><code>apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: kong-ingress-zcrm365
  # namespace: default
  annotations:
    # kubernetes.io/ingress.class: "nginx" # Don't include it in order to use kong-ingress-controller
    # add an annotation indicating the issuer to use.
spec:
  rules: 
  - host: test1kongletsencrypt.possibilit.nl
    http:
      paths:
        - path: "/"
          backend:
            serviceName: zcrm365dev
            servicePort: 80
      #- backend:
      #    serviceName: zcrm365dev
      #    servicePort: 80
      #  path: /

  tls: # &lt; placing a host in the TLS config will indicate a cert should be created
  - hosts: 
    - test1kongletsencrypt.possibilit.nl
    secretName:  cert-manager-webhook-webhook-tls 
    # for the moment I've included this secret which was created by cert-manager installation
</code></pre>

<ul>
<li>Apply it.</li>
</ul>

<pre><code>⟩ kubectl apply -f 03-zcrm365-ingress.yaml 
ingress.extensions/kong-ingress-zcrm365 created
[I] 
</code></pre>

<ul>
<li>And my ingress is taking the kong-ingress-controller</li>
</ul>

<pre><code>⟩ kubectl describe ingress  kong-ingress-zcrm365 
Name:             kong-ingress-zcrm365
Namespace:        default
Address:          52.166.60.158
Default backend:  default-http-backend:80 (&lt;none&gt;)
TLS:
  cert-manager-webhook-webhook-tls terminates test1kongletsencrypt.possibilit.nl
Rules:
  Host                                Path  Backends
  ----                                ----  --------
  test1kongletsencrypt.possibilit.nl  
                                      /   zcrm365dev:80 (&lt;none&gt;)
Annotations:
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{},"name":"kong-ingress-zcrm365","namespace":"default"},"spec":{"rules":[{"host":"test1kongletsencrypt.possibilit.nl","http":{"paths":[{"backend":{"serviceName":"zcrm365dev","servicePort":80},"path":"/"}]}}],"tls":[{"hosts":["test1kongletsencrypt.possibilit.nl"],"secretName":"cert-manager-webhook-webhook-tls"}]}}

Events:
  Type    Reason  Age    From                     Message
  ----    ------  ----   ----                     -------
  Normal  CREATE  3m30s  kong-ingress-controller  Ingress default/kong-ingress-zcrm365
  Normal  UPDATE  3m28s  kong-ingress-controller  Ingress default/kong-ingress-zcrm365
[I] 
</code></pre>

<h1>Creating a ClusterIssuer</h1>

<ul>
<li>I am going to create a <code>ClusterIssuer</code>, I'd can create a Issuer, but I've started with a ClusterIssuer. What is the best alternative? This depen of our deployment and requirements to future, mostly in the namespace situations</li>
</ul>

<pre><code>apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
 name: letsencrypt-staging
spec:
 acme:
   # The ACME server URL
   server: https://acme-staging-v02.api.letsencrypt.org/directory
   # Email address used for ACME registration
   email: b.garcia@possibilit.nl
   # Name of a secret used to store the ACME account private key
   privateKeySecretRef:
     name: letsencrypt-staging
   # Enable the HTTP-01 challenge provider
   http01: {}
</code></pre>

<ul>
<li>Apply it</li>
</ul>

<pre><code>⟩ kubectl apply -f 01-lets-encrypt-issuer-staging.yaml 
clusterissuer.certmanager.k8s.io/letsencrypt-staging created
[I] 
</code></pre>

<ul>
<li>This ClusterIssuer was registered on ACME letsencrypt server</li>
</ul>

<pre><code>⟩ kubectl describe clusterissuers letsencrypt-staging
Name:         letsencrypt-staging
Namespace:    
Labels:       &lt;none&gt;
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"certmanager.k8s.io/v1alpha1","kind":"ClusterIssuer","metadata":{"annotations":{},"name":"letsencrypt-staging"},"spec":{"acm...
API Version:  certmanager.k8s.io/v1alpha1
Kind:         ClusterIssuer
Metadata:
  Creation Timestamp:  2019-03-15T11:38:03Z
  Generation:          1
  Resource Version:    623999
  Self Link:           /apis/certmanager.k8s.io/v1alpha1/clusterissuers/letsencrypt-staging
  UID:                 cb48b391-4716-11e9-a113-e27267a7d354
Spec:
  Acme:
    Email:  b.garcia@possibilit.nl
    Http 01:
    Private Key Secret Ref:
      Name:  letsencrypt-staging
    Server:  https://acme-staging-v02.api.letsencrypt.org/directory
Status:
  Acme:
    Uri:  https://acme-staging-v02.api.letsencrypt.org/acme/acct/8579841
  Conditions:
    Last Transition Time:  2019-03-15T11:38:05Z
    Message:               The ACME account was registered with the ACME server
    Reason:                ACMEAccountRegistered
    Status:                True
    Type:                  Ready
Events:                    &lt;none&gt;
[I] 
</code></pre>

<h1>Modifying my ingress resource created previously</h1>

<ul>
<li>Now that I've created our Let's Encrypt staging ClusterIssuer, I am ready to modify the Ingress Resource we created above and enable TLS encryption for the test1kongletsencrypt.possibilit.nl paths adding the following</li>
</ul>

<p>I am going to add <code>certmanager.k8s.io/cluster-issuer: letsencrypt-staging</code> annotation and use the secret created with the <code>letsencrypt-staging</code> ClusterIssuer named <code>letsencrypt-staging</code></p>

<p>Our ingress has been stayed of this way:</p>

<pre><code>apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: kong-ingress-zcrm365
  #namespace: default
  annotations:
    # kubernetes.io/ingress.class: "nginx" #new
    # certmanager.k8s.io/acme-challenge-type: http01

    # add an annotation indicating the issuer to use.
    certmanager.k8s.io/cluster-issuer: letsencrypt-staging

spec:
  rules: 
  - host: test1kongletsencrypt.possibilit.nl
    http:
      paths:
        - path: "/"
          backend:
            serviceName: zcrm365dev
            servicePort: 80    
  tls: 
  - hosts: 
    - test1kongletsencrypt.possibilit.nl
    secretName:  letsencrypt-staging # I've added this secret of letsencrypt cluster issuer
</code></pre>

<ul>
<li>Apply it</li>
</ul>

<pre><code>⟩ kubectl apply -f 03-zcrm365-ingress.yaml 
ingress.extensions/kong-ingress-zcrm365 configured
[I]
</code></pre>

<p>This process update on the ingress, create one ingres named <code>cm-acme-http-solver-jr4fg</code></p>

<pre><code>⟩ kubectl get ingress 
NAME                        HOSTS                                ADDRESS         PORTS     AGE
cm-acme-http-solver-jr4fg   test1kongletsencrypt.possibilit.nl                   80        33s
kong-ingress-zcrm365        test1kongletsencrypt.possibilit.nl   52.166.60.158   80, 443   56m
[I] 
</code></pre>

<p>The detail of <code>cm-acme-http-solver-jr4fg</code> ingress is:</p>

<pre><code>⟩ kubectl get ingress cm-acme-http-solver-jr4fg -o yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/whitelist-source-range: 0.0.0.0/0
  creationTimestamp: "2019-03-15T12:10:57Z"
  generateName: cm-acme-http-solver-
  generation: 1
  labels:
    certmanager.k8s.io/acme-http-domain: "4095675862"
    certmanager.k8s.io/acme-http-token: "657526223"
  name: cm-acme-http-solver-jr4fg
  namespace: default
  ownerReferences:
  - apiVersion: certmanager.k8s.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: Challenge
    name: letsencrypt-staging-2613163196-0
    uid: 638f1701-471b-11e9-a113-e27267a7d354
  resourceVersion: "628284"
  selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/cm-acme-http-solver-jr4fg
  uid: 640ef483-471b-11e9-a113-e27267a7d354
spec:
  rules:
  - host: test1kongletsencrypt.possibilit.nl
    http:
      paths:
      - backend:
          serviceName: cm-acme-http-solver-svmvw
          servicePort: 8089
        path: /.well-known/acme-challenge/W7-9-KuPao_jg6EF5E2FXitFs8shOEsY5PlT9EEvNxE
status:
  loadBalancer:
    ingress:
    - ip: 52.166.60.158 
</code></pre>

<ul>
<li>And the detail of our <code>kong-ingress-zcrm365</code> resource ingress is:</li>
</ul>

<pre><code>⟩ kubectl describe  ingress kong-ingress-zcrm365
Name:             kong-ingress-zcrm365
Namespace:        default
Address:          52.166.60.158
Default backend:  default-http-backend:80 (&lt;none&gt;)
TLS:
  letsencrypt-staging terminates test1kongletsencrypt.possibilit.nl
Rules:
  Host                                Path  Backends
  ----                                ----  --------
  test1kongletsencrypt.possibilit.nl  
                                      /   zcrm365dev:80 (&lt;none&gt;)
Annotations:
  certmanager.k8s.io/cluster-issuer:                 letsencrypt-staging
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"certmanager.k8s.io/cluster-issuer":"letsencrypt-staging"},"name":"kong-ingress-zcrm365","namespace":"default"},"spec":{"rules":[{"host":"test1kongletsencrypt.possibilit.nl","http":{"paths":[{"backend":{"serviceName":"zcrm365dev","servicePort":80},"path":"/"}]}}],"tls":[{"hosts":["test1kongletsencrypt.possibilit.nl"],"secretName":"letsencrypt-staging"}]}}

Events:
  Type    Reason             Age                  From                     Message
  ----    ------             ----                 ----                     -------
  Normal  CREATE             60m                  kong-ingress-controller  Ingress default/kong-ingress-zcrm365
  Normal  UPDATE             4m25s (x2 over 60m)  kong-ingress-controller  Ingress default/kong-ingress-zcrm365
  Normal  CreateCertificate  4m25s                cert-manager             Successfully created Certificate "letsencrypt-staging"
[I] 
</code></pre>

<p>We can see that our ingress even using the <code>kong-ingress-controller</code> and the <code>letsencrypt-staging</code> certificate has been created in the default namespace: </p>

<pre><code>⟩ kubectl get certificates 
NAME
letsencrypt-staging
[I]
</code></pre>

<p>The letsencypt-staging certificate have the following detail</p>

<pre><code>⟩ kubectl describe certificate letsencrypt-staging
Name:         letsencrypt-staging
Namespace:    default
Labels:       &lt;none&gt;
Annotations:  &lt;none&gt;
API Version:  certmanager.k8s.io/v1alpha1
Kind:         Certificate
Metadata:
  Creation Timestamp:  2019-03-15T12:10:55Z
  Generation:          1
  Owner References:
    API Version:           extensions/v1beta1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Ingress
    Name:                  kong-ingress-zcrm365
    UID:                   8643558f-4713-11e9-a113-e27267a7d354
  Resource Version:        628164
  Self Link:               /apis/certmanager.k8s.io/v1alpha1/namespaces/default/certificates/letsencrypt-staging
  UID:                     62b3a31e-471b-11e9-a113-e27267a7d354
Spec:
  Acme:
    Config:
      Domains:
        test1kongletsencrypt.possibilit.nl
      Http 01:
  Dns Names:
    test1kongletsencrypt.possibilit.nl
  Issuer Ref:
    Kind:       ClusterIssuer
    Name:       letsencrypt-staging
  Secret Name:  letsencrypt-staging
Status:
  Conditions:
    Last Transition Time:  2019-03-15T12:10:55Z
    Message:               Certificate issuance in progress. Temporary certificate issued.
    Reason:                TemporaryCertificate
    Status:                False
    Type:                  Ready
Events:
  Type    Reason              Age    From          Message
  ----    ------              ----   ----          -------
  Normal  Generated           7m24s  cert-manager  Generated new private key
  Normal  GenerateSelfSigned  7m24s  cert-manager  Generated temporary self signed certificate
  Normal  OrderCreated        7m23s  cert-manager  Created Order resource "letsencrypt-staging-2613163196"
[I] 
~/workspace/ZCRM365/Deployments/Kubernetes/cert-manager · (Deployments±)
</code></pre>

<p>I can see that my order issue is not completed, only was created in the <code>OrderCreated</code> event, and this order already have 7 minutes since I've created this certificate and the order was not completed and by that reason the certificate is not issued successfully</p>

<p>Another thing that happens to me, is that the <code>letsencrypt-staging</code> secret created by the l<code>etsencrypt-staging</code> cluster Issuer and their respective certificate, only have the <code>tls.key</code>:</p>

<pre><code>⟩ kubectl describe secrets letsencrypt-staging -n kube-system
Name:         letsencrypt-staging
Namespace:    kube-system
Labels:       &lt;none&gt;
Annotations:  &lt;none&gt;

Type:  Opaque

Data
====
tls.key:  1675 bytes
[I] 
</code></pre>

<p>According to I understand, is that if the letsencrypt certificate complete the order and the certificate is issued, in the <code>letsencrypt-staging</code> secret I would have one tls.crt key and maybe my <code>letsencrypt-staging</code> will be of tls type and not Opaque?</p>

<ul>
<li>When I see the logs of my cert-manager pod I get the following output, I think that the http challenge is not executed:</li>
</ul>

<pre><code>I0315 12:10:57.833858       1 logger.go:103] Calling Discover
I0315 12:10:57.856136       1 pod.go:64] No existing HTTP01 challenge solver pod found for Certificate "default/letsencrypt-staging-2613163196-0". One will be created.
I0315 12:10:57.923080       1 service.go:51] No existing HTTP01 challenge solver service found for Certificate "default/letsencrypt-staging-2613163196-0". One will be created.
I0315 12:10:57.989596       1 ingress.go:49] Looking up Ingresses for selector certmanager.k8s.io/acme-http-domain=4095675862,certmanager.k8s.io/acme-http-token=657526223
I0315 12:10:57.989682       1 ingress.go:98] No existing HTTP01 challenge solver ingress found for Challenge "default/letsencrypt-staging-2613163196-0". One will be created.
I0315 12:10:58.014803       1 controller.go:178] ingress-shim controller: syncing item 'default/cm-acme-http-solver-jr4fg'
I0315 12:10:58.014842       1 sync.go:64] Not syncing ingress default/cm-acme-http-solver-jr4fg as it does not contain necessary annotations
I0315 12:10:58.014846       1 controller.go:184] ingress-shim controller: Finished processing work item "default/cm-acme-http-solver-jr4fg"
I0315 12:10:58.015447       1 ingress.go:49] Looking up Ingresses for selector certmanager.k8s.io/acme-http-domain=4095675862,certmanager.k8s.io/acme-http-token=657526223
I0315 12:10:58.033431       1 sync.go:173] propagation check failed: wrong status code '404', expected '200'
I0315 12:10:58.079504       1 controller.go:212] challenges controller: Finished processing work item "default/letsencrypt-staging-2613163196-0"
I0315 12:10:58.079616       1 controller.go:206] challenges controller: syncing item 'default/letsencrypt-staging-2613163196-0'
I0315 12:10:58.079569       1 controller.go:184] orders controller: syncing item 'default/letsencrypt-staging-2613163196'
</code></pre>

<p>I get this message <code>No existing HTTP01 challenge solver pod found for Certificate "default/letsencrypt-staging-2613163196-0"</code>
According to this, I decide add the   <code>certmanager.k8s.io/acme-challenge-type: http01</code> annotation to my <code>kong-ingress-zcrm365</code> ingress but nothing happened  ... my ingress is updated, but nothing more.</p>

<p>All this process confirms that the TLS certificate <strong>was not successfully</strong> issued and HTTPS encryption is not active for my domains test1kongletsencrypt.possibilit.nl configured.</p>

<p>This make that my letsencrypt-staging certificate have a <code>Status:False</code>, and the order created event does not advance to completed to be issued.</p>

<pre><code>

  Conditions:
    Last Transition Time:  2019-03-15T12:10:55Z
    Message:               Certificate issuance in progress. Temporary certificate issued.
    Reason:                TemporaryCertificate
    Status:                False
    Type:                  Ready
Events:
  Type    Reason              Age                From          Message
  ----    ------              ----               ----          -------
  Normal  Generated           51m                cert-manager  Generated new private key
  Normal  GenerateSelfSigned  51m                cert-manager  Generated temporary self signed certificate
  Normal  Cleanup             5m42s              cert-manager  Deleting old Order resource "letsencrypt-staging-2613163196"
  Normal  OrderCreated        5m42s              cert-manager  Created Order resource "letsencrypt-staging-2965106631"
  Normal  OrderCreated        39s (x2 over 51m)  cert-manager  Created Order resource "letsencrypt-staging-2613163196"
  Normal  Cleanup             39s                cert-manager  Deleting old Order resource "letsencrypt-staging-2965106631"
[I] 
~/workspace/ZCRM365/Deployments/Kubernetes/cert-manager · (Deployments±)
</code></pre>

<p>How to can I to my certificate to be signed and succesfully issued by letsencrypt CA and active the https encryption active?
What is happening with these logs messages?</p>

<pre><code>kubectl logs -n kube-system cert-manager-6f68b58796-q7txg

0315 13:06:11.027204       1 logger.go:103] Calling Discover
I0315 13:06:11.032299       1 ingress.go:49] Looking up Ingresses for selector certmanager.k8s.io/acme-http-domain=4095675862,certmanager.k8s.io/acme-http-token=657526223
I0315 13:06:11.046081       1 sync.go:173] propagation check failed: wrong status code '404', expected '200'
I0315 13:06:11.046109       1 controller.go:212] challenges controller: Finished processing work item "default/letsencrypt-staging-2613163196-0"
I0315 13:06:21.046242       1 controller.go:206] challenges controller: syncing item 'default/letsencrypt-staging-2613163196-0'
</code></pre>

<p>I've heared that letsencrypt-staging environment only have test certificates and these are a kind of 'fake certificates' and maybe some clients like my chrome/firefox browser doesn’t trust certificate issuer ...  </p>

<p>Is this a reason to I cannot enable https encryption on my domain?
In affirmative case, should I change from staging environment to production environment?</p>

<p>In <a href="https://community.letsencrypt.org/t/using-lets-encrypt-staging-certificates-on-internal-testing-staging-development-environments/7981/6" rel="nofollow noreferrer">this question</a> some people talk about that but they emphasize: </p>

<blockquote>
  <p>that the staging environment should be used just to test that your client is working fine and can generate the challenges, certificates</p>
</blockquote>

<p>In my case the http challenge is not generated still in staging environment. :(</p>

bgarcial
<p>here are the annotation I'm usually using for this:</p>

<pre><code>"ingress.kubernetes.io/ssl-redirect": "true",
"certmanager.k8s.io/cluster-issuer": "letsencrypt-production",
# I'd suggest adding these 2 below
"kubernetes.io/tls-acme": "true",
"kubernetes.io/ingress.class": "nginx"
</code></pre>

<p>also, you didnt spot this error:</p>

<pre><code>I0315 12:10:58.033431       1 sync.go:173] propagation check failed: wrong status code '404', expected '200'
</code></pre>

<p>I'm not sure what is wrong here exactly, your domain name should resolve to your ingress, you should be able to access <code>yourdomain.name/.well-known/acme-challenge/W7-9-KuPao_jg6EF5E2FXitFs8shOEsY5PlT9EEvNxE</code> (this is lets encrypt validation response url, according to your logs)</p>
