title: CertManager Letsencrypt CertificateRequest "failed to perform self check GET request"
body: <p><em><code>Waiting for http-01 challenge propagation: failed to perform self check GET request</code>, it's similar to this bug <a href="https://github.com/jetstack/cert-manager/issues/656" rel="noreferrer">https://github.com/jetstack/cert-manager/issues/656</a>
  but all solutions from GitHub ticket comments didn't help.</em></p>
  <p>I'm trying to set up <code>CertManager</code> on DigitalOcean as described in this tutorial: <a href="https://www.digitalocean.com/community/tutorials/how-to-set-up-an-nginx-ingress-with-cert-manager-on-digitalocean-kubernetes" rel="noreferrer">https://www.digitalocean.com/community/tutorials/how-to-set-up-an-nginx-ingress-with-cert-manager-on-digitalocean-kubernetes</a>
  I didn't get any error but the request from <code>CertManager</code> is waiting in pending state more than 40 hours.</p>
  <p>I have successfully configured Ingress with Nginx, then I created a namespace and created <code>CertManager</code> CRDs:</p>
  <pre class="lang-sh prettyprint-override"><code>$ kubectl create namespace cert-manager
  $ kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.12.0/cert-manager.yaml
  </code></pre>
  <p>I can see all <code>CertManager</code> pods as expected:</p>
  <pre><code>$ kubectl get pods --namespace cert-manager
  NAME                                       READY   STATUS    RESTARTS   AGE
  cert-manager-5c47f46f57-gxhwv              1/1     Running   0          42h
  cert-manager-cainjector-6659d6844d-xp75s   1/1     Running   0          42h
  cert-manager-webhook-547567b88f-k4dv2      1/1     Running   0          42h
  </code></pre>
  <p>Then I created staging issuer:</p>
  <pre class="lang-yaml prettyprint-override"><code>---
  apiVersion: cert-manager.io/v1alpha2
  kind: ClusterIssuer
  metadata:
    name: letsencrypt-staging
    namespace: cert-manager
  spec:
    acme:
      server: https://acme-staging-v02.api.letsencrypt.org/directory
      email: some@email.here
      privateKeySecretRef:
        name: letsencrypt-staging
      solvers:
        - http01:
            ingress:
              class: nginx
  </code></pre>
  <p>And updated Ingress configuration:</p>
  <pre class="lang-yaml prettyprint-override"><code>---
  apiVersion: networking.k8s.io/v1beta1
  kind: Ingress
  metadata:
    name: echo-ingress
    annotations:
      kubernetes.io/ingress.class: &quot;nginx&quot;
      # cert-manager.io/cluster-issuer: &quot;letsencrypt-prod&quot;
      cert-manager.io/cluster-issuer: &quot;letsencrypt-staging&quot;
  spec:
    tls:
      - hosts:
          - echo.some.domain
        secretName: ingress-tls
    rules:
      - host: echo.some.domain
        http:
          paths:
            - backend:
                serviceName: echo1
                servicePort: 80
  </code></pre>
  <p>But after that, <code>CertManager</code> didn't update certificate and waiting in <code>InProgress</code> state:</p>
  <pre class="lang-sh prettyprint-override"><code>$ date
  Wed 18 Dec 2019 01:58:08 PM MSK
  
  $ kubectl describe cert
  
  ...
  Status:
    Conditions:
      Last Transition Time:  2019-12-16T17:23:56Z
      Message:               Waiting for CertificateRequest &quot;ingress-tls-1089568541&quot; to complete
      Reason:                InProgress
      Status:                False
      Type:                  Ready
  Events:                    &lt;none&gt;
  </code></pre>
  <p>Instead of using <code>Fake LE Intermediate X1</code> as a <code>CN</code> it returns <code>CN=Kubernetes Ingress Controller Fake Certificate,O=Acme Co</code></p>
  <pre class="lang-sh prettyprint-override"><code>$ kubectl describe CertificateRequest 
  Status:
    Conditions:
      Last Transition Time:  2019-12-16T17:50:05Z
      Message:               Waiting on certificate issuance from order default/ingress-tls-1089568541-1576201144: &quot;pending&quot;
      Reason:                Pending
      Status:                False
      Type:                  Ready
  Events:                    &lt;none&gt;
  </code></pre>
  <p>What can be the problem with <code>CertManager</code> and how to fix it?</p>
  <hr />
  <p>Update:</p>
  <p>Ingress logs contains these errors:</p>
  <pre><code>$ kubectl -n ingress-nginx logs  nginx-ingress-controller-7754db565c-g557h 
  
  I1218 17:24:30.331127       6 status.go:295] updating Ingress default/cm-acme-http-solver-4dkdn status from [] to [{xxx.xxx.xxx.xxx }]
  I1218 17:24:30.333250       6 status.go:295] updating Ingress default/cm-acme-http-solver-9dpqc status from [] to [{xxx.xxx.xxx.xxx }]
  I1218 17:24:30.341292       6 event.go:209] Event(v1.ObjectReference{Kind:&quot;Ingress&quot;, Namespace:&quot;default&quot;, Name:&quot;cm-acme-http-solver-4dkdn&quot;, UID:&quot;2e523b74-8bbb-41c7-be8a-44d8db8abd6e&quot;, APIVersion:&quot;extensions/v1beta1&quot;, ResourceVersion:&quot;722472&quot;, FieldPath:&quot;&quot;}): type: 'Normal' reason: 'UPDATE' Ingress default/cm-acme-http-solver-4dkdn
  I1218 17:24:30.344340       6 event.go:209] Event(v1.ObjectReference{Kind:&quot;Ingress&quot;, Namespace:&quot;default&quot;, Name:&quot;cm-acme-http-solver-9dpqc&quot;, UID:&quot;b574a3b6-6c5b-4266-a4e2-6ff2de2d78e0&quot;, APIVersion:&quot;extensions/v1beta1&quot;, ResourceVersion:&quot;722473&quot;, FieldPath:&quot;&quot;}): type: 'Normal' reason: 'UPDATE' Ingress default/cm-acme-http-solver-9dpqc
  W1218 17:24:30.442276       6 controller.go:1042] Error getting SSL certificate &quot;default/ingress-tls&quot;: local SSL certificate default/ingress-tls was not found. Using default certificate
  W1218 17:24:30.442950       6 controller.go:1042] Error getting SSL certificate &quot;default/ingress-tls&quot;: local SSL certificate default/ingress-tls was not found. Using default certificate
  W1218 17:24:33.775476       6 controller.go:1042] Error getting SSL certificate &quot;default/ingress-tls&quot;: local SSL certificate default/ingress-tls was not found. Using default certificate
  W1218 17:24:33.775956       6 controller.go:1042] Error getting SSL certificate &quot;default/ingress-tls&quot;: local SSL certificate default/ingress-tls was not found. Using default certificate
  </code></pre>
  <hr />
  <p>Update2:</p>
  <p>Secret for <code>ingress-tls</code> is available as expected:</p>
  <pre class="lang-yaml prettyprint-override"><code>$ kubectl get secret ingress-tls -o yaml
  
  apiVersion: v1
  data:
    ca.crt: &quot;&quot;
    tls.crt: &quot;&quot;
    tls.key: &lt;secret-key-data-base64-encoded&gt;
  kind: Secret
  metadata:
    annotations:
      cert-manager.io/certificate-name: ingress-tls
      cert-manager.io/issuer-kind: ClusterIssuer
      cert-manager.io/issuer-name: letsencrypt-staging
    creationTimestamp: &quot;2019-12-16T17:23:56Z&quot;
    name: ingress-tls
    namespace: default
    resourceVersion: &quot;328801&quot;
    selfLink: /api/v1/namespaces/default/secrets/ingress-tls
    uid: 5d640b66-1572-44a1-94e4-6d85a73bf21c
  type: kubernetes.io/tls
  </code></pre>
  <hr />
  <p>Update3:</p>
  <p>I've found that <code>cert-manager</code> pod is failing with log:</p>
  <pre><code>E1219 11:06:08.294011       1 sync.go:184] cert-manager/controller/challenges &quot;msg&quot;=&quot;propagation check failed&quot; &quot;error&quot;=&quot;failed to perform self check GET request 'http://&lt;some.domain&gt;/.well-known/acme-challenge/&lt;some-path&gt;': Get http://&lt;some.domain&gt;/.well-known/acme-challenge/&lt;some-path&gt;: dial tcp xxx.xxx.xxx.xxx:80: connect: connection timed out&quot; &quot;dnsName&quot;=&quot;&lt;some.domain&gt;&quot; &quot;resource_kind&quot;=&quot;Challenge&quot; &quot;resource_name&quot;=&quot;ingress-tls-1089568541-1576201144-1086699008&quot; &quot;resource_namespace&quot;=&quot;default&quot; &quot;type&quot;=&quot;http-01&quot; 
  </code></pre>
  <p>Challenge status:</p>
  <pre><code>$ kubectl describe challenge ingress-tls-1089568541-1576201144-471532423
  
  Name:         ingress-tls-1089568541-1576201144-471532423
  Namespace:    default
  Labels:       &lt;none&gt;
  Annotations:  &lt;none&gt;
  API Version:  acme.cert-manager.io/v1alpha2
  Kind:         Challenge
  Metadata:
    Creation Timestamp:  2019-12-19T11:32:19Z
    Finalizers:
      finalizer.acme.cert-manager.io
    Generation:  1
    Owner References:
      API Version:           acme.cert-manager.io/v1alpha2
      Block Owner Deletion:  true
      Controller:            true
      Kind:                  Order
      Name:                  ingress-tls-1089568541-1576201144
      UID:                   7d19d86f-0b56-4756-aa20-bb85caf80b9e
    Resource Version:        872062
    Self Link:               /apis/acme.cert-manager.io/v1alpha2/namespaces/default/challenges/ingress-tls-1089568541-1576201144-471532423
    UID:                     503a8b4e-dc60-4080-91d9-2847815af1cc
  Spec:
    Authz URL:  https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/123456
    Dns Name:   &lt;domain&gt;
    Issuer Ref:
      Group:  cert-manager.io
      Kind:   ClusterIssuer
      Name:   letsencrypt-staging
    Key:      &lt;key&gt;
    Solver:
      http01:
        Ingress:
          Class:  nginx
    Token:        &lt;token&gt;
    Type:         http-01
    URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12345/abc
    Wildcard:     false
  Status:
    Presented:   true
    Processing:  true
    Reason:      Waiting for http-01 challenge propagation: failed to perform self check GET request 'http://&lt;domain&gt;/.well-known/acme-challenge/&lt;token&gt;': Get http://&lt;domain&gt;/.well-known/acme-challenge/&lt;token&gt;: dial tcp xxx.xxx.xxx.xxx:80: connect: connection timed out
    State:       pending
  Events:
    Type    Reason     Age    From          Message
    ----    ------     ----   ----          -------
    Normal  Started    4m28s  cert-manager  Challenge scheduled for processing
    Normal  Presented  4m28s  cert-manager  Presented challenge using http-01 challenge mechanism
  </code></pre>
  <p>I tried to delete the challenge to re-trigger it, but it failed with same error after one-two minutes. I checked that I can access challenge URL from cluster nodes (using <code>kubectl run -it ...</code> and <code>wget http://&lt;domain&gt;/.well-known/acme-challenge/&lt;token&gt;</code> from inside new pod.</p>