<p><em><code>Waiting for http-01 challenge propagation: failed to perform self check GET request</code>, it's similar to this bug <a href="https://github.com/jetstack/cert-manager/issues/656" rel="noreferrer">https://github.com/jetstack/cert-manager/issues/656</a>
but all solutions from GitHub ticket comments didn't help.</em></p>
<p>I'm trying to set up <code>CertManager</code> on DigitalOcean as described in this tutorial: <a href="https://www.digitalocean.com/community/tutorials/how-to-set-up-an-nginx-ingress-with-cert-manager-on-digitalocean-kubernetes" rel="noreferrer">https://www.digitalocean.com/community/tutorials/how-to-set-up-an-nginx-ingress-with-cert-manager-on-digitalocean-kubernetes</a>
I didn't get any error but the request from <code>CertManager</code> is waiting in pending state more than 40 hours.</p>
<p>I have successfully configured Ingress with Nginx, then I created a namespace and created <code>CertManager</code> CRDs:</p>
<pre class="lang-sh prettyprint-override"><code>$ kubectl create namespace cert-manager
$ kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.12.0/cert-manager.yaml
</code></pre>
<p>I can see all <code>CertManager</code> pods as expected:</p>
<pre><code>$ kubectl get pods --namespace cert-manager
NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-5c47f46f57-gxhwv              1/1     Running   0          42h
cert-manager-cainjector-6659d6844d-xp75s   1/1     Running   0          42h
cert-manager-webhook-547567b88f-k4dv2      1/1     Running   0          42h
</code></pre>
<p>Then I created staging issuer:</p>
<pre class="lang-yaml prettyprint-override"><code>---
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
  namespace: cert-manager
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: some@email.here
    privateKeySecretRef:
      name: letsencrypt-staging
    solvers:
      - http01:
          ingress:
            class: nginx
</code></pre>
<p>And updated Ingress configuration:</p>
<pre class="lang-yaml prettyprint-override"><code>---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: echo-ingress
  annotations:
    kubernetes.io/ingress.class: &quot;nginx&quot;
    # cert-manager.io/cluster-issuer: &quot;letsencrypt-prod&quot;
    cert-manager.io/cluster-issuer: &quot;letsencrypt-staging&quot;
spec:
  tls:
    - hosts:
        - echo.some.domain
      secretName: ingress-tls
  rules:
    - host: echo.some.domain
      http:
        paths:
          - backend:
              serviceName: echo1
              servicePort: 80
</code></pre>
<p>But after that, <code>CertManager</code> didn't update certificate and waiting in <code>InProgress</code> state:</p>
<pre class="lang-sh prettyprint-override"><code>$ date
Wed 18 Dec 2019 01:58:08 PM MSK

$ kubectl describe cert

...
Status:
  Conditions:
    Last Transition Time:  2019-12-16T17:23:56Z
    Message:               Waiting for CertificateRequest &quot;ingress-tls-1089568541&quot; to complete
    Reason:                InProgress
    Status:                False
    Type:                  Ready
Events:                    &lt;none&gt;
</code></pre>
<p>Instead of using <code>Fake LE Intermediate X1</code> as a <code>CN</code> it returns <code>CN=Kubernetes Ingress Controller Fake Certificate,O=Acme Co</code></p>
<pre class="lang-sh prettyprint-override"><code>$ kubectl describe CertificateRequest 
Status:
  Conditions:
    Last Transition Time:  2019-12-16T17:50:05Z
    Message:               Waiting on certificate issuance from order default/ingress-tls-1089568541-1576201144: &quot;pending&quot;
    Reason:                Pending
    Status:                False
    Type:                  Ready
Events:                    &lt;none&gt;
</code></pre>
<p>What can be the problem with <code>CertManager</code> and how to fix it?</p>
<hr />
<p>Update:</p>
<p>Ingress logs contains these errors:</p>
<pre><code>$ kubectl -n ingress-nginx logs  nginx-ingress-controller-7754db565c-g557h 

I1218 17:24:30.331127       6 status.go:295] updating Ingress default/cm-acme-http-solver-4dkdn status from [] to [{xxx.xxx.xxx.xxx }]
I1218 17:24:30.333250       6 status.go:295] updating Ingress default/cm-acme-http-solver-9dpqc status from [] to [{xxx.xxx.xxx.xxx }]
I1218 17:24:30.341292       6 event.go:209] Event(v1.ObjectReference{Kind:&quot;Ingress&quot;, Namespace:&quot;default&quot;, Name:&quot;cm-acme-http-solver-4dkdn&quot;, UID:&quot;2e523b74-8bbb-41c7-be8a-44d8db8abd6e&quot;, APIVersion:&quot;extensions/v1beta1&quot;, ResourceVersion:&quot;722472&quot;, FieldPath:&quot;&quot;}): type: 'Normal' reason: 'UPDATE' Ingress default/cm-acme-http-solver-4dkdn
I1218 17:24:30.344340       6 event.go:209] Event(v1.ObjectReference{Kind:&quot;Ingress&quot;, Namespace:&quot;default&quot;, Name:&quot;cm-acme-http-solver-9dpqc&quot;, UID:&quot;b574a3b6-6c5b-4266-a4e2-6ff2de2d78e0&quot;, APIVersion:&quot;extensions/v1beta1&quot;, ResourceVersion:&quot;722473&quot;, FieldPath:&quot;&quot;}): type: 'Normal' reason: 'UPDATE' Ingress default/cm-acme-http-solver-9dpqc
W1218 17:24:30.442276       6 controller.go:1042] Error getting SSL certificate &quot;default/ingress-tls&quot;: local SSL certificate default/ingress-tls was not found. Using default certificate
W1218 17:24:30.442950       6 controller.go:1042] Error getting SSL certificate &quot;default/ingress-tls&quot;: local SSL certificate default/ingress-tls was not found. Using default certificate
W1218 17:24:33.775476       6 controller.go:1042] Error getting SSL certificate &quot;default/ingress-tls&quot;: local SSL certificate default/ingress-tls was not found. Using default certificate
W1218 17:24:33.775956       6 controller.go:1042] Error getting SSL certificate &quot;default/ingress-tls&quot;: local SSL certificate default/ingress-tls was not found. Using default certificate
</code></pre>
<hr />
<p>Update2:</p>
<p>Secret for <code>ingress-tls</code> is available as expected:</p>
<pre class="lang-yaml prettyprint-override"><code>$ kubectl get secret ingress-tls -o yaml

apiVersion: v1
data:
  ca.crt: &quot;&quot;
  tls.crt: &quot;&quot;
  tls.key: &lt;secret-key-data-base64-encoded&gt;
kind: Secret
metadata:
  annotations:
    cert-manager.io/certificate-name: ingress-tls
    cert-manager.io/issuer-kind: ClusterIssuer
    cert-manager.io/issuer-name: letsencrypt-staging
  creationTimestamp: &quot;2019-12-16T17:23:56Z&quot;
  name: ingress-tls
  namespace: default
  resourceVersion: &quot;328801&quot;
  selfLink: /api/v1/namespaces/default/secrets/ingress-tls
  uid: 5d640b66-1572-44a1-94e4-6d85a73bf21c
type: kubernetes.io/tls
</code></pre>
<hr />
<p>Update3:</p>
<p>I've found that <code>cert-manager</code> pod is failing with log:</p>
<pre><code>E1219 11:06:08.294011       1 sync.go:184] cert-manager/controller/challenges &quot;msg&quot;=&quot;propagation check failed&quot; &quot;error&quot;=&quot;failed to perform self check GET request 'http://&lt;some.domain&gt;/.well-known/acme-challenge/&lt;some-path&gt;': Get http://&lt;some.domain&gt;/.well-known/acme-challenge/&lt;some-path&gt;: dial tcp xxx.xxx.xxx.xxx:80: connect: connection timed out&quot; &quot;dnsName&quot;=&quot;&lt;some.domain&gt;&quot; &quot;resource_kind&quot;=&quot;Challenge&quot; &quot;resource_name&quot;=&quot;ingress-tls-1089568541-1576201144-1086699008&quot; &quot;resource_namespace&quot;=&quot;default&quot; &quot;type&quot;=&quot;http-01&quot; 
</code></pre>
<p>Challenge status:</p>
<pre><code>$ kubectl describe challenge ingress-tls-1089568541-1576201144-471532423

Name:         ingress-tls-1089568541-1576201144-471532423
Namespace:    default
Labels:       &lt;none&gt;
Annotations:  &lt;none&gt;
API Version:  acme.cert-manager.io/v1alpha2
Kind:         Challenge
Metadata:
  Creation Timestamp:  2019-12-19T11:32:19Z
  Finalizers:
    finalizer.acme.cert-manager.io
  Generation:  1
  Owner References:
    API Version:           acme.cert-manager.io/v1alpha2
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Order
    Name:                  ingress-tls-1089568541-1576201144
    UID:                   7d19d86f-0b56-4756-aa20-bb85caf80b9e
  Resource Version:        872062
  Self Link:               /apis/acme.cert-manager.io/v1alpha2/namespaces/default/challenges/ingress-tls-1089568541-1576201144-471532423
  UID:                     503a8b4e-dc60-4080-91d9-2847815af1cc
Spec:
  Authz URL:  https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/123456
  Dns Name:   &lt;domain&gt;
  Issuer Ref:
    Group:  cert-manager.io
    Kind:   ClusterIssuer
    Name:   letsencrypt-staging
  Key:      &lt;key&gt;
  Solver:
    http01:
      Ingress:
        Class:  nginx
  Token:        &lt;token&gt;
  Type:         http-01
  URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12345/abc
  Wildcard:     false
Status:
  Presented:   true
  Processing:  true
  Reason:      Waiting for http-01 challenge propagation: failed to perform self check GET request 'http://&lt;domain&gt;/.well-known/acme-challenge/&lt;token&gt;': Get http://&lt;domain&gt;/.well-known/acme-challenge/&lt;token&gt;: dial tcp xxx.xxx.xxx.xxx:80: connect: connection timed out
  State:       pending
Events:
  Type    Reason     Age    From          Message
  ----    ------     ----   ----          -------
  Normal  Started    4m28s  cert-manager  Challenge scheduled for processing
  Normal  Presented  4m28s  cert-manager  Presented challenge using http-01 challenge mechanism
</code></pre>
<p>I tried to delete the challenge to re-trigger it, but it failed with same error after one-two minutes. I checked that I can access challenge URL from cluster nodes (using <code>kubectl run -it ...</code> and <code>wget http://&lt;domain&gt;/.well-known/acme-challenge/&lt;token&gt;</code> from inside new pod.</p>

<p>This might be worthwhile to look at. I was facing similar issue with <code>Connection Timeout</code></p>

<p>Change <code>LoadBalancer</code> in <code>ingress-nginx</code> service.</p>

<p>Add/Change <code>externalTrafficPolicy: Cluster</code>.</p>

<p>Reason being, pod with the certificate-issuer wound up on a different node than the load balancer did, so it couldn’t talk to itself through the ingress.</p>

<p>Below is complete block taken from <a href="https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.26.1/deploy/static/provider/cloud-generic.yaml" rel="noreferrer">https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.26.1/deploy/static/provider/cloud-generic.yaml</a></p>

<pre><code>kind: Service
apiVersion: v1
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  #CHANGE/ADD THIS
  externalTrafficPolicy: Cluster
  type: LoadBalancer
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  ports:
    - name: http
      port: 80
      targetPort: http
    - name: https
      port: 443
      targetPort: https

---
</code></pre>
