<p>I've been trying to get the http-01 challenge method working with traefik v2 and cert-manager, both installed through their current helm charts. The LB endpoint can be requested through the ip and hostname, and I've tested that the http host passes on letsdebug (<code>No issues were found with &lt;domain&gt;</code>).</p>
<p>Traefik lives in the <code>traefik</code> namespace, while cert-manager lives in its own <code>cert-manager</code> namespace. I've created a <code>ClusterIssuer</code> inside the <code>cert-manager</code> namespace:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: removed@example.com
    privateKeySecretRef:
      name: letsencrypt-staging
    solvers:
      - http01:
          ingress:
            class: traefik
            ingressTemplate:
              metadata:
                namespace: cert-manager
                annotations:
                  traefik.ingress.kubernetes.io/router.entrypoints: web
</code></pre>
<p>The <code>ingressTemplate</code> part is my attempt at making the randomly created ingress from cert-manager map to the correct traefik endpoint - this hasn't changed anything, but I leave it in in case I've fubared anything here.</p>
<p>I've then created a <code>Certificate</code> and applied it - I've tried using both the cert-manager, traefik and default namespace for this, without any differing luck (the actual domain name has been replaced with domain.example.com):</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: domain.example.com
spec:
  secretName: domain-example-com-tls
  issuerRef:
    kind: ClusterIssuer
    name: letsencrypt-staging
  commonName: domain.example.com
  dnsNames:
    - domain.example.com
</code></pre>
<p>Looking at the logs for the cert-manager pod, I can see both a 404 error and then a &quot;DNS A record error&quot; - the DNS record error seems spurious as it can be resolved with other services and has been present for &gt; 24hrs.</p>
<pre><code>I0413 12:37:51.478359       1 conditions.go:201] Setting lastTransitionTime for Certificate &quot;domain.example.com&quot; condition &quot;Issuing&quot; to 2022-04-13 12:37:51.478353098 +0000 UTC m=+6998.327004050
I0413 12:37:51.760018       1 controller.go:161] cert-manager/certificates-key-manager &quot;msg&quot;=&quot;re-queuing item due to optimistic locking on resource&quot; &quot;key&quot;=&quot;default/domain.example.com&quot; &quot;error&quot;=&quot;Operation cannot be fulfilled on certificates.cert-manager.io \&quot;domain.example.com\&quot;: the object has been modified; please apply your changes to the latest version and try again&quot;
I0413 12:37:51.769026       1 conditions.go:261] Setting lastTransitionTime for CertificateRequest &quot;domain.example.com-r98k2&quot; condition &quot;Approved&quot; to 2022-04-13 12:37:51.769016958 +0000 UTC m=+6998.617667914
I0413 12:37:51.836517       1 conditions.go:261] Setting lastTransitionTime for CertificateRequest &quot;domain.example.com-r98k2&quot; condition &quot;Ready&quot; to 2022-04-13 12:37:51.836496254 +0000 UTC m=+6998.685147170
I0413 12:37:51.868932       1 conditions.go:261] Setting lastTransitionTime for CertificateRequest &quot;domain.example.com-r98k2&quot; condition &quot;Ready&quot; to 2022-04-13 12:37:51.868921204 +0000 UTC m=+6998.717572135
I0413 12:37:51.888553       1 controller.go:161] cert-manager/certificaterequests-issuer-acme &quot;msg&quot;=&quot;re-queuing item due to optimistic locking on resource&quot; &quot;key&quot;=&quot;default/domain.example.com-r98k2&quot; &quot;error&quot;=&quot;Operation cannot be fulfilled on certificaterequests.cert-manager.io \&quot;domain.example.com-r98k2\&quot;: the object has been modified; please apply your changes to the latest version and try again&quot;
E0413 12:37:53.529269       1 controller.go:210] cert-manager/challenges/scheduler &quot;msg&quot;=&quot;error scheduling challenge for processing&quot; &quot;error&quot;=&quot;Operation cannot be fulfilled on challenges.acme.cert-manager.io \&quot;domain.example.com-r98k2-2809069211-587139531\&quot;: the object has been modified; please apply your changes to the latest version and try again&quot; &quot;resource_kind&quot;=&quot;Challenge&quot; &quot;resource_name&quot;=&quot;domain.example.com-r98k2-2809069211-587139531&quot; &quot;resource_namespace&quot;=&quot;default&quot; &quot;resource_version&quot;=&quot;v1&quot;
I0413 12:37:55.028477       1 pod.go:71] cert-manager/challenges/http01/ensurePod &quot;msg&quot;=&quot;creating HTTP01 challenge solver pod&quot; &quot;dnsName&quot;=&quot;domain.example.com&quot; &quot;resource_kind&quot;=&quot;Challenge&quot; &quot;resource_name&quot;=&quot;domain.example.com-r98k2-2809069211-587139531&quot; &quot;resource_namespace&quot;=&quot;default&quot; &quot;resource_version&quot;=&quot;v1&quot; &quot;type&quot;=&quot;HTTP-01&quot;
I0413 12:37:55.237109       1 pod.go:59] cert-manager/challenges/http01/selfCheck/http01/ensurePod &quot;msg&quot;=&quot;found one existing HTTP01 solver pod&quot; &quot;dnsName&quot;=&quot;domain.example.com&quot; &quot;related_resource_kind&quot;=&quot;Pod&quot; &quot;related_resource_name&quot;=&quot;cm-acme-http-solver-k8wl8&quot; &quot;related_resource_namespace&quot;=&quot;default&quot; &quot;related_resource_version&quot;=&quot;v1&quot; &quot;resource_kind&quot;=&quot;Challenge&quot; &quot;resource_name&quot;=&quot;domain.example.com-r98k2-2809069211-587139531&quot; &quot;resource_namespace&quot;=&quot;default&quot; &quot;resource_version&quot;=&quot;v1&quot; &quot;type&quot;=&quot;HTTP-01&quot;
I0413 12:37:55.237350       1 service.go:43] cert-manager/challenges/http01/selfCheck/http01/ensureService &quot;msg&quot;=&quot;found one existing HTTP01 solver Service for challenge resource&quot; &quot;dnsName&quot;=&quot;domain.example.com&quot; &quot;related_resource_kind&quot;=&quot;Service&quot; &quot;related_resource_name&quot;=&quot;cm-acme-http-solver-gvvkt&quot; &quot;related_resource_namespace&quot;=&quot;default&quot; &quot;related_resource_version&quot;=&quot;v1&quot; &quot;resource_kind&quot;=&quot;Challenge&quot; &quot;resource_name&quot;=&quot;domain.example.com-r98k2-2809069211-587139531&quot; &quot;resource_namespace&quot;=&quot;default&quot; &quot;resource_version&quot;=&quot;v1&quot; &quot;type&quot;=&quot;HTTP-01&quot;
I0413 12:37:55.237539       1 ingress.go:99] cert-manager/challenges/http01/selfCheck/http01/ensureIngress &quot;msg&quot;=&quot;found one existing HTTP01 solver ingress&quot; &quot;dnsName&quot;=&quot;domain.example.com&quot; &quot;related_resource_kind&quot;=&quot;Ingress&quot; &quot;related_resource_name&quot;=&quot;cm-acme-http-solver-pbs7c&quot; &quot;related_resource_namespace&quot;=&quot;default&quot; &quot;related_resource_version&quot;=&quot;v1&quot; &quot;resource_kind&quot;=&quot;Challenge&quot; &quot;resource_name&quot;=&quot;domain.example.com-r98k2-2809069211-587139531&quot; &quot;resource_namespace&quot;=&quot;default&quot; &quot;resource_version&quot;=&quot;v1&quot; &quot;type&quot;=&quot;HTTP-01&quot;
E0413 12:37:55.260608       1 sync.go:186] cert-manager/challenges &quot;msg&quot;=&quot;propagation check failed&quot; &quot;error&quot;=&quot;wrong status code '404', expected '200'&quot; &quot;dnsName&quot;=&quot;domain.example.com&quot; &quot;resource_kind&quot;=&quot;Challenge&quot; &quot;resource_name&quot;=&quot;domain.example.com-r98k2-2809069211-587139531&quot; &quot;resource_namespace&quot;=&quot;default&quot; &quot;resource_version&quot;=&quot;v1&quot; &quot;type&quot;=&quot;HTTP-01&quot;
I0413 12:37:55.299879       1 pod.go:59] cert-manager/challenges/http01/selfCheck/http01/ensurePod &quot;msg&quot;=&quot;found one existing HTTP01 solver pod&quot; &quot;dnsName&quot;=&quot;domain.example.com&quot; &quot;related_resource_kind&quot;=&quot;Pod&quot; &quot;related_resource_name&quot;=&quot;cm-acme-http-solver-k8wl8&quot; &quot;related_resource_namespace&quot;=&quot;default&quot; &quot;related_resource_version&quot;=&quot;v1&quot; &quot;resource_kind&quot;=&quot;Challenge&quot; &quot;resource_name&quot;=&quot;domain.example.com-r98k2-2809069211-587139531&quot; &quot;resource_namespace&quot;=&quot;default&quot; &quot;resource_version&quot;=&quot;v1&quot; &quot;type&quot;=&quot;HTTP-01&quot;
I0413 12:37:55.300223       1 service.go:43] cert-manager/challenges/http01/selfCheck/http01/ensureService &quot;msg&quot;=&quot;found one existing HTTP01 solver Service for challenge resource&quot; &quot;dnsName&quot;=&quot;domain.example.com&quot; &quot;related_resource_kind&quot;=&quot;Service&quot; &quot;related_resource_name&quot;=&quot;cm-acme-http-solver-gvvkt&quot; &quot;related_resource_namespace&quot;=&quot;default&quot; &quot;related_resource_version&quot;=&quot;v1&quot; &quot;resource_kind&quot;=&quot;Challenge&quot; &quot;resource_name&quot;=&quot;domain.example.com-r98k2-2809069211-587139531&quot; &quot;resource_namespace&quot;=&quot;default&quot; &quot;resource_version&quot;=&quot;v1&quot; &quot;type&quot;=&quot;HTTP-01&quot;
I0413 12:37:55.300570       1 ingress.go:99] cert-manager/challenges/http01/selfCheck/http01/ensureIngress &quot;msg&quot;=&quot;found one existing HTTP01 solver ingress&quot; &quot;dnsName&quot;=&quot;domain.example.com&quot; &quot;related_resource_kind&quot;=&quot;Ingress&quot; &quot;related_resource_name&quot;=&quot;cm-acme-http-solver-pbs7c&quot; &quot;related_resource_namespace&quot;=&quot;default&quot; &quot;related_resource_version&quot;=&quot;v1&quot; &quot;resource_kind&quot;=&quot;Challenge&quot; &quot;resource_name&quot;=&quot;domain.example.com-r98k2-2809069211-587139531&quot; &quot;resource_namespace&quot;=&quot;default&quot; &quot;resource_version&quot;=&quot;v1&quot; &quot;type&quot;=&quot;HTTP-01&quot;
E0413 12:37:55.316802       1 sync.go:186] cert-manager/challenges &quot;msg&quot;=&quot;propagation check failed&quot; &quot;error&quot;=&quot;wrong status code '404', expected '200'&quot; &quot;dnsName&quot;=&quot;domain.example.com&quot; &quot;resource_kind&quot;=&quot;Challenge&quot; &quot;resource_name&quot;=&quot;domain.example.com-r98k2-2809069211-587139531&quot; &quot;resource_namespace&quot;=&quot;default&quot; &quot;resource_version&quot;=&quot;v1&quot; &quot;type&quot;=&quot;HTTP-01&quot;
I0413 12:38:05.261345       1 pod.go:59] cert-manager/challenges/http01/selfCheck/http01/ensurePod &quot;msg&quot;=&quot;found one existing HTTP01 solver pod&quot; &quot;dnsName&quot;=&quot;domain.example.com&quot; &quot;related_resource_kind&quot;=&quot;Pod&quot; &quot;related_resource_name&quot;=&quot;cm-acme-http-solver-k8wl8&quot; &quot;related_resource_namespace&quot;=&quot;default&quot; &quot;related_resource_version&quot;=&quot;v1&quot; &quot;resource_kind&quot;=&quot;Challenge&quot; &quot;resource_name&quot;=&quot;domain.example.com-r98k2-2809069211-587139531&quot; &quot;resource_namespace&quot;=&quot;default&quot; &quot;resource_version&quot;=&quot;v1&quot; &quot;type&quot;=&quot;HTTP-01&quot;
I0413 12:38:05.263416       1 service.go:43] cert-manager/challenges/http01/selfCheck/http01/ensureService &quot;msg&quot;=&quot;found one existing HTTP01 solver Service for challenge resource&quot; &quot;dnsName&quot;=&quot;domain.example.com&quot; &quot;related_resource_kind&quot;=&quot;Service&quot; &quot;related_resource_name&quot;=&quot;cm-acme-http-solver-gvvkt&quot; &quot;related_resource_namespace&quot;=&quot;default&quot; &quot;related_resource_version&quot;=&quot;v1&quot; &quot;resource_kind&quot;=&quot;Challenge&quot; &quot;resource_name&quot;=&quot;domain.example.com-r98k2-2809069211-587139531&quot; &quot;resource_namespace&quot;=&quot;default&quot; &quot;resource_version&quot;=&quot;v1&quot; &quot;type&quot;=&quot;HTTP-01&quot;
I0413 12:38:05.263822       1 ingress.go:99] cert-manager/challenges/http01/selfCheck/http01/ensureIngress &quot;msg&quot;=&quot;found one existing HTTP01 solver ingress&quot; &quot;dnsName&quot;=&quot;domain.example.com&quot; &quot;related_resource_kind&quot;=&quot;Ingress&quot; &quot;related_resource_name&quot;=&quot;cm-acme-http-solver-pbs7c&quot; &quot;related_resource_namespace&quot;=&quot;default&quot; &quot;related_resource_version&quot;=&quot;v1&quot; &quot;resource_kind&quot;=&quot;Challenge&quot; &quot;resource_name&quot;=&quot;domain.example.com-r98k2-2809069211-587139531&quot; &quot;resource_namespace&quot;=&quot;default&quot; &quot;resource_version&quot;=&quot;v1&quot; &quot;type&quot;=&quot;HTTP-01&quot;
E0413 12:38:25.541964       1 sync.go:386] cert-manager/challenges/acceptChallenge &quot;msg&quot;=&quot;error waiting for authorization&quot; &quot;error&quot;=&quot;context deadline exceeded&quot; &quot;dnsName&quot;=&quot;domain.example.com&quot; &quot;resource_kind&quot;=&quot;Challenge&quot; &quot;resource_name&quot;=&quot;domain.example.com-r98k2-2809069211-587139531&quot; &quot;resource_namespace&quot;=&quot;default&quot; &quot;resource_version&quot;=&quot;v1&quot; &quot;type&quot;=&quot;HTTP-01&quot;
E0413 12:38:25.542087       1 controller.go:166] cert-manager/challenges &quot;msg&quot;=&quot;re-queuing item due to error processing&quot; &quot;error&quot;=&quot;context deadline exceeded&quot; &quot;key&quot;=&quot;default/domain.example.com-r98k2-2809069211-587139531&quot;
I0413 12:38:30.542803       1 pod.go:59] cert-manager/challenges/http01/selfCheck/http01/ensurePod &quot;msg&quot;=&quot;found one existing HTTP01 solver pod&quot; &quot;dnsName&quot;=&quot;domain.example.com&quot; &quot;related_resource_kind&quot;=&quot;Pod&quot; &quot;related_resource_name&quot;=&quot;cm-acme-http-solver-k8wl8&quot; &quot;related_resource_namespace&quot;=&quot;default&quot; &quot;related_resource_version&quot;=&quot;v1&quot; &quot;resource_kind&quot;=&quot;Challenge&quot; &quot;resource_name&quot;=&quot;domain.example.com-r98k2-2809069211-587139531&quot; &quot;resource_namespace&quot;=&quot;default&quot; &quot;resource_version&quot;=&quot;v1&quot; &quot;type&quot;=&quot;HTTP-01&quot;
I0413 12:38:30.543062       1 service.go:43] cert-manager/challenges/http01/selfCheck/http01/ensureService &quot;msg&quot;=&quot;found one existing HTTP01 solver Service for challenge resource&quot; &quot;dnsName&quot;=&quot;domain.example.com&quot; &quot;related_resource_kind&quot;=&quot;Service&quot; &quot;related_resource_name&quot;=&quot;cm-acme-http-solver-gvvkt&quot; &quot;related_resource_namespace&quot;=&quot;default&quot; &quot;related_resource_version&quot;=&quot;v1&quot; &quot;resource_kind&quot;=&quot;Challenge&quot; &quot;resource_name&quot;=&quot;domain.example.com-r98k2-2809069211-587139531&quot; &quot;resource_namespace&quot;=&quot;default&quot; &quot;resource_version&quot;=&quot;v1&quot; &quot;type&quot;=&quot;HTTP-01&quot;
I0413 12:38:30.543218       1 ingress.go:99] cert-manager/challenges/http01/selfCheck/http01/ensureIngress &quot;msg&quot;=&quot;found one existing HTTP01 solver ingress&quot; &quot;dnsName&quot;=&quot;domain.example.com&quot; &quot;related_resource_kind&quot;=&quot;Ingress&quot; &quot;related_resource_name&quot;=&quot;cm-acme-http-solver-pbs7c&quot; &quot;related_resource_namespace&quot;=&quot;default&quot; &quot;related_resource_version&quot;=&quot;v1&quot; &quot;resource_kind&quot;=&quot;Challenge&quot; &quot;resource_name&quot;=&quot;domain.example.com-r98k2-2809069211-587139531&quot; &quot;resource_namespace&quot;=&quot;default&quot; &quot;resource_version&quot;=&quot;v1&quot; &quot;type&quot;=&quot;HTTP-01&quot;
E0413 12:38:46.682039       1 sync.go:386] cert-manager/challenges/acceptChallenge &quot;msg&quot;=&quot;error waiting for authorization&quot; &quot;error&quot;=&quot;acme: authorization error for domain.example.com: 400 urn:ietf:params:acme:error:dns: During secondary validation: DNS problem: query timed out looking up A for domain.example.com; DNS problem: query timed out looking up AAAA for domain.example.com&quot; &quot;dnsName&quot;=&quot;domain.example.com&quot; &quot;resource_kind&quot;=&quot;Challenge&quot; &quot;resource_name&quot;=&quot;domain.example.com-r98k2-2809069211-587139531&quot; &quot;resource_namespace&quot;=&quot;default&quot; &quot;resource_version&quot;=&quot;v1&quot; &quot;type&quot;=&quot;HTTP-01&quot;
E0413 12:38:46.888731       1 controller.go:102] ingress 'default/cm-acme-http-solver-pbs7c' in work queue no longer exists
</code></pre>
<p>Looking at Traefik's pod log, I can see that the ingress gets created, but that Traefik is unable to route any requests to it because it can't find the endpoint (this is what I tried to fix with the annotation in the ingressTemplate above):</p>
<pre><code>time=&quot;2022-04-13T12:37:57Z&quot; level=error msg=&quot;Skipping service: no endpoints found&quot; providerName=kubernetes namespace=default servicePort=&quot;&amp;ServiceBackendPort{Name:,Number:8089,}&quot; ingress=cm-acme-http-solver-pbs7c serviceName=cm-acme-http-solver-gvvkt
time=&quot;2022-04-13T12:38:46Z&quot; level=error msg=&quot;Skipping service: no endpoints found&quot; serviceName=cm-acme-http-solver-gvvkt servicePort=&quot;&amp;ServiceBackendPort{Name:,Number:8089,}&quot; providerName=kubernetes ingress=cm-acme-http-solver-pbs7c namespace=default
time=&quot;2022-04-13T12:38:46Z&quot; level=error msg=&quot;Cannot create service: service not found&quot; servicePort=&quot;&amp;ServiceBackendPort{Name:,Number:8089,}&quot; providerName=kubernetes ingress=cm-acme-http-solver-pbs7c namespace=default serviceName=cm-acme-http-solver-gvvkt
time=&quot;2022-04-13T12:38:46Z&quot; level=error msg=&quot;Cannot create service: service not found&quot; servicePort=&quot;&amp;ServiceBackendPort{Name:,Number:8089,}&quot; namespace=default providerName=kubernetes serviceName=cm-acme-http-solver-gvvkt ingress=cm-acme-http-solver-pbs7c
</code></pre>
<p>And there's where I'm stuck currently, since the plan is to use Traefik's <code>IngressRoute</code> CRD for exposing hosts and not use regular ingress entries. Another option would be to test the experimental Gateway support, but as this is the initial setup for a prod cluster I'm not planning to go down that route yet.</p>
<p>Any ideas or further debug information that could be useful?</p>

<p>We have faced the same issue and the problem was related to the fact, that the Ingress generated by the <code>certificate manger</code> contained the <code>Ingress Controller</code> reference using the deprecated Annotation <code>kubernetes.io/ingress.class</code>.</p>
<p>What we wanted:</p>
<pre><code>spec:
  ingressClassName: my-traefik-controller
</code></pre>
<p>What we got:</p>
<pre><code>annotations:
  kubernetes.io/ingress.class: &quot;my-traefik-controller&quot;
</code></pre>
<p>This way, the <code>traefik</code> Ingress Controlelr found the Ingress, but was not able to find the service.
There is a whole discussion on this topic in the <a href="https://github.com/cert-manager/cert-manager/issues/4821" rel="nofollow noreferrer">cert-manger Github repo</a>.</p>
<p>The solution was to use the <a href="https://cert-manager.io/docs/usage/ingress/#supported-annotations" rel="nofollow noreferrer">cert-manager Annotation</a> <code>acme.cert-manager.io/http01-edit-in-place: &quot;true&quot;</code> on an existing Ingress.</p>
<pre><code>  annotations:
    cert-manager.io/cluster-issuer: my-issuer
    acme.cert-manager.io/http01-edit-in-place: &quot;true&quot;
  spec:
    ingressClassName: my-traefik-controller
</code></pre>
<p>This way, only the existing Ingress (containing the correct <code>ingressClassName</code> reference) gets modified and no new solver Ingress gets created.</p>
