<p>We have a dozen of services exposed using a ingress-nginx controller in GKE.</p>
<p>In order to route the traffic correctly on the same domain name, we need to use a rewrite-target rule.</p>
<p>The services worked well without any maintenance since their launch in 2019, that is until recently; when cert-manager suddenly stopped renewing the Let's Encrypt certificates, we &quot;resolved&quot; this by temporarily removing the &quot;tls&quot; section from the ingress definition, forcing our clients to use the http version.</p>
<p>After that we removed all traces of cert-manager attempting to set it up from scratch.</p>
<p>Now, the cert-manager is creating the certificate signing request, spawns an acme http solver pod and adds it to the ingress, however upon accessing its url I can see that it returns an empty response, and not the expected token.</p>
<p><strong>This has to do with the rewrite-target annotation that messes up the routing of the acme challenge.
What puzzles me the most, is that this used to work before. (It was set up by a former employee)</strong></p>
<p>Disabling rewrite-target is unfortunately not an option, because it will stop the routing from working correctly.</p>
<p>Using dns01 won't work because our ISP does not support programmatic changes of the DNS records.</p>
<p><strong>Is there a way to make this work without disabling rewrite-target?</strong></p>
<p>P.S.
Here's a number of similar cases reported on Github:</p>
<ul>
<li><a href="https://github.com/cert-manager/cert-manager/issues/2826" rel="nofollow noreferrer">https://github.com/cert-manager/cert-manager/issues/2826</a></li>
<li><a href="https://github.com/cert-manager/cert-manager/issues/286" rel="nofollow noreferrer">https://github.com/cert-manager/cert-manager/issues/286</a></li>
<li><a href="https://github.com/cert-manager/cert-manager/issues/487" rel="nofollow noreferrer">https://github.com/cert-manager/cert-manager/issues/487</a></li>
</ul>
<p>None of them help.</p>
<p>Here's the definition of my ClusterIssuer</p>
<pre><code>apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    # The ACME server URL
    server: https://acme-v02.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email: mail@domain.com
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-prod
    # Enable the HTTP-01 challenge provider
    solvers:
    - http01:
        ingress:
            class: nginx
</code></pre>

<p>Please share the cluster issuer or issue you are using.</p>
<p><strong>ingressClass</strong></p>
<blockquote>
<p>If the <strong>ingressClass</strong> field is specified, <code>cert-manager</code> will create
new <strong>Ingress</strong> resources in order to route traffic to the
<code>acmesolver</code> pods, which are responsible for responding to <strong>ACME</strong>
challenge validation requests.</p>
</blockquote>
<p>Ref : <a href="https://cert-manager.io/v0.12-docs/configuration/acme/http01/#ingressclass" rel="nofollow noreferrer">https://cert-manager.io/v0.12-docs/configuration/acme/http01/#ingressclass</a></p>
<p>Mostly we don't see the HTTP solver challenge it comes and get removed if DNS or HTTP working fine.</p>
<p>Also, make sure your ingress doesn't have SSL-redirect annotation that could be also once reason behind <strong>certs</strong> not getting generated.</p>
<p>Did you try checking the other object of cert-manager like order and certificate status request ? <code>kubectl describe challenge</code> are you getting <strong>404</strong> there ?</p>
<p>If you are trying continuously there could be chance you hit rate limit of let's encrypt to request generating certificates.</p>
<p>Troubleshooting : <a href="https://cert-manager.io/docs/faq/troubleshooting/#troubleshooting-a-failed-certificate-request" rel="nofollow noreferrer">https://cert-manager.io/docs/faq/troubleshooting/#troubleshooting-a-failed-certificate-request</a></p>
