<p>I wrote a bash script to automate the deployment of an application in a kubernetes cluster using helm and kubectl. I use <strong>cert-manager</strong> to automate issuing and renewing of TLS certificates, obtained by <strong>letsencrypt</strong>, needed by the application itself.</p>
<p>The script can deploy the application in either one of many environments such as testing (test) and production (prod) using different settings and manifests as needed. For each environment I create a separate namespace and deploy the needed resources in it. In production I use the <strong>letsencrypt</strong> production server (spec.acme.server: <a href="https://acme-v02.api.letsencrypt.org/directory" rel="nofollow noreferrer">https://acme-v02.api.letsencrypt.org/directory</a>) whereas, in any other env such as testing, I use the staging server (spec.acme.server: <a href="https://acme-staging-v02.api.letsencrypt.org/directory" rel="nofollow noreferrer">https://acme-staging-v02.api.letsencrypt.org/directory</a>). <strong>The hostnames I request the certificates for are a different set depending on the environment: xyz.test.mysite.tld in testing vs xyz.mysite.tld in production</strong>. I provide the same contact e-mail address for all environments.</p>
<p>Here the full manifest of the letsencrypt issuer for testing:</p>
<pre><code>apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    email: operations@mysite.tld
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-staging-issuer-private-key
    solvers:
    - http01:
        ingress:
          class: public-test-it-it
</code></pre>
<p>And here the full manifest of the letsencrypt issuer for production:</p>
<pre><code>apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: letsencrypt-production
spec:
  acme:
    email: operations@mysite.tld
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-production-issuer-private-key
    solvers:
    - http01:
        ingress:
          class: public-prod-it-it
</code></pre>
<p>When I deploy the application the first time, either in test or prod environements, everything works as expected, and cert-manager gets the TLS certificates signed by letsencrypt (staging or production server respectively) and stored in secrets.
But when I deploy the application in another environment (so that I have both test and prod running in parallel), cert-manager can't get the certificates signed anymore, and the chain certificaterequest-&gt;order-&gt;challenge stops at the challenge step with the following output:</p>
<pre><code>kubectl describe challenge xyz-tls-certificate
...
Status:
  Presented:   true
  Processing:  true
  Reason:      Waiting for HTTP-01 challenge propagation: wrong status code '404', expected '200'
  State:       pending
Events:        &lt;none&gt;
</code></pre>
<p>and I can verify that indeed I get a 404 when trying to curl any of the challenges' URLs:</p>
<pre><code>curl -v http://xyz.test.mysite.tld/.well-known/acme-challenge/IECcFDmQF_fzGKcA9hJvFGEWRjDCAE_fs8dnBXlr_wY
*   Trying vvv.xxx.yyy.zzz:80...
* Connected to xyz.test.mysite.tld (vvv.xxx.yyy.zzz) port 80 (#0)
&gt; GET /.well-known/acme-challenge/IECcFDmQF_fzGKcA9hJvFGEWRjDCAE_fs8dnBXlr_wY HTTP/1.1
&gt; Host: xyz.test.mysite.tld
&gt; User-Agent: curl/7.74.0
&gt; Accept: */*
&gt; 
* Mark bundle as not supporting multiuse
&lt; HTTP/1.1 404 Not Found
&lt; date: Thu, 21 Jul 2022 09:48:08 GMT
&lt; content-length: 21
&lt; content-type: text/plain; charset=utf-8
&lt; 
* Connection #0 to host xyz.test.mysite.tld left intact
default backend - 404
</code></pre>
<p>So letsencrypt can't access the challenges' URLs and won't sign the TLS certs.</p>
<p>I tried to debug the 404 error and found that I can successfully curl the pods and services backing the challenges from another pod running in the cluster/namespace, but I get 404s from the outside world. This seems like an issue with the ingress controller (HAProxytech/kubernetes-ingress in my case), but I can't explain why the mechanism worked upon first deployment and then not anymore..</p>
<p>I inspected the cert-manager logs and found lines such:</p>
<pre><code>kubectl logs -n cert-manager cert-manager-...
I0721 13:27:45.517637       1 ingress.go:99] cert-manager/challenges/http01/selfCheck/http01/ensureIngress &quot;msg&quot;=&quot;found one existing HTTP01 solver ingress&quot; &quot;dnsName&quot;=&quot;xyz.test.mysite.tld&quot; &quot;related_resource_kind&quot;=&quot;Ingress&quot; &quot;related_resource_name&quot;=&quot;cm-acme-http-solver-8668s&quot; &quot;related_resource_namespace&quot;=&quot;app-test-it-it&quot; &quot;related_resource_version&quot;=&quot;v1&quot; &quot;resource_kind&quot;=&quot;Challenge&quot; &quot;resource_name&quot;=&quot;xyz-tls-certificate-hwvjf-2516368856-1193545890&quot; &quot;resource_namespace&quot;=&quot;app-test-it-it&quot; &quot;resource_version&quot;=&quot;v1&quot; &quot;type&quot;=&quot;HTTP-01&quot; 
E0721 13:27:45.527238       1 sync.go:186] cert-manager/challenges &quot;msg&quot;=&quot;propagation check failed&quot; &quot;error&quot;=&quot;wrong status code '404', expected '200'&quot; &quot;dnsName&quot;=&quot;xyz.test.mysite.tld&quot; &quot;resource_kind&quot;=&quot;Challenge&quot; &quot;resource_name&quot;=&quot;xyz-tls-certificate-hwvjf-2516368856-1193545890&quot; &quot;resource_namespace&quot;=&quot;app-test-it-it&quot; &quot;resource_version&quot;=&quot;v1&quot; &quot;type&quot;=&quot;HTTP-01&quot; 
</code></pre>
<p>which seems to confirm that cert-manager could self-check, from within the cluster, that the challenges' URLs are in place, but those are not reachable by the outside world (propagation check failed).
It seems like cert-manager set-up challenges' pods/services/ingresses all right, but then requests sent to the challenges' URLs are not routed to the backing pods/services. And this only the second time I try to deploy the app..</p>
<p>I also verified that, after issuing the certificates upon the first deployment, cert-manager (correctly) removed all related pods/services/ingresses from the related namespace, so there should not be any conflict from duplicated challenges' resources.</p>
<p>I restate here that the certificates are issued flawlessly the first time I deploy the application, either in test or prod environment, but they won't be issued anymore if I deploy the app again in a different environment.</p>
<p>Any idea why this is the case?</p>

<p>I finally found out what the issue was..</p>
<p>Basically, I was installing a separate HAProxy ingress controller (haproxytech/kubernetes-ingress) per environment (test/prod), and therefore each namespace had its own ingress controller which I referenced in my manifests.
This should have worked in principle, but it turned out cert-manager could not reference the right ingress-controller upon setting up the letsencrypt challenges.</p>
<p>The solution consisted in creating a single HAproxy ingress controller (in its own separate namespace) to serve the whole cluster and be referenced by all other environments/namespaces. This way the challenges for both testing and production environment where correctly set-up by cert-manager and verified by letsencrypt, which signed the required certificates.</p>
<p><strong>In the end I highly recommend using a single HAproxy ingress controller per cluster, installed in its own namespace.</strong>
This configuration is less redundant and eliminates potential issues such as the one I faced.</p>
