<p>I recently upgraded <code>ingress-nginx</code> to version 1.0.3.</p>
<p>As a result, I removed the <code>kubernetes.io/ingress.class</code> annotation from my ingress, and put <code>.spec.ingressClassName</code> instead.</p>
<p>I am running <code>cert-manager-v1.4.0</code>.</p>
<p>This morning I had an email saying that my Let's Encrypt certificate will expire in 10 days. I tried to figure out what was wrong with it - not positive that it was entirely due to the ingress-nginx upgrade.</p>
<p>I deleted the <code>CertificateRequest</code> to see if it would fix itself. I got a new <code>Ingress</code> with the challenge, but:</p>
<ol>
<li><p>The challenge ingress had the <code>kubernetes.io/ingress.class</code> annotation set correctly, even though my ingress has <code>.spec.ingressClassName</code> instead - don't know how or why, but it seems like it should be OK.</p>
</li>
<li><p>However, the challenge ingress wasn't picked up by the ingress controller, it said:</p>
</li>
</ol>
<p><code>ingress class annotation is not equal to the expected by Ingress Controller</code></p>
<p>I guess it wants only the <code>.spec.ingressClassName</code> even though I thought the annotation was supposed to work as well.</p>
<p>So I manually set <code>.spec.ingressClassName</code> on the challenge ingress.  It was immediately seen by the ingress controller, and the rest of the process ran smoothly, and I got a new cert - yay.</p>
<p>It seems to me like this will happen again, so I need to know how to either:</p>
<ol>
<li><p>Convince <code>cert-manager</code> to create the challenge ingress with <code>.spec.ingressClassName</code> instead of <code>kubernetes.io/ingress.class</code>.  Maybe this is fixed in 1.5 or 1.6?</p>
</li>
<li><p>Convince <code>ingress-nginx</code> to respect the <code>kubernetes.io/ingress.class</code> annotation for the challenge ingress.  I don't know why this doesn't work.</p>
</li>
</ol>

<h2>Issue</h2>
<p>The issue was fixed by certificate renewal, it works fine without manually set <code>spec.ingressClassName</code> in challenge ingress (I saw it with older version), issue was somewhere else.</p>
<p>Also with last available (at the writing moment) <code>cert-manager v1.5.4</code> challenge ingress has the right setup &quot;out of the box&quot;:</p>
<pre><code>spec:
  ingressClassName: nginx
---
$ kubectl get ing
NAME                        CLASS    HOSTS            ADDRESS         PORTS     AGE
cm-acme-http-solver-szxfg   nginx    dummy-host       ip_address      80        11s
</code></pre>
<h2>How it works (concept)</h2>
<p>I'll describe main steps how this process works so troubleshooting will be straight-forward in almost all cases. I'll take a <code>letsencypt staging</code> as an <code>issuer</code>.</p>
<p>There's a chain when <code>certificate</code> is requested to be created which <code>issuer</code> follows to complete (all resources have owners - previous resource in chain):</p>
<p><code>main ingress resource</code> -&gt; <code>certificate</code> -&gt; <code>certificaterequest</code> -&gt; <code>order</code> -&gt; <code>challenge</code> -&gt; <code>challenge ingress</code>.</p>
<p>Knowing this, if something failed, you can go down by the chain and using <code>kubectl describe</code> command find where the issue appeared.</p>
<h2>Troubleshooting example</h2>
<p>I intentionally added a wrong domain in ingress to <code>.spec.tls.hosts</code> and applied it. Below how the chain will look like (all names will be unique!):</p>
<p>See certificates:</p>
<pre><code>$ kubectl get cert
NAME                     READY   SECRET                          AGE
lets-secret-test-2       False   lets-secret-test-2              15m
</code></pre>
<p>Describe <code>certificate</code> we are interested in (you can notice I changed domain, there was already secret):</p>
<pre><code>$ kubectl describe cert lets-secret-test-2
Events:
  Type    Reason     Age   From          Message
  ----    ------     ----  ----          -------
  Normal  Issuing    16m   cert-manager  Existing issued Secret is not up to date for spec: [spec.commonName spec.dnsNames]
  Normal  Reused     16m   cert-manager  Reusing private key stored in existing Secret resource &quot;lets-secret-test-2&quot;
  Normal  Requested  16m   cert-manager  Created new CertificateRequest resource &quot;lets-secret-test-2-pvb25&quot;
</code></pre>
<p>Nothing suspicious here, moving forward.</p>
<pre><code>$ kubectl get certificaterequest
NAME                           APPROVED   DENIED   READY   ISSUER                REQUESTOR                                         AGE
lets-secret-test-2-pvb25       True                False   letsencrypt-staging   system:serviceaccount:cert-manager:cert-manager   19m
</code></pre>
<p>Describing <code>certificaterequest</code>:</p>
<pre><code>$ kubectl describe certificaterequest lets-secret-test-2-pvb25
Events:
  Type    Reason           Age   From          Message
  ----    ------           ----  ----          -------
  Normal  cert-manager.io  19m   cert-manager  Certificate request has been approved by cert-manager.io
  Normal  OrderCreated     19m   cert-manager  Created Order resource default/lets-secret-test-2-pvb25-2336849393
</code></pre>
<p>Again, everything looks fine, no errors, moving forward to <code>order</code>:</p>
<pre><code>$ kubectl get order
NAME                                  STATE     AGE
lets-secret-test-2-pvb25-2336849393   pending   21m
</code></pre>
<p>It says <code>pending</code>, that's closer:</p>
<pre><code>$ kubectl describe order lets-secret-test-2-pvb25-2336849393

Events:
  Type    Reason   Age   From          Message
  ----    ------   ----  ----          -------
  Normal  Created  21m   cert-manager  Created Challenge resource &quot;lets-secret-test-2-pvb25-2336849393-3788447910&quot; for domain &quot;dummy-domain&quot;
</code></pre>
<p><code>Challenge</code> may shed some light, moving forward:</p>
<pre><code>$ kubectl get challenge
NAME                                             STATE     DOMAIN           AGE
lets-secret-test-2-pvb25-2336849393-3788447910   pending   dummy-domain   23m
</code></pre>
<p>Describing it:</p>
<pre><code>$ kubectl describe challenge lets-secret-test-2-pvb25-2336849393-3788447910
</code></pre>
<p>Checking <code>status</code>:</p>
<pre><code>Status:
  Presented:   true
  Processing:  true
  Reason:      Waiting for HTTP-01 challenge propagation: failed to perform self check GET request 'http://dummy-domain/.well-known/acme-challenge/xxxxyyyyzzzzz': Get &quot;http://dummy-domain/.well-known/acme-challenge/xxxxyyyyzzzzz&quot;: dial tcp: lookup dummy-domain on xx.yy.zz.ww:53: no such host
  State:       pending
</code></pre>
<p>Now it's clear that something is wrong with <code>domain</code>, worth checking it:</p>
<p>Found and fixed the &quot;mistake&quot;:</p>
<pre><code>$ kubectl apply -f ingress.yaml
ingress.networking.k8s.io/ingress configured
</code></pre>
<p>Certificate is <code>ready</code>!</p>
<pre><code>$ kubectl get cert
NAME                     READY   SECRET                          AGE
lets-secret-test-2       True    lets-secret-test-2              26m
</code></pre>
<h2>Correct way to renew a certificate using cert-manager</h2>
<p>It's possible to renew a certificate by deleting corresponding secret, however <a href="https://cert-manager.io/docs/usage/certificate/#actions-triggering-private-key-rotation" rel="nofollow noreferrer">documentation says it's not recommended</a>:</p>
<blockquote>
<p>Deleting the Secret resource associated with a Certificate resource is
<strong>not a recommended solution</strong> for manually rotating the private key. The
recommended way to manually rotate the private key is to trigger the
reissuance of the Certificate resource with the following command
(requires the kubectl cert-manager plugin):</p>
<p><code>kubectl cert-manager renew cert-1</code></p>
</blockquote>
<p><code>Kubectl cert-manager</code> command installation process is described <a href="https://cert-manager.io/docs/usage/kubectl-plugin/#installation" rel="nofollow noreferrer">here</a> as well as other commands and examples.</p>
<h2>Useful links:</h2>
<ul>
<li><a href="https://cert-manager.io/docs/concepts/certificate/" rel="nofollow noreferrer">Certificate in cert-manager - concept</a></li>
<li><a href="https://cert-manager.io/docs/concepts/certificaterequest/" rel="nofollow noreferrer">Cert-manager - certificate request</a></li>
<li><a href="https://cert-manager.io/docs/concepts/acme-orders-challenges/" rel="nofollow noreferrer">ACME Orders and Challenges</a></li>
</ul>
