<p>I'm trying to generate an SSL certificate with <code>certbot/certbot</code> docker container in kubernetes. I am using <a href="https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/" rel="noreferrer"><code>Job</code> controller</a> for this purpose which looks as the most suitable option. When I run the standalone option, I get the following error:</p>

<blockquote>
  <p>Failed authorization procedure. staging.ishankhare.com (http-01):
  urn:ietf:params:acme:error:connection :: The server could not connect
  to the client to verify the domain :: Fetching
  <a href="http://staging.ishankhare.com/.well-known/acme-challenge/tpumqbcDWudT7EBsgC7IvtSzZvMAuooQ3PmSPh9yng8" rel="noreferrer">http://staging.ishankhare.com/.well-known/acme-challenge/tpumqbcDWudT7EBsgC7IvtSzZvMAuooQ3PmSPh9yng8</a>:
  Timeout during connect (likely firewall problem)</p>
</blockquote>

<p>I've made sure that this isn't due to misconfigured DNS entries by running a simple nginx container, and it resolves properly. Following is my <code>Jobs</code> file:</p>

<pre><code>apiVersion: batch/v1
kind: Job
metadata:
  #labels:
  #  app: certbot-generator
  name: certbot
spec:
  template:
    metadata:
      labels:
        app: certbot-generate
    spec:
      volumes:
        - name: certs
      containers:
        - name: certbot
          image: certbot/certbot
          command: ["certbot"]
          #command: ["yes"]
          args: ["certonly", "--noninteractive", "--agree-tos", "--staging", "--standalone", "-d", "staging.ishankhare.com", "-m", "me@ishankhare.com"]

          volumeMounts:
            - name: certs
              mountPath: "/etc/letsencrypt/"
              #- name: certs
              #mountPath: "/opt/"
          ports:
            - containerPort: 80
            - containerPort: 443
      restartPolicy: "OnFailure"
</code></pre>

<p>and my service:</p>

<pre><code>apiVersion: v1
kind: Service
metadata:
  name: certbot-lb
  labels:
    app: certbot-lb
spec:
  type: LoadBalancer
  loadBalancerIP: 35.189.170.149
  ports:
    - port: 80
      name: "http"
      protocol: TCP
    - port: 443
      name: "tls"
      protocol: TCP
  selector:
    app: certbot-generator
</code></pre>

<p>the full error message is something like this:</p>

<pre><code>Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator standalone, Installer None
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for staging.ishankhare.com
Waiting for verification...
Cleaning up challenges
Failed authorization procedure. staging.ishankhare.com (http-01): urn:ietf:params:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching http://staging.ishankhare.com/.well-known/acme-challenge/tpumqbcDWudT7EBsgC7IvtSzZvMAuooQ3PmSPh9yng8: Timeout during connect (likely firewall problem)
IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: staging.ishankhare.com
   Type:   connection
   Detail: Fetching
   http://staging.ishankhare.com/.well-known/acme-challenge/tpumqbcDWudT7EBsgC7IvtSzZvMAuooQ3PmSPh9yng8:
   Timeout during connect (likely firewall problem)

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address. Additionally, please check that
   your computer has a publicly routable IP address and that no
   firewalls are preventing the server from communicating with the
   client. If you're using the webroot plugin, you should also verify
   that you are serving files from the webroot path you provided.
 - Your account credentials have been saved in your Certbot
   configuration directory at /etc/letsencrypt. You should make a
   secure backup of this folder now. This configuration directory will
   also contain certificates and private keys obtained by Certbot so
   making regular backups of this folder is ideal.
</code></pre>

<p>I've also tried running this as a simple <code>Pod</code> but to no help. Although I still feel running it as a <code>Job</code> to completion is the way to go.</p>

Ishan Khare
<p>First, be aware your <code>Job</code> definition is valid, but the <code>spec.template.metadata.labels.app: certbot-generate</code> value does <strong>not</strong> match with your <code>Service</code> definition <code>spec.selector.app: certbot-generator</code>: one is <code>certbot-generate</code>, the second is <code>certbot-generator</code>. So the pod run by the job controller is never added as an endpoint to the service.</p>

<p>Adjust one or the other, but they have to match, and that might just work :)</p>

<p>Although, I'm not sure using a <code>Service</code> with a selector targeting short-lived pods from a <code>Job</code> controller would work, neither with a simple <code>Pod</code> as you tested. The <code>certbot-randomId</code> pod created by the job (or whatever simple pod you create) takes about 15 seconds total to run/fail, and the HTTP validation challenge is triggered after just a few seconds of the pod life: it's not clear to me that would be enough time for kubernetes proxying to be already working between the service and the pod.</p>

<p>We can safely assume that the <code>Service</code> is actually working because you mentioned that you tested DNS resolution, so you can easily ensure that's not a timing issue by adding a <code>sleep 10</code> (or more!) to give more time for the pod to be added as an endpoint to the service and being proxied appropriately <em>before</em> the HTTP challenge is triggered by certbot. Just change your <code>Job</code> command and args for those:</p>

<pre><code>command: ["/bin/sh"]
args: ["-c", "sleep 10 &amp;&amp; certbot certonly --noninteractive --agree-tos --staging --standalone -d staging.ishankhare.com -m me@ishankhare.com"]
</code></pre>

<p>And here too, that might just work :)</p>

<hr>

<p>That being said, I'd warmly recommend you to use <a href="https://cert-manager.readthedocs.io/en/latest/index.html" rel="nofollow noreferrer">cert-manager</a> which you can install easily through its <a href="https://github.com/helm/charts/tree/master/stable/cert-manager" rel="nofollow noreferrer">stable Helm chart</a>: the <code>Certificate</code> custom resource that it introduces will store your certificate in a <a href="https://kubernetes.io/docs/concepts/configuration/secret/" rel="nofollow noreferrer"><code>Secret</code></a> which will make it straightforward to reuse from whatever K8s resource, and it takes care of renewal automatically so you can just forget about it all.</p>
