<h3>What I'm trying to achieve</h3>
<p>I'm trying to deploy an elixir (phoenix) application in a microk8s cluster namespace with TLS using let's encrypt. The cluster is hosted on an AWS EC2 instance.</p>
<h3>The problem I'm facing</h3>
<ul>
<li>The ingress is created in the namespace</li>
<li>ingress routes to the correct domain</li>
<li>the application is working and displayed on the given domain</li>
</ul>
<p><strong>The TLS secret is not being created in the namespace and a 'default' one is created</strong></p>
<p>The secrets after deploying both phoenix app and httpbin app:</p>
<pre class="lang-sh prettyprint-override"><code>me@me:~/Documents/kubernetes-test$ kubectl get secret -n production
NAME                           TYPE                                  DATA   AGE
default-token-jmgrg            kubernetes.io/service-account-token   3      20m
httpbin-tls                    kubernetes.io/tls                     2      81s
</code></pre>
<p><strong>The domain is insecure, i.e the TLS is not working.</strong></p>
<p>Logs from the ingress controller after applying the yml files:</p>
<pre><code>W0106 17:26:36.967036       6 controller.go:1192] Error getting SSL certificate &quot;production/phoenix-app-tls&quot;: local SSL certificate production/phoenix-app-tls was not found. Using default certificate
W0106 17:26:46.445248       6 controller.go:1192] Error getting SSL certificate &quot;production/phoenix-app-tls&quot;: local SSL certificate production/phoenix-app-tls was not found. Using default certificate
W0106 17:26:49.779680       6 controller.go:1192] Error getting SSL certificate &quot;production/phoenix-app-tls&quot;: local SSL certificate production/phoenix-app-tls was not found. Using default certificate
I0106 17:26:56.431925       6 status.go:281] &quot;updating Ingress status&quot; namespace=&quot;production&quot; ingress=&quot;phoenix-app-ingress&quot; currentValue=[] newValue=[{IP:127.0.0.1 Hostname: Ports:[]}]
I0106 17:26:56.443405       6 event.go:282] Event(v1.ObjectReference{Kind:&quot;Ingress&quot;, Namespace:&quot;production&quot;, Name:&quot;phoenix-app-ingress&quot;, UID:&quot;REDACTED&quot;, APIVersion:&quot;networking.k8s.io/v1beta1&quot;, ResourceVersion:&quot;1145907&quot;, FieldPath:&quot;&quot;}): type: 'Normal' reason: 'Sync' Scheduled for sync
W0106 17:26:56.443655       6 backend_ssl.go:46] Error obtaining X.509 certificate: no object matching key &quot;production/phoenix-app-tls&quot; in local store
W0106 17:26:56.443781       6 controller.go:1192] Error getting SSL certificate &quot;production/phoenix-app-tls&quot;: local SSL certificate production/phoenix-app-tls was not found. Using default certificate
</code></pre>
<p>The description of the created ingress, note that here at the bottom it says <code>Successfully created Certificate &quot;phoenix-app-tls&quot; but the secret does not exist</code>:</p>
<pre class="lang-sh prettyprint-override"><code>me@me:~/Documents/kubernetes-test$ kubectl describe ing phoenix-app-ingress -n production
Name:             phoenix-app-ingress
Labels:           app=phoenix-app
Namespace:        production
Address:          127.0.0.1
Default backend:  default-http-backend:80 (&lt;error: endpoints &quot;default-http-backend&quot; not found&gt;)
TLS:
  phoenix-app-tls terminates phoenix.sub.mydomain.com
Rules:
  Host                           Path  Backends
  ----                           ----  --------
  phoenix.sub.mydomain.com  
                                 /   phoenix-app-service-headless:8000 (REDACTED_IP:4000,REDACTED_IP:4000)
Annotations:                     cert-manager.io/cluster-issuer: letsencrypt
                                 nginx.ingress.kubernetes.io/cors-allow-credentials: true
                                 nginx.ingress.kubernetes.io/cors-allow-methods: GET, POST, OPTIONS
                                 nginx.ingress.kubernetes.io/cors-allow-origin: *
                                 nginx.ingress.kubernetes.io/enable-cors: true
Events:
  Type    Reason             Age                  From                      Message
  ----    ------             ----                 ----                      -------
  Normal  CreateCertificate  29m                  cert-manager              Successfully created Certificate &quot;phoenix-app-tls&quot;
  Normal  Sync               8m43s (x3 over 29m)  nginx-ingress-controller  Scheduled for sync
</code></pre>
<h3>Resources</h3>
<p>The deployment yml:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: phoenix-app
  labels:
    app: phoenix-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: phoenix-app
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: phoenix-app
    spec:
      containers:
      - name: phoenix-app
        image: REDACTED
        imagePullPolicy: Always
        command: [&quot;./bin/hello&quot;, &quot;start&quot;]
        lifecycle:
          preStop:
            exec:
              command: [&quot;./bin/hello&quot;, &quot;stop&quot;]
        ports:
        - containerPort: 4000
        env:
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        envFrom:
        - configMapRef:
            name: phoenix-app-config
        - secretRef:
            name: phoenix-app-secrets
      imagePullSecrets:
      - name: gitlab-pull-secret
</code></pre>
<p>The service yml:</p>
<pre><code>apiVersion: v1
kind: Service
metadata:
  name: phoenix-app-service-headless
  labels:
    app: phoenix-app
spec:
  clusterIP: None
  selector:
    app: phoenix-app
  ports:
  - name: http
    port: 8000
    targetPort: 4000 # The exposed port by the phoenix app
</code></pre>
<p>Note: I removed my actual domain</p>
<p>The ingress yml:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: phoenix-app-ingress
  labels:
    app: phoenix-app
  annotations:
    nginx.ingress.kubernetes.io/enable-cors: &quot;true&quot;
    nginx.ingress.kubernetes.io/cors-allow-methods: &quot;GET, POST, OPTIONS&quot;
    nginx.ingress.kubernetes.io/cors-allow-origin: &quot;*&quot;
    nginx.ingress.kubernetes.io/cors-allow-credentials: &quot;true&quot;
    cert-manager.io/cluster-issuer: &quot;letsencrypt&quot;
spec:
  tls:
  - hosts:
    - &quot;phoenix.sub.mydomain.com&quot;
    secretName: phoenix-app-tls
  rules:
  - host: &quot;phoenix.sub.mydomain.com&quot;
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: phoenix-app-service-headless
            port:
              number: 8000 # Same port as in service.yml
</code></pre>
<h3>Tested with different service</h3>
<p>I deployed a sample service using httpbin (is not a headless service) and the TLS works fine in the same namespace. Here are the resources that I used to deploy it:</p>
<p>deplyoment.yml</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
  labels:
    app: httpbin
spec:
  replicas: 1
  selector: 
    matchLabels:
      app: httpbin
      version: v1
  template:
    metadata:
      labels:
        app: httpbin
        version: v1
    spec:
      containers:
      - image: docker.io/kennethreitz/httpbin
        imagePullPolicy: Always
        name: httpbin
        ports:
        - containerPort: 80
</code></pre>
<p>The service yml:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: Service
metadata:
  name: httpbin
  labels:
    app: httpbin
spec:
  ports:
  - name: http
    port: 8000
    targetPort: 80
  selector:
    app: httpbin
</code></pre>
<p>The ingress yml:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: httpbin
  labels:
    app: httpbin
  annotations:
    cert-manager.io/cluster-issuer: &quot;letsencrypt&quot;
spec:
  tls:
  - hosts:
    - &quot;httpbin.sub.mydomain.com&quot;
    secretName: httpbin-tls
  rules:
  - host: &quot;httpbin.sub.mydomain.com&quot; # This is a subdomain we want to route these requests to
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: httpbin
            port:
              number: 8000
</code></pre>
<p>My best guess is that it has something to do with the fact that the service is headless, but I have no clue as to how I can resolve the issue.</p>

Beefcake
<p>I found out that you can actually check for certificates with kubectl:
<code>kubectl get certificate -n production</code></p>
<p>The status of this certificate was READY = FALSE.</p>
<p>I checked the description:
<code>kubectl describe certificate &lt;certificate_name&gt; -n production</code></p>
<p>At the bottom it said:
Too many certificates have been created in the last 164 hours for this exact domain.</p>
<p>I just changed the domain and voila! It works.</p>
