<p>I'm new to Kubernetes.</p>

<ul>
<li>I have installed a freshly new Kubernetes Cluster by using RKE (rancher tool for creating k8 clusters).</li>
<li>I added the gitlab chart (<a href="https://charts.gitlab.io/" rel="nofollow noreferrer">https://charts.gitlab.io/</a>) and launch it.</li>
<li>Being on several issues with PersistentStorage, etc that I managed to resolve.</li>
</ul>

<p>But I'm now stuck on one last issue: the pod for <code>gitlab-runner</code> is failing with the following logs:</p>

<pre><code>ERROR: Registering runner... failed runner=Mk5hMxa5 status=couldn't execute POST against https://gitlab.mydomain.com/api/v4/runners: Post https://gitlab.mydomain.com/api/v4/runners: x509: certificate is valid for ingress.local, not gitlab.mydomain.com
PANIC: Failed to register this runner. Perhaps you are having network problems
</code></pre>

<p>Description of the certificate using <code>kubectl describe certificate gitlab-gitlab-tls -n gitlab</code>: </p>

<pre><code>Name:         gitlab-gitlab-tls
Namespace:    gitlab
Labels:       app=unicorn
              chart=unicorn-2.4.6
              heritage=Tiller
              io.cattle.field/appId=gitlab
              release=gitlab
Annotations:  &lt;none&gt;
API Version:  certmanager.k8s.io/v1alpha1
Kind:         Certificate
Metadata:
  Creation Timestamp:  2019-11-13T13:49:10Z
  Generation:          3
  Owner References:
    API Version:           extensions/v1beta1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Ingress
    Name:                  gitlab-unicorn
    UID:                   5640645f-550b-4073-bdf0-df8b089b0c94
  Resource Version:        6824
  Self Link:               /apis/certmanager.k8s.io/v1alpha1/namespaces/gitlab/certificates/gitlab-gitlab-tls
  UID:                     30ac32bd-c7f3-4f9b-9e3b-966b6090e1a9
Spec:
  Acme:
    Config:
      Domains:
        gitlab.mydomain.com
      http01:
        Ingress Class:  gitlab-nginx
  Dns Names:
    gitlab.mydomain.com
  Issuer Ref:
    Kind:       Issuer
    Name:       gitlab-issuer
  Secret Name:  gitlab-gitlab-tls
Status:
  Conditions:
    Last Transition Time:  2019-11-13T13:49:10Z
    Message:               Certificate issuance in progress. Temporary certificate issued.
    Reason:                TemporaryCertificate
    Status:                False
    Type:                  Ready
Events:                    &lt;none&gt;
</code></pre>

<p>Description of the issuer using <code>kubectl describe issuer gitlab-issuer -n gitlab</code>:</p>

<pre><code>Name:         gitlab-issuer
Namespace:    gitlab
Labels:       app=certmanager-issuer
              chart=certmanager-issuer-0.1.0
              heritage=Tiller
              release=gitlab
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"certmanager.k8s.io/v1alpha1","kind":"Issuer","metadata":{"annotations":{},"creationTimestamp":"2019-11-13T13:49:10Z","gener...
API Version:  certmanager.k8s.io/v1alpha1
Kind:         Issuer
Metadata:
  Creation Timestamp:  2019-11-13T13:49:10Z
  Generation:          4
  Resource Version:    24537
  Self Link:           /apis/certmanager.k8s.io/v1alpha1/namespaces/gitlab/issuers/gitlab-issuer
  UID:                 b9971d7a-5220-47ca-a7f9-607aa3f9be4f
Spec:
  Acme:
    Email:  mh@mydomain.com
    http01:
    Private Key Secret Ref:
      Name:  gitlab-acme-key
    Server:  https://acme-v02.api.letsencrypt.org/directory
Status:
  Acme:
    Last Registered Email:  mh@mydomain.com
    Uri:                    https://acme-v02.api.letsencrypt.org/acme/acct/71695690
  Conditions:
    Last Transition Time:  2019-11-13T13:49:12Z
    Message:               The ACME account was registered with the ACME server
    Reason:                ACMEAccountRegistered
    Status:                True
    Type:                  Ready
Events:                    &lt;none&gt;
</code></pre>

<p>Description of the challenge using <code>kubectl describe challenges.certmanager.k8s.io -n gitlab gitlab-gitlab-tls-3386074437-0</code>:</p>

<pre><code>Name:         gitlab-gitlab-tls-3386074437-0
Namespace:    gitlab
Labels:       acme.cert-manager.io/order-name=gitlab-gitlab-tls-3386074437
Annotations:  &lt;none&gt;
API Version:  certmanager.k8s.io/v1alpha1
Kind:         Challenge
Metadata:
  Creation Timestamp:  2019-11-13T13:49:15Z
  Finalizers:
    finalizer.acme.cert-manager.io
  Generation:  4
  Owner References:
    API Version:           certmanager.k8s.io/v1alpha1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Order
    Name:                  gitlab-gitlab-tls-3386074437
    UID:                   1f01771e-2e38-491f-9b2d-ab5f4fda60e2
  Resource Version:        6915
  Self Link:               /apis/certmanager.k8s.io/v1alpha1/namespaces/gitlab/challenges/gitlab-gitlab-tls-3386074437-0
  UID:                     4c115a6f-a76f-4859-a5db-6acd9c039d71
Spec:
  Authz URL:  https://acme-v02.api.letsencrypt.org/acme/authz-v3/1220588820
  Config:
    http01:
      Ingress Class:  gitlab-nginx
  Dns Name:           gitlab.mydomain.com
  Issuer Ref:
    Kind:    Issuer
    Name:    gitlab-issuer
  Key:       lSJdy9Os7BmI56EQCkcEl8t36pcR1hWNjri2Vvq0iv8.lPWns02SmS3zXwFzHdma_RyhwwlzWLRDkdlugFXDlZY
  Token:     lSJdy9Os7BmI56EQCkcEl8t36pcR1hWNjri2Vvq0iv8
  Type:      http-01
  URL:       https://acme-v02.api.letsencrypt.org/acme/chall-v3/1220588820/AwsnPw
  Wildcard:  false
Status:
  Presented:   true
  Processing:  true
  Reason:      Waiting for http-01 challenge propagation: wrong status code '404', expected '200'
  State:       pending
Events:        &lt;none&gt;
</code></pre>

<p>Logs found in <code>cert-manager</code> pod:</p>

<pre><code>I1113 14:20:21.857235       1 pod.go:58] cert-manager/controller/challenges/http01/selfCheck/http01/ensurePod "level"=0 "msg"="found one existing HTTP01 solver pod" "dnsName"="gitlab.mydomain.com" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-ttkmj" "related_resource_namespace"="gitlab" "resource_kind"="Challenge" "resource_name"="gitlab-gitlab-tls-3386074437-0" "resource_namespace"="gitlab" "type"="http-01" 
I1113 14:20:21.857458       1 service.go:43] cert-manager/controller/challenges/http01/selfCheck/http01/ensureService "level"=0 "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="gitlab.mydomain.com" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-sdlw7" "related_resource_namespace"="gitlab" "resource_kind"="Challenge" "resource_name"="gitlab-gitlab-tls-3386074437-0" "resource_namespace"="gitlab" "type"="http-01" 
I1113 14:20:21.857592       1 ingress.go:91] cert-manager/controller/challenges/http01/selfCheck/http01/ensureIngress "level"=0 "msg"="found one existing HTTP01 solver ingress" "dnsName"="gitlab.mydomain.com" "related_resource_kind"="Ingress" "related_resource_name"="cm-acme-http-solver-7jzwk" "related_resource_namespace"="gitlab" "resource_kind"="Challenge" "resource_name"="gitlab-gitlab-tls-3386074437-0" "resource_namespace"="gitlab" "type"="http-01" 
E1113 14:20:21.864785       1 sync.go:183] cert-manager/controller/challenges "msg"="propagation check failed" "error"="wrong status code '404', expected '200'" "dnsName"="gitlab.mydomain.com" "resource_kind"="Challenge" "resource_name"="gitlab-gitlab-tls-3386074437-0" "resource_namespace"="gitlab" "type"="http-01" 
</code></pre>

<ul>
<li>The DNS gitlab.mydomain.com is set to point to the IP of my LoadBalancer where NGINX is running.</li>
<li>If I go to <code>https://gitlab.mydomain.com</code> in the browser: 

<ul>
<li>The browser is saying the connexion is not secure</li>
<li>The result is "default backend - 404".</li>
</ul></li>
</ul>

<h1>Edits</h1>

<p>Description of the ingress-controller by using <code>kubectl describe svc gitlab-nginx-ingress-controller -n gitlab</code>:</p>

<pre><code>Name:              gitlab-nginx-ingress-controller
Namespace:         gitlab
Labels:            app=nginx-ingress
                   chart=nginx-ingress-0.30.0-1
                   component=controller
                   heritage=Tiller
                   io.cattle.field/appId=gitlab
                   release=gitlab
Annotations:       field.cattle.io/ipAddresses: null
                   field.cattle.io/targetDnsRecordIds: null
                   field.cattle.io/targetWorkloadIds: null
Selector:          &lt;none&gt;
Type:              ExternalName
IP:
External Name:     gitlab.mydomain.com
Port:              http  80/TCP
TargetPort:        http/TCP
NodePort:          http  31487/TCP
Endpoints:         10.42.0.7:80,10.42.1.9:80,10.42.2.12:80
Port:              https  443/TCP
TargetPort:        https/TCP
NodePort:          https  31560/TCP
Endpoints:         10.42.0.7:443,10.42.1.9:443,10.42.2.12:443
Port:              gitlab-shell  22/TCP
TargetPort:        gitlab-shell/TCP
NodePort:          gitlab-shell  30539/TCP
Endpoints:         10.42.0.7:22,10.42.1.9:22,10.42.2.12:22
Session Affinity:  None
Events:            &lt;none&gt;
</code></pre>

<p>Running <code>kubectl get ingress -n gitlab</code> gives me a bunch of ingress:</p>

<pre><code>NAME                        HOSTS                 ADDRESS             PORTS     AGE
cm-acme-http-solver-5rjg4   minio.mydomain.com      gitlab.mydomain.com   80        4d23h
cm-acme-http-solver-7jzwk   gitlab.mydomain.com     gitlab.mydomain.com   80        4d23h
cm-acme-http-solver-tzs25   registry.mydomain.com   gitlab.mydomain.com   80        4d23h
gitlab-minio                minio.mydomain.com      gitlab.mydomain.com   80, 443   4d23h
gitlab-registry             registry.mydomain.com   gitlab.mydomain.com   80, 443   4d23h
gitlab-unicorn              gitlab.mydomain.com     gitlab.mydomain.com   80, 443   4d23h
</code></pre>

<p>Description of the <code>gitlab-unicorn</code> by using <code>kubectl describe ingress  gitlab-unicron -n gitlab</code></p>

<pre><code>Name:             gitlab-unicorn
Namespace:        gitlab
Address:          gitlab.mydomain.com
Default backend:  default-http-backend:80 (&lt;none&gt;)
TLS:
  gitlab-gitlab-tls terminates gitlab.mydomain.com
Rules:
  Host               Path  Backends
  ----               ----  --------
  gitlab.mydomain.com
                     /                gitlab-unicorn:8181 (10.42.0.9:8181,10.42.1.8:8181)
                     /admin/sidekiq   gitlab-unicorn:8080 (10.42.0.9:8080,10.42.1.8:8080)
Annotations:
  certmanager.k8s.io/issuer:                          gitlab-issuer
  field.cattle.io/publicEndpoints:                    [{"addresses":[""],"port":443,"protocol":"HTTPS","serviceName":"gitlab:gitlab-unicorn","ingressName":"gitlab:gitlab-unicorn","hostname":"gitlab.mydomain.com","path":"/","allNodes":false},{"addresses":[""],"port":443,"protocol":"HTTPS","serviceName":"gitlab:gitlab-unicorn","ingressName":"gitlab:gitlab-unicorn","hostname":"gitlab.mydomain.com","path":"/admin/sidekiq","allNodes":false}]
  kubernetes.io/ingress.class:                        gitlab-nginx
  kubernetes.io/ingress.provider:                     nginx
  nginx.ingress.kubernetes.io/proxy-body-size:        512m
  nginx.ingress.kubernetes.io/proxy-connect-timeout:  15
  nginx.ingress.kubernetes.io/proxy-read-timeout:     600
Events:                                               &lt;none&gt;
</code></pre>

<p>Description of <code>cm-acme-http-solver-7jzwk</code> by using <code>kubectl describe ingress cm-acme-http-solver-7jzwk  -n gitlab</code>:</p>

<pre><code>Name:             cm-acme-http-solver-7jzwk
Namespace:        gitlab
Address:          gitlab.mydomain.com
Default backend:  default-http-backend:80 (&lt;none&gt;)
Rules:
  Host               Path  Backends
  ----               ----  --------
  gitlab.mydomain.com
                     /.well-known/acme-challenge/lSJdy9Os7BmI56EQCkcEl8t36pcR1hWNjri2Vvq0iv8   cm-acme-http-solver-sdlw7:8089 (10.42.2.19:8089)
Annotations:
  field.cattle.io/publicEndpoints:                     [{"addresses":[""],"port":80,"protocol":"HTTP","serviceName":"gitlab:cm-acme-http-solver-sdlw7","ingressName":"gitlab:cm-acme-http-solver-7jzwk","hostname":"gitlab.mydomain.com","path":"/.well-known/acme-challenge/lSJdy9Os7BmI56EQCkcEl8t36pcR1hWNjri2Vvq0iv8","allNodes":false}]
  kubernetes.io/ingress.class:                         gitlab-nginx
  nginx.ingress.kubernetes.io/whitelist-source-range:  0.0.0.0/0,::/0
Events:                                                &lt;none&gt;
</code></pre>

<p>Ports open on my LoadBalancer and on every nodes of my cluster (I know I should close somes but I will first manage to make my gitlab setup working):</p>

<pre><code>80/tcp                     ALLOW       Anywhere
443/tcp                    ALLOW       Anywhere
22/tcp                     ALLOW       Anywhere
2376/tcp                   ALLOW       Anywhere
2379/tcp                   ALLOW       Anywhere
2380/tcp                   ALLOW       Anywhere
6443/tcp                   ALLOW       Anywhere
6783/tcp                   ALLOW       Anywhere
6783:6784/udp              ALLOW       Anywhere
8472/udp                   ALLOW       Anywhere
4789/udp                   ALLOW       Anywhere
9099/tcp                   ALLOW       Anywhere
10250/tcp                  ALLOW       Anywhere
10254/tcp                  ALLOW       Anywhere
30000:32767/tcp            ALLOW       Anywhere
30000:32767/udp            ALLOW       Anywhere
80/tcp (v6)                ALLOW       Anywhere (v6)
443/tcp (v6)               ALLOW       Anywhere (v6)
22/tcp (v6)                ALLOW       Anywhere (v6)
2376/tcp (v6)              ALLOW       Anywhere (v6)
2379/tcp (v6)              ALLOW       Anywhere (v6)
2380/tcp (v6)              ALLOW       Anywhere (v6)
6443/tcp (v6)              ALLOW       Anywhere (v6)
6783/tcp (v6)              ALLOW       Anywhere (v6)
6783:6784/udp (v6)         ALLOW       Anywhere (v6)
8472/udp (v6)              ALLOW       Anywhere (v6)
4789/udp (v6)              ALLOW       Anywhere (v6)
9099/tcp (v6)              ALLOW       Anywhere (v6)
10250/tcp (v6)             ALLOW       Anywhere (v6)
10254/tcp (v6)             ALLOW       Anywhere (v6)
30000:32767/tcp (v6)       ALLOW       Anywhere (v6)
30000:32767/udp (v6)       ALLOW       Anywhere (v6)
</code></pre>

<p><code>kubectl get pods -n gitlab</code></p>

<pre><code>cm-acme-http-solver-4d8s5                               1/1     Running            0          5d
cm-acme-http-solver-ttkmj                               1/1     Running            0          5d
cm-acme-http-solver-ws7kv                               1/1     Running            0          5d
gitlab-certmanager-57bc6fb4fd-6rfds                     1/1     Running            0          5d
gitlab-gitaly-0                                         1/1     Running            0          5d
gitlab-gitlab-exporter-57b99467d4-knbgk                 1/1     Running            0          5d
gitlab-gitlab-runner-64b74bcd59-mxwvm                   0/1     CrashLoopBackOff   10         55m
gitlab-gitlab-shell-cff8b68f7-zng2c                     1/1     Running            0          5d
gitlab-gitlab-shell-cff8b68f7-zqvfr                     1/1     Running            0          5d
gitlab-issuer.1-lqs7c                                   0/1     Completed          0          5d
gitlab-migrations.1-c4njn                               0/1     Completed          0          5d
gitlab-minio-75567fcbb6-jjxhw                           1/1     Running            6          5d
gitlab-minio-create-buckets.1-6zljh                     0/1     Completed          0          5d
gitlab-nginx-ingress-controller-698fbc4c64-4wt97        1/1     Running            0          5d
gitlab-nginx-ingress-controller-698fbc4c64-5kv2h        1/1     Running            0          5d
gitlab-nginx-ingress-controller-698fbc4c64-jxljq        1/1     Running            0          5d
gitlab-nginx-ingress-default-backend-6cd54c5f86-2jrkd   1/1     Running            0          5d
gitlab-nginx-ingress-default-backend-6cd54c5f86-cxlmx   1/1     Running            0          5d
gitlab-postgresql-66d8d9574b-hbx78                      2/2     Running            0          5d
gitlab-prometheus-server-6fb685b9c7-c8bqj               2/2     Running            0          5d
gitlab-redis-7668c4d476-tcln5                           2/2     Running            0          5d
gitlab-registry-7bb984c765-7ww6j                        1/1     Running            0          5d
gitlab-registry-7bb984c765-t5jjq                        1/1     Running            0          5d
gitlab-sidekiq-all-in-1-8fd95bf7b-hfnjz                 1/1     Running            0          5d
gitlab-task-runner-5cd7bf5bb9-gnv8p                     1/1     Running            0          5d
gitlab-unicorn-864bd864f5-47zxg                         2/2     Running            0          5d
gitlab-unicorn-864bd864f5-gjms2                         2/2     Running            0          5d
</code></pre>

<p>Their are 3 acme-http-solver:</p>

<ul>
<li>One for registry.mydomain.com</li>
<li>One for minio.mydomain.com</li>
<li>One for gitlab.mydomain.com</li>
</ul>

<p>The logs for the one pointing to <code>gitlab.mydomain.com</code>:</p>

<pre><code>I1113 13:49:21.207782       1 solver.go:39] cert-manager/acmesolver "level"=0 "msg"="starting listener"  "expected_domain"="gitlab.mydomain.com" "expected_key"="lSJdy9Os7BmI56EQCkcEl8t36pcR1hWNjri2Vvq0iv8.lPWns02SmS3zXwFzHdma_RyhwwlzWLRDkdlugFXDlZY" "expected_token"="lSJdy9Os7BmI56EQCkcEl8t36pcR1hWNjri2Vvq0iv8" "listen_port"=8089
</code></pre>

<p>Results of <code>kubectl get svc -n gitlab</code>:</p>

<pre><code>cm-acme-http-solver-48b2j                 NodePort       10.43.58.52     &lt;none&gt;              8089:30090/TCP                            5d23h
cm-acme-http-solver-h42mk                 NodePort       10.43.23.141    &lt;none&gt;              8089:30415/TCP                            5d23h
cm-acme-http-solver-sdlw7                 NodePort       10.43.86.27     &lt;none&gt;              8089:32309/TCP                            5d23h
gitlab-gitaly                             ClusterIP      None            &lt;none&gt;              8075/TCP,9236/TCP                         5d23h
gitlab-gitlab-exporter                    ClusterIP      10.43.187.247   &lt;none&gt;              9168/TCP                                  5d23h
gitlab-gitlab-shell                       ClusterIP      10.43.246.124   &lt;none&gt;              22/TCP                                    5d23h
gitlab-minio-svc                          ClusterIP      10.43.117.249   &lt;none&gt;              9000/TCP                                  5d23h
gitlab-nginx-ingress-controller           ExternalName   &lt;none&gt;          gitlab.mydomain.com   80:31487/TCP,443:31560/TCP,22:30539/TCP   5d23h
gitlab-nginx-ingress-controller-metrics   ClusterIP      10.43.152.252   &lt;none&gt;              9913/TCP                                  5d23h
gitlab-nginx-ingress-controller-stats     ClusterIP      10.43.173.191   &lt;none&gt;              18080/TCP                                 5d23h
gitlab-nginx-ingress-default-backend      ClusterIP      10.43.116.121   &lt;none&gt;              80/TCP                                    5d23h
gitlab-postgresql                         ClusterIP      10.43.97.139    &lt;none&gt;              5432/TCP                                  5d23h
gitlab-prometheus-server                  ClusterIP      10.43.67.220    &lt;none&gt;              80/TCP                                    5d23h
gitlab-redis                              ClusterIP      10.43.36.138    &lt;none&gt;              6379/TCP,9121/TCP                         5d23h
gitlab-registry                           ClusterIP      10.43.54.244    &lt;none&gt;              5000/TCP                                  5d23h
gitlab-unicorn                            ClusterIP      10.43.76.61     &lt;none&gt;              8080/TCP,8181/TCP                         5d23h
</code></pre>

<p>Logs of the pod <code>gitlab-nginx-ingress-controller-698fbc4c64-jxljq</code> (others nginx-ingress-controller gives same logs): <a href="https://textuploader.com/1o9we" rel="nofollow noreferrer">https://textuploader.com/1o9we</a></p>

<hr>

<p>Any hint on what could be wrong in my configuration ?</p>

<p>Fell free to ask for more information on my setup.</p>

<p>Many thanks.</p>

MHogge
<p>Well the issue is, Gitlab requires a valid SSL certificate for the domain in question. Which you do not seem to have according to the output of:</p>

<pre><code>E1113 14:20:21.864785       1 sync.go:183] cert-manager/controller/challenges "msg"="propagation check failed" "error"="wrong status code '404', expected '200'" "dnsName"="gitlab.mydomain.com" "resource_kind"="Challenge" "resource_name"="gitlab-gitlab-tls-3386074437-0" "resource_namespace"="gitlab" "type"="http-01" 
</code></pre>

<pre><code>Status:
  Presented:   true
  Processing:  true
  Reason:      Waiting for http-01 challenge propagation: wrong status code '404', expected '200'
  State:       pending
</code></pre>

<p>The http-01 challenge is where it will try to do a web request to your domain, and it should return a 200 HTTP response. When you said yourself that <a href="https://gitlab.mydomain.com" rel="nofollow noreferrer">https://gitlab.mydomain.com</a> gives you a 404 response (hence it will fail to issue a valid certificate). To further diagnose this, check the output of the ingress responsible for the domain, and follow it down the "chain" until you identify where the 404 is being responded by.</p>
