<p>Good day, i'm newby in kubernetes and try to setup my first environment.
I want to following scheme:</p>

<ul>
<li>My organization has public IP (x.x.x.x)</li>
<li>This IP routed to server in private LAN (i.e. <code>192.168.0.10</code>) with win server + IIS. On IIS i have URL rewrite module and it's act as reverse proxy</li>
<li>I have kubernetes cluster</li>
<li>I have some service deployed to k8s</li>
<li>I want to access this service from the internet with SSL, gained from let's encrypt</li>
</ul>

<p>I have already setup k8s cluster, deploy traefik (<code>v1.7</code>) ingress and configure them for let's encrypt (setup <code>http-&gt;https</code> redirect, setup acme challenge). This works fine - I can observe it from LAN or WAN, and there is no warning about certificate - i see green lock.
Now I deploy service (in my case this is graylog). Again - I can observe it from LAN and WAN, but in this case i see warning about certificate (it was issued by <code>TRAEFIK_DEFAULT_CERT</code>).
After I saw this I try to search more information and found that I need cert-manager. I deploy cert-manager, create let's encrypt issuer (with ClusterIssuer role), but then I try to issue certificate I get following error (found in challenge description):</p>

<pre><code>Waiting for http-01 challenge propagation: wrong status code '404', expected '200'
</code></pre>

<p>My traefik configmap:</p>

<pre><code>apiVersion: v1
kind: ConfigMap
metadata:
  name: traefik-conf
  namespace: kube-system
data:
  traefik.toml: |
    # traefik.toml
    defaultEntryPoints = ["http","https"]
    [entryPoints]
      [entryPoints.http]
      address = ":80"
      [entryPoints.http.redirect]
        regex = "^http://(.*)"
        replacement = "https://$1"
      [entryPoints.https]
      address = ":443"
        [entryPoints.https.tls]
    [acme]
      email = "mymail@example.xyz"
      storage = "/acme/acme.json"
      entryPoint = "https"
      onHostRule = true
        [acme.httpChallenge]
        entryPoint = "http"
    [[acme.domains]]
      main = "my-public-domain.com"
</code></pre>

<p>I also try to use wildcard there:</p>

<pre><code>apiVersion: v1
kind: ConfigMap
metadata:
  name: traefik-conf
  namespace: kube-system
data:
  traefik.toml: |
    # traefik.toml
    defaultEntryPoints = ["http","https"]
    [entryPoints]
      [entryPoints.http]
      address = ":80"
      [entryPoints.http.redirect]
        regex = "^http://(.*)"
        replacement = "https://$1"
      [entryPoints.https]
      address = ":443"
        [entryPoints.https.tls]
    [acme]
      email = "mymail@example.xyz"
      storage = "/acme/acme.json"
      entryPoint = "https"
      onHostRule = true
        [acme.httpChallenge]
        entryPoint = "http"
    [[acme.domains]]
      main = "*.my-public-domain.com"
      sans = ["my-public-domain.com"]
</code></pre>

<p>My cluster issuer:</p>

<pre><code>apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  name: letsencrypt-dev
spec:
  acme:
    email: mymail@example.xyz
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Secret resource used to store the account's private key.
      name: example-issuer-account-key
    # Add a single challenge solver, HTTP01 using nginx
    solvers:
    - http01:
        ingress:
          class: traefik
</code></pre>

<p>And my test certificate:</p>

<pre><code>apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: example-com
  namespace: default
spec:
  secretName: example-com-tls
  renewBefore: 360h # 15d
  commonName: logger.my-public-domain.com
  dnsNames:
  - logger.my-public-domain.com
  issuerRef:
    name: letsencrypt-dev
    kind: ClusterIssuer
</code></pre>

<p>I have DNS entries for wildcard domain and can ping it</p>

<p>My certificates stuck in state <code>OrderCreated</code>:</p>

<pre><code>Status:
  Conditions:
    Last Transition Time:  2019-10-08T09:40:30Z
    Message:               Certificate issuance in progress. Temporary certificate issued.
    Reason:                TemporaryCertificate
    Status:                False
    Type:                  Ready
Events:
  Type    Reason        Age   From          Message
  ----    ------        ----  ----          -------
  Normal  OrderCreated  47m   cert-manager  Created Order resource "example-com-3213698372"
</code></pre>

<p>Order stuck in state <code>Created</code>:</p>

<pre><code>Status:
  Challenges:
    Authz URL:  &lt;url&gt;
    Dns Name:   logger.my-public-domain.com
    Issuer Ref:
      Kind:  ClusterIssuer
      Name:  letsencrypt-dev
    Key:     &lt;key&gt;
    Solver:
      http01:
        Ingress:
          Class:  traefik
    Token:        &lt;token&gt;
    Type:         http-01
    URL:          &lt;url&gt;
    Wildcard:     false
  Finalize URL:   &lt;url&gt;
  State:          pending
  URL:            &lt;url&gt;
Events:
  Type    Reason   Age   From          Message
  ----    ------   ----  ----          -------
  Normal  Created  48m   cert-manager  Created Challenge resource "example-com-3213698372-0" for domain "logger.my-public-domain.com"
</code></pre>

<p>And, at last, by challenge:</p>

<pre><code>Spec:
  Authz URL:  &lt;url&gt;
  Dns Name:   logger.my-public-domain.com
  Issuer Ref:
    Kind:  ClusterIssuer
    Name:  letsencrypt-dev
  Key:     &lt;key&gt;
  Solver:
    http01:
      Ingress:
        Class:  traefik
  Token:        &lt;token&gt;
  Type:         http-01
  URL:          &lt;url&gt;
  Wildcard:     false
Status:
  Presented:   true
  Processing:  true
  Reason:      Waiting for http-01 challenge propagation: wrong status code '404', expected '200'
  State:       pending
Events:
  Type    Reason     Age   From          Message
  ----    ------     ----  ----          -------
  Normal  Started    55m   cert-manager  Challenge scheduled for processing
  Normal  Presented  55m   cert-manager  Presented challenge using http-01 challenge mechanism
</code></pre>

<p>I see that there is 404 response, but I can't understand the reason of it.
On my IIS I have following rewrite rules:
Let's encrypt bypass - all url matched by <code>.well-known/*</code> rewrited to kubernetes host. http to https - all url not matched by let's encrypt redirected to https
sub-domain redirect - all subdomains request rewrited to kubernetes.</p>

<p>In my LAN i have own DNS server, there all domains from <code>my-public-domain.com</code> has mapping to internal addresses, so I can redirect public hostname <code>logger.my-public-domain.com (x.x.x.x)</code> to internal <code>logger.my-public-domain.com (192.168.0.y)</code>.</p>

<p>While challenge active I cat see new backend and frontend in traefik dashboard.</p>

<p>Maybe I misunderstand how it should works, but I expected that cert-manager issue certificate with let's encrypt and I can observe my service.</p>

<p>In your <code>traefik.toml</code> ConfigMap you're redirecting to HTTPS:</p>
<pre><code>[entryPoints.http.redirect]
regex = &quot;^http://(.*)&quot;
replacement = &quot;https://$1&quot;
</code></pre>
<p>Remove that replacement using <code>kubectl edit configmap -n kube-system traefik</code>, save the changes then restart the Traefik pod. You should be good to go then. Use your Ingress to manage the redirects with annotations instead of putting them in the Traefik config.</p>
