<p>I have a local K3s kubernetes cluster with its traefik ingress controller.</p>

<p>(Mac OSX, Multipass Hyper-V based local VMs: <code>v1.18.3+k3s1    Ubuntu 16.04.6 LTS   4.4.0-179-generic   containerd://1.3.3-k3s2</code>)</p>

<p>what I want is <strong>having an ingress that is tls enabled</strong> AND forwarding to vault port 8200 via tls/https</p>

<pre><code>k get -n kube-system svc traefik
NAME      TYPE           CLUSTER-IP    EXTERNAL-IP    PORT(S)                      AGE
traefik   LoadBalancer   10.43.105.6   192.168.64.5   80:30303/TCP,443:30142/TCP   4h21m
</code></pre>

<pre><code>$ kubectl get all
NAME          READY   STATUS    RESTARTS   AGE
pod/vault-0   1/1     Running   0          4h31m

NAME                     TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)             AGE
service/vault-internal   ClusterIP   None          &lt;none&gt;        8200/TCP,8201/TCP   4h31m
service/vault            ClusterIP   10.43.8.235   &lt;none&gt;        8200/TCP,8201/TCP   4h31m

NAME                     READY   AGE
statefulset.apps/vault   1/1     4h31m
</code></pre>

<p>I deployed vault via helm chart standalone vault (non dev mode) and tls enabled (values.yaml see below)</p>

<p>vault's cert is signed by k3s itself: <code>kubectl -n "${NAMESPACE}" certificate approve "${CSR_NAME}"</code></p>

<pre><code>certinfo tmp/localK3s/certs/vault/vault.crt
        Version: 3 (0x2)
        Serial Number:
            ed:8f:07:da:0d:3d:8d:55:3d:73:aa:93:9d:98:d2:69
    Signature Algorithm: ecdsa-with-SHA256
        Issuer: CN=k3s-server-ca@1591718124
        Validity
            Not Before: Jun  9 15:53:56 2020 GMT
            Not After : Jun  9 15:53:56 2021 GMT
        Subject: CN=vault.vault.svc
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Alternative Name:
                DNS:vault, DNS:vault.vault, DNS:vault.vault.svc, DNS:vault.vault.svc.iac.local, DNS:localhost, IP Address:127.0.0.1
</code></pre>

<p>now <strong>I can</strong> access the vault service directly by e.g.:</p>

<pre><code>$ kubectl -n vault port-forward service/vault 8200:8200 &amp;
$
$ export VAULT_ADDR=https://127.0.0.1:8200
$ export VAULT_CAPATH=$(pwd)/tmp/localK3s/certs/localK3s_root.ca
$ export VAULT_CLIENT_CERT=$(pwd)/tmp/localK3s/certs/vault/vault.crt
$ export VAULT_CLIENT_KEY=$(pwd)/tmp/localK3s/certs/vault/vault.key
$ 
$ vault status
Handling connection for 8200
Handling connection for 8200
Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    1
Threshold       1
Version         1.4.2
Cluster Name    vault-cluster-5bc9e954
Cluster ID      ca5496a6-525d-2b86-22dd-f771da82d5e0
HA Enabled      false
</code></pre>

<p>now what I want is <strong>having an ingress that is tls enabled</strong> AND forwarding to vault port 8200 via tls/https</p>

<p>so I have </p>

<pre class="lang-yaml prettyprint-override"><code>$ kubectl get ingress vault -o yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: vault
  namespace: vault
  annotations:
    meta.helm.sh/release-name: vault
    meta.helm.sh/release-namespace: vault
  labels:
    helm.sh/chart: vault-0.6.0
spec:
  rules:
  - host: vault.iac.local
    http:
      paths:
      - backend:
          serviceName: vault
          servicePort: 8200
        path: /
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - vault.iac.local
    secretName: vault-tls
status:
  loadBalancer: {}
</code></pre>

<pre><code>$ export VAULT_ADDR=https://vault.iac.local
$ export VAULT_CAPATH=$(pwd)/tmp/localK3s/certs/localK3s_root.ca
$ export VAULT_CLIENT_CERT=$(pwd)/tmp/localK3s/certs/vault/vault.crt
$ export VAULT_CLIENT_KEY=$(pwd)/tmp/localK3s/certs/vault/vault.key
$
$ vault status
vault status -tls-skip-verify
Error checking seal status: Error making API request.

URL: GET https://vault.iac.local/v1/sys/seal-status
Code: 404. Raw Message:

404 page not found
</code></pre>

<p>helm vault values.yaml</p>

<pre class="lang-yaml prettyprint-override"><code>global:
  enabled: true
  tlsDisable: false
injector:
  enabled: false
server:
  extraEnvironmentVars:
    VAULT_CACERT: /vault/userconfig/vault-tls/vault.ca
  extraVolumes:
  - type: secret
    name: vault-tls
  standalone:
    enabled: true
    config: |
      ui = true

      listener "tcp" {
        tls_disable = "false" # 1
        # address = "0.0.0.0:8200"
        address = "[::]:8200"
        cluster_address = "[::]:8201"
        tls_cert_file =      "/vault/userconfig/vault-tls/vault.crt"
        tls_key_file  =      "/vault/userconfig/vault-tls/vault.key"
        tls_client_ca_file = "/vault/userconfig/vault-tls/vault.ca"
      }
      storage "file" {
        path = "/vault/data"
      }
  ingress:
    enabled: true
    hosts:
    - host: vault.iac.local
    tls:
    - secretName: vault-tls
      hosts:
      - vault.iac.local
</code></pre>

<p>any ideas anybody?</p>

Dirk Hoffmann
<p>ok, it's always helpful to read the logs (<em>sigh</em>), e.g. of the ingress controller itself, e.g.:</p>

<pre><code>INGCTRL=traefik &amp;&amp; \
kubectl -n kube-system logs \
    pod/$(kubectl -n kube-system get pods -l app=$INGCTRL | sed -n -E "s/^($INGCTRL-[a-z0-9-]+).*$/\1/p")
</code></pre>

<p>if you use generic secrets for ingress tls, beware the secret keys have to be <code>tls.crt</code> and <code>tls.key</code> (or use <code>kubectl create secret tls</code> and not generic in the first place)</p>

<p>also check that your target service has an endpoint at all and not</p>

<pre><code>k describe svc theService
...
Endpoints:         &lt;none&gt;
...
</code></pre>
