<p>I am running a HashiCorp Vault stateless set in K8s with 3 pods on three nodes.</p>

<p>After deployment I manually unseal Vault. Vault then stays unsealed all the time.</p>

<p>The problem is when one of the nodes is restarted, the Vault pod restarts in unsealed mode.
<em>Is there any way to automatically unseal the Vault node by server-server-communication with one of the already unsealed pods?</em></p>

<p>I do not want to have to unseal the Vault pods manually when my Kubernetes environment gets updated and all nodes get restarted (rolling update - one by one).</p>

<p>I also do not want to store the unseal keys in a K8s secret or even a file as this makes encryption of my secrets useless.</p>

<p><em>This is my yaml:</em></p>

<pre><code>apiVersion: v1
kind: Service
metadata:
  name: vault
spec:
  clusterIP: None
  ports:
    - name: http
      port: 8200
    - name: server
      port: 8201
  selector:
    xyz.service: vault
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: vault
  labels:
    xyz.service: vault
spec:
  serviceName: "vault"
  selector:
    matchLabels:
      xyz.service: vault
  replicas: 3
  template:
    metadata:
      labels:
        xyz.service: vault
    spec:
      imagePullSecrets:
      - name: reg-dhc-xyzoms-pull-secret
      securityContext:
        runAsUser: 100
        fsGroup: 1000
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: xyz.service
                    operator: In
                    values:
                      - vault
              topologyKey: kubernetes.io/hostname
      containers:
        - name: vault
          image: vault:0.11.0
          resources:
            requests:
              memory: "100Mi"
          env:
          - name: SKIP_SETCAP
            value: dontcare
          - name: POD_IP
            valueFrom:
              fieldRef:
                fieldPath: "status.podIP"
          - name: "VAULT_CLUSTER_ADDR"
            value: "https://$(POD_IP):8201"
          ports:
            - name: http
              containerPort: 8200
              protocol: "TCP"
            - name: server
              containerPort: 8201
              protocol: "TCP"
</code></pre>

<p>After some more digging I learned, what I want is impossible.
Whenever a Vault instance is restarted it will be unsealed first and there is no way to unseal it automatically using Vault-own techniques.</p>

<p>You can find a lot of <em>"vault-unsealer"</em> implementations in GitHub and <a href="https://hub.docker.com/search/?q=unsealer" rel="nofollow noreferrer">Docker store</a> which try to fill this gap by regularly checking the Vault pods state and unsealing it if necessary.</p>

<p>It is suggested to use an K8s readinessprobe to avoid that services access a sealed Vault pod.</p>

<p>As there is no official "vault-unsealer" image, the public implementations must be used with caution. I ended up writing my own "vault-unsealer" to avoid security flaws and licensing problems.</p>

<p>My solution is a sidecar-container with each Vault pod. The unseal keys first have to be entered once manually with <code>kubectl exec ...</code> at one sidecar.
The sidecars regularly check all Vault pods and communicate the unseal keys to the other sidecar if sealed. If a sidecar receives unseal keys, they are stored in memory and are used to unseal its own Vault instance.</p>

<ol>
<li><code>kubect aply -f vault.yaml</code> -> vault-0 starting</li>
<li><code>kubectl exec vault-0 -c sidecar ...</code> to enter unseal keys -> vault-0 sidecar unseals vault-0 and is ready</li>
<li>vault-1 starting</li>
<li>vault-0 sidecar detects vault-1 unsealed and calls the vault-1 sidecar to transmit the unseal keys. -> vault-1 sidecar unseals vault-0 and is ready</li>
<li>and so on...</li>
</ol>
