<p>I have a kubernetes cluster with two node groups in AWS. One for Spot instances and the other for on demand instances. I have installed Vault and CSI driver to manage the secrets.</p>
<p>When I create this deployment everything works fine, the pods are created, run and the secrets are there.</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: vault-test
  name: vault-test
  namespace: development
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vault-test
  strategy: {}
  template:
    metadata:
      labels:
        app: vault-test
    spec:
      containers:
      - image: jweissig/app:0.0.1
        name: app
        envFrom:
        - secretRef:
            name: dev-secrets
        resources: {}
        volumeMounts:
        - name: secrets-store-inline
          mountPath: &quot;/mnt/secrets&quot;
          readOnly: true
      serviceAccountName: vault-sa
      volumes:
        - name: secrets-store-inline
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: dev
status: {}
</code></pre>
<p>But when I add nodeAffinity and tolerations to create the pods in the Spot machines the pods stay in a ContainerCreating status with the following error:</p>
<blockquote>
<p>Warning  FailedMount  10m (x716 over 24h)   kubelet  MountVolume.SetUp failed for volume &quot;secrets-store-inline&quot; : rpc error: code = Unknown desc = failed to mount secrets store objects for pod development/pod-name, err: error connecting to provider &quot;vault&quot;: provider not found: provider &quot;vault&quot;</p>
</blockquote>
<p>I created two applications to test the vault behavior, one with no tolerations just for testing and the real one, with the tolerations and nodeAffinity. And after a lot of tests I realized the problem was where the pods are being scheduled, but I don't understand why that behavior</p>

Mateo Arboleda
<p>The problem is the vault CSI driver configuration, the <code>DaemonSet</code> is not running in all nodes because of the missing <code>tolerations</code>. I had to add the <code>tolerations</code> to the <code>DaemonSet</code> manifest so there is a <code>Pod</code> in all nodes, and this way all nodes know what vault is.</p>
