<p>I got two types of strange situations when deploying Vault in Kubernetes and using <code>Kubernetes Auth</code> method</p>
<blockquote>
<p>Kubernetes version: v1.25.6<br />
Vault version: v1.12.1</p>
</blockquote>
<h3><strong>1. It kept getting 403 <code>permission denied</code> from <code>/v1/auth/kubernetes/login</code> for about 30 minutes long time before suddenly got desired secrets successfully at <code>vault-agent-init</code> stage. Sometime it never got success after even several hours.</strong></h3>
<p><strong>Error:</strong></p>
<pre class="lang-bash prettyprint-override"><code>==&gt; Vault agent started! Log data will stream in below:

==&gt; Vault agent configuration:

                     Cgo: disabled
               Log Level: info
                 Version: Vault v1.12.1, built 2022-10-27T12:32:05Z
             Version Sha: e34f8a14fb7a88af4640b09f3ddbb5646b946d9c

2023-04-03T15:42:38.374Z [INFO]  sink.file: creating file sink
2023-04-03T15:42:38.374Z [INFO]  sink.file: file sink configured: path=/home/vault/.vault-token mode=-rw-r-----
2023-04-03T15:42:38.374Z [INFO]  template.server: starting template server
2023-04-03T15:42:38.375Z [INFO] (runner) creating new runner (dry: false, once: false)
2023-04-03T15:42:38.374Z [INFO]  sink.server: starting sink server
2023-04-03T15:42:38.375Z [INFO]  auth.handler: starting auth handler
2023-04-03T15:42:38.375Z [INFO]  auth.handler: authenticating
2023-04-03T15:42:38.375Z [INFO] (runner) creating watcher
2023-04-03T15:42:38.381Z [ERROR] auth.handler: error authenticating:
  error=
  | Error making API request.
  | 
  | URL: PUT http://vault.vault.svc:8200/v1/auth/kubernetes/login
  | Code: 403. Errors:
  | 
  | * permission denied
   backoff=1s
2023-04-03T15:42:39.381Z [INFO]  auth.handler: authenticating
2023-04-03T15:42:39.383Z [ERROR] auth.handler: error authenticating:
  error=
  | Error making API request.
  | 
  | URL: PUT http://vault.vault.svc:8200/v1/auth/kubernetes/login
  | Code: 403. Errors:
  | 
  | * permission denied
   backoff=1.62s
</code></pre>
<h3><strong>2. Sometime it got authenticated at <code>/v1/auth/kubernetes/login</code> soon, but then threw the error like:</strong></h3>
<pre class="lang-bash prettyprint-override"><code># (Can't get the output now, but something like this:)
vault.read(myapp/data/postgres/config), vault.read(myapp/data/postgres/config)

URL: GET http://vault.vault.svc:8200/v1/myapp/data/postgres/config
Code: 403. Errors:

* permission denied
</code></pre>
<hr />
<h2><strong>How I installed Vault in namespace vault:</strong></h2>
<p><strong>helm vaules.yaml:</strong></p>
<pre class="lang-yaml prettyprint-override"><code>ui:
  enabled: true

server:
  logLevel: trace

  ha:
    enabled: true
    replicas: 3
  
    raft:
      enabled: true
  
  dataStorage:
    storageClass: cstor-csi
  
  auditStorage:
    storageClass: cstor-csi
  
  authDelegator:
    enabled: true

injector:
  enabled: true
  logLevel: trace
</code></pre>
<pre class="lang-bash prettyprint-override"><code>helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update
# Install a spceified version vault in namespace `vault`.
helm upgrade --install vault hashicorp/vault --namespace vault -f vault-values.yaml --version 0.23.0 --create-namespace

# Unseal
kubectl exec -ti vault-0 -n vault -- vault operator init &gt; keys.txt
kubectl exec -ti vault-1 -n vault -- vault operator init &gt;&gt; keys.txt
kubectl exec -ti vault-2 -n vault -- vault operator init &gt;&gt; keys.txt
kubectl exec -ti vault-0 -n vault -- vault operator unseal
kubectl exec -ti vault-1 -n vault -- vault operator unseal
kubectl exec -ti vault-2 -n vault -- vault operator unseal

kubectl exec -it vault-0 -n vault -- /bin/sh

vault login

vault auth enable kubernetes

# Do this as document says:
vault write auth/kubernetes/config \
    kubernetes_host=&quot;https://$KUBERNETES_PORT_443_TCP_ADDR:443&quot;

vault secrets enable -path=myapp kv-v2

vault kv put myapp/postgres/config POSTGRES_DB=&quot;myapp&quot; POSTGRES_USER=&quot;myapp&quot; POSTGRES_PASSWORD=&quot;myapp&quot;

vault kv get myapp/postgres/config

vault policy write myapp - &lt;&lt;EOF
path &quot;myapp/data/postgres/config&quot; {
  capabilities = [&quot;read&quot;]
}
EOF

vault write auth/kubernetes/role/myapp \
    bound_service_account_names=myapp-sa \
    bound_service_account_namespaces=myapp \
    policies=myapp \
    ttl=3d

# create sa myapp-sa in namespace myapp, not the namespace with vault.
kubectl create sa myapp-sa -n myapp
</code></pre>
<hr />
<h2><strong>Deployemnt of myapp in namespace myapp</strong></h2>
<pre class="lang-yaml prettyprint-override"><code># I don't know if this ClusterRoleBinding needed?
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: myapp-sa-rbac
  namespace: myapp
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: myapp-sa
  namespace: myapp
---
apiVersion: v1
kind: Service
metadata:
  name: myapp
  namespace: myapp
spec:
  type: ClusterIP
  ports:
    - name: myapp
      port: 8080
      targetPort: 8080
  selector:
    app: myapp
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  namespace: myapp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
      annotations:
        vault.hashicorp.com/agent-inject: &quot;true&quot;
        vault.hashicorp.com/agent-inject-status: 'update'
        vault.hashicorp.com/role: &quot;myapp&quot;
        vault.hashicorp.com/agent-inject-secret-database-config: &quot;myapp/data/postgres/config&quot;
        # Environment variable export template
        vault.hashicorp.com/agent-inject-template-database-config: |
          {{ with secret &quot;myapp/data/postgres/config&quot; -}}
          export POSTGRES_DB=&quot;{{ .Data.data.POSTGRES_DB }}&quot;
          export POSTGRES_USER=&quot;{{ .Data.data.POSTGRES_USER }}&quot;
          export POSTGRES_PASSWORD=&quot;{{ .Data.data.POSTGRES_PASSWORD }}&quot;
          {{- end }}
    spec:
      serviceAccountName: myapp-sa
      containers:
        - name: myapp
          image: nginx:latest
          imagePullPolicy: Always
          ports:
            - containerPort: 8080
          command: [&quot;sh&quot;, &quot;-c&quot;]
          args:
            - . /vault/secrets/database-config
          env:
            - name: POSTGRES_HOST
              value: postgres
            - name: POSTGRES_PORT
              value: &quot;5432&quot;
</code></pre>

Suge
<p>Fixed:</p>
<p>We should do <code>raft join</code> rather than <code>vault operator init</code> for three times:</p>
<p><a href="https://developer.hashicorp.com/vault/tutorials/kubernetes/kubernetes-minikube-raft" rel="nofollow noreferrer">https://developer.hashicorp.com/vault/tutorials/kubernetes/kubernetes-minikube-raft</a></p>
<pre class="lang-bash prettyprint-override"><code># Join the vault-1 pod to the Raft cluster.
kubectl exec -ti vault-1 -- vault operator raft join http://vault-0.vault-internal:8200

# Join the vault-2 pod to the Raft cluster.
kubectl exec -ti vault-2 -- vault operator raft join http://vault-0.vault-internal:8200
</code></pre>
