<p>I am trying to explore vault enterprise but getting permission denied for sidecar when I use the vault enterprise but seems to work fine when I tried to use local vault server.</p>
<p>Here is the repository that contains a working example with the local vault <a href="https://github.com/Adiii717/vault-sidecar-injector-app" rel="nofollow noreferrer">vault-sidecar-injector-app</a></p>
<p><a href="https://i.stack.imgur.com/oBZxm.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/oBZxm.png" alt="enter image description here" /></a></p>
<p><strong>Vault config</strong></p>
<pre><code>export VAULT_ADDR=&quot;https://vault-cluster.vault.c1c633fa-91ef-4e86-b025-4f31b3f14730.aws.hashicorp.cloud:8200&quot;
export VAULT_NAMESPACE=&quot;admin&quot;

#install agent
helm upgrade --install vault hashicorp/vault --set &quot;injector.externalVaultAddr=$VAULT_ADDR&quot;
vault auth enable kubernetes
# get certs &amp; host
VAULT_HELM_SECRET_NAME=$(kubectl get secrets --output=json | jq -r '.items[].metadata | select(.name|startswith(&quot;vault-token-&quot;)).name')
TOKEN_REVIEW_JWT=$(kubectl get secret $VAULT_HELM_SECRET_NAME --output='go-template={{ .data.token }}' | base64 --decode)
KUBE_CA_CERT=$(kubectl config view --raw --minify --flatten --output='jsonpath={.clusters[].cluster.certificate-authority-data}' | base64 --decode)
KUBE_HOST=$(kubectl config view --raw --minify --flatten --output='jsonpath={.clusters[].cluster.server}')

# set Kubernetes config
vault write auth/kubernetes/config \
        token_reviewer_jwt=&quot;$TOKEN_REVIEW_JWT&quot; \
        kubernetes_host=&quot;$KUBE_HOST&quot; \
        kubernetes_ca_cert=&quot;$KUBE_CA_CERT&quot; \
        issuer=&quot;https://kubernetes.default.svc.cluster.local&quot; \
        disable_iss_validation=&quot;true&quot; \
        disable_local_ca_jwt=&quot;true&quot;


vault auth enable approle
# create admin policy
vault policy write admin admin-policy.hcl
vault write auth/approle/role/admin policies=&quot;admin&quot;
vault read auth/approle/role/admin/role-id
# generate secret
vault write -f  auth/approle/role/admin/secret-id
#Enable KV
vault secrets enable -version=2 kv
</code></pre>
<p>I can see the role and policy
<a href="https://i.stack.imgur.com/DGOnr.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/DGOnr.png" alt="enter image description here" /></a></p>
<p><strong>Admin policy</strong></p>
<p>Here is the admin policy for the enterprise</p>
<pre><code>path &quot;*&quot; {
capabilities = [&quot;create&quot;, &quot;read&quot;, &quot;update&quot;, &quot;delete&quot;, &quot;list&quot;, &quot;sudo&quot;]
}
</code></pre>
<p><strong>Deploy Script for helm</strong></p>
<p>here is the deploy script, tried <code>hcp-root</code> root policy but no luck</p>
<pre class="lang-sh prettyprint-override"><code>RELEASE_NAME=demo-managed
NAMESPACE=default
ENVIRONMENT=develop
export role_id=&quot;f9782a53-823e-2c08-81ae-abc&quot;
export secret_id=&quot;1de3b8c5-18c7-60e3-24ca-abc&quot;
export VAULT_ADDR=&quot;https://vault-cluster.vault.c1c633fa-91ef-4e86-b025-4f31b3f14730.aws.hashicorp.cloud:8200&quot;
export VAULT_TOKEN=$(vault write -field=&quot;token&quot;  auth/approle/login role_id=&quot;${role_id}&quot; secret_id=&quot;${secret_id}&quot;)
vault write auth/kubernetes/role/${NAMESPACE}-${RELEASE_NAME} bound_service_account_names=${RELEASE_NAME} bound_service_account_namespaces=${NAMESPACE} policies=hcp-root ttl=1h
helm upgrade --install $RELEASE_NAME ../helm-chart --set environment=$ENVIRONMENT --set nameOverride=$RELEASE_NAME

</code></pre>
<p>also tried with root token</p>
<pre class="lang-sh prettyprint-override"><code>RELEASE_NAME=demo-managed
NAMESPACE=default
ENVIRONMENT=develop
vault write auth/kubernetes/role/${NAMESPACE}-${RELEASE_NAME} bound_service_account_names=${RELEASE_NAME} bound_service_account_namespaces=${NAMESPACE} policies=hcp-root ttl=1h
helm upgrade --install $RELEASE_NAME ../helm-chart --set environment=$ENVIRONMENT --set nameOverride=$RELEASE_NAME

</code></pre>
<p><strong>Sidecar config</strong></p>
<p>With namespace annotation, as my understanding namespace is required</p>
<blockquote>
<p>vault.hashicorp.com/namespace - configures the Vault Enterprise namespace to be used when requesting secrets from Vault.</p>
</blockquote>
<p><a href="https://www.vaultproject.io/docs/platform/k8s/injector/annotations" rel="nofollow noreferrer">https://www.vaultproject.io/docs/platform/k8s/injector/annotations</a></p>
<pre><code>vault.hashicorp.com/namespace : &quot;admin&quot;
</code></pre>
<p><strong>Error</strong></p>
<pre><code>  | Error making API request.
  |
  | URL: PUT https://vault-cluster.vault.c1c633fa-91ef-4e86-b025-4f31b3f14730.aws.hashicorp.cloud:8200/v1/admin/auth/kubernetes/login
  | Code: 403. Errors:
  |
  | * permission denied
</code></pre>
<p>Without namespace annotation getting below error</p>
<pre><code>  | URL: PUT https://vault-cluster.vault.c1c633fa-91ef-4e86-b025-4f31b3f14730.aws.hashicorp.cloud:8200/v1/auth/kubernetes/login
  | Code: 400. Errors:
  |
  | * missing client token
</code></pre>
<p>Even enabling debug logs <code>vault.hashicorp.com/log-level : &quot;debug&quot;</code> does not help me with this error,  any help or suggestions will be appreciated.</p>
<p>Also tried
<a href="https://support.hashicorp.com/hc/en-us/articles/4404389946387-Kubernetes-auth-method-Permission-Denied-error" rel="nofollow noreferrer">https://support.hashicorp.com/hc/en-us/articles/4404389946387-Kubernetes-auth-method-Permission-Denied-error</a></p>
<p>So seems like I am missing something very specific to the vault enterprise</p>

<p>Finally able to resolve the weird issue with vault, posting as an answer might help someone else.</p>
<p>The only thing that I missed to understand the flow between vault server, sidecar, and Kubernetes.</p>
<p><a href="https://i.stack.imgur.com/qHDWm.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/qHDWm.png" alt="enter image description here" /></a></p>
<p>Kubernetes should be reachable to vault enterprise for Token review API calls. As you can see the when sidecar makes a request to the vault, then the vault enterprise server performs a token review API call.</p>
<blockquote>
<p>Use the /config endpoint to configure <strong>Vault to talk to Kubernetes</strong>. Use kubectl cluster-info to validate the <strong>Kubernetes host address</strong> and TCP port.</p>
</blockquote>
<p><a href="https://www.vaultproject.io/docs/auth/kubernetes" rel="nofollow noreferrer">https://www.vaultproject.io/docs/auth/kubernetes</a></p>
<pre><code>
  | Error making API request.
  |
  | URL: PUT https://vault-cluster.vault.c1c633fa-91ef-4e86-b025-4f31b3f14730.aws.hashicorp.cloud:8200/v1/admin/auth/kubernetes/login
  | Code: 403. Errors:
  |
  | * permission denied
   backoff=2.99s
</code></pre>
<p>This error does not indicate it has a connectivity issue but this also happens when the vault is not able to communicate with the Kubernetes cluster.</p>
<h1>Kube Host</h1>
<pre><code>vault write auth/kubernetes/config \
        token_reviewer_jwt=&quot;$TOKEN_REVIEW_JWT&quot; \
        kubernetes_host=&quot;$KUBE_HOST&quot; \
        kubernetes_ca_cert=&quot;$KUBE_CA_CERT&quot; \
        issuer=&quot;https://kubernetes.default.svc.cluster.local&quot;
</code></pre>
<p><code>KUBE_HOST</code> should be reachable for vault enterprise for tokenreview process.</p>
<p>So for the vault to communicate with our cluster, we need a few changes.</p>
<pre><code>minikube start --apiserver-ips=14.55.145.30 --vm-driver=none
</code></pre>
<p>Now update the <code>vaul-config.sh</code> file</p>
<pre><code>KUBE_HOST=$(kubectl config view --raw --minify --flatten --output='jsonpath={.clusters[].cluster.server}')
</code></pre>
<p>change this to</p>
<pre><code>KUBE_HOST=&quot;&quot;https://14.55.145.30:8443/&quot;
</code></pre>
<p>No manual steps, for the first time configuration run</p>
<pre><code>./vault-config.sh
</code></pre>
<p>and for the rest of the deployment in your CI/CD you can use</p>
<pre><code>./vault.sh
</code></pre>
<p>Each release has only been able to access its own secrets.</p>
<p>Furter details can be found <a href="https://github.com/Adiii717/vault-sidecar-injector-app/blob/main/README.md#start-minikube-in-ec2" rel="nofollow noreferrer">start-minikube-in-ec2</a></p>
<p>TLDR,</p>
<p><strong>Note: Kubernetes cluster should be reachable to vault enterprise for authentication, so vault enterprise would not able to communicate with your local minikube cluster. Better to test it out on EC2</strong></p>
