<p>GKE version - 1.14
Currently I have two private gke cluster ( Vault cluster and app cluster)</p>
<p>Getting following errors:</p>
<pre><code>vault errors -
auth.kubernetes.auth_kubernetes_b0f01fa6: login unauthorized due to: Post &quot;https://10.V.V.194:443/apis/authentication.k8s.io/v1/tokenreviews&quot;: dial tcp `10.V.V.194`:443: i/o timeout
</code></pre>
<p>-&gt; where</p>
<pre><code>10.V.V.194 -- is master IP address (no https://) via `kubectl cluster-info
</code></pre>
<p>Application pod logs</p>
<pre><code>    * permission denied&quot; backoff=1.324573453
2020-10-12T14:39:46.421Z [INFO]  auth.handler: authenticating
2020-10-12T14:40:16.427Z [ERROR] auth.handler: error authenticating: error=&quot;Error making API request.

URL: PUT http://10.LB.LB.38:8200/v1/auth/kubernetes/login
Code: 403. Errors:

* permission denied&quot; backoff=2.798763368
</code></pre>
<p>-&gt; Where</p>
<pre><code>http://10.LB.LB.38:8200 is Internal LB IP 
</code></pre>
<p>Vault setup</p>
<pre><code>    NAME   TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                         
test-vault  LoadBalancer   240.130.0.59   10.LB.LB.38   8200:32105/TCP,8201:31147/TCP  
</code></pre>
<p>How is K8s auth methods enable</p>
<pre><code>    $ export VAULT_SA_NAME=$(kubectl get sa vault-auth -o jsonpath=&quot;{.secrets[*]['name']}&quot;)
$ export SA_JWT_TOKEN=$(kubectl get secret $VAULT_SA_NAME -o jsonpath=&quot;{.data.token}&quot; | base64 --decode; echo)
$ export SA_CA_CRT=$(kubectl get secret $VAULT_SA_NAME -o jsonpath=&quot;{.data['ca\.crt']}&quot; | base64 --decode; echo) 

# determine Kubernetes master IP address (no https://) via `kubectl cluster-info`
$ export K8S_HOST=&lt;K8S_MASTER_IP&gt;   ----- App cluster ip 

# set VAULT_TOKEN &amp; VAULT_ADDR before next steps
$ vault auth enable kubernetes
$ vault write auth/kubernetes/config \
        token_reviewer_jwt=&quot;$SA_JWT_TOKEN&quot; \
        kubernetes_host=&quot;https://$K8S_HOST:443&quot; \
        kubernetes_ca_cert=&quot;$SA_CA_CRT&quot;
</code></pre>
<p>How is vault inject setup in application cluster</p>
<pre><code>name: AGENT_INJECT_VAULT_ADDR
      value: http://10.LB.LB.38:8200
</code></pre>
<p>Cluster B ( app cluster )</p>
<pre><code>kubectl create serviceaccount vault-auth -n default
-----
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: role-tokenreview-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: vault-auth
  namespace: default

vault auth enable kubernetes
-----------
vault write auth/kubernetes/config kubernetes_host=&quot;${K8S_HOST}&quot;   
    kubernetes_ca_cert=&quot;${VAULT_SA_CA_CRT}&quot; 
    token_reviewer_jwt=&quot;${TR_ACCOUNT_TOKEN}&quot;
-----------
vault secrets enable -path=secret/ kv
-----------
vault policy write myapp-kv-rw - &lt;&lt;EOF
 path &quot;secret/myapp/*&quot; {
 capabilities = [&quot;create&quot;, &quot;read&quot;, &quot;update&quot;, &quot;delete&quot;, &quot;list&quot;]
}
--------------
vault write auth/kubernetes/role/myapp-role \
 bound_service_account_names=default \
 bound_service_account_namespaces=default \
 policies=default,myapp-kv-rw \
 ttl=15m
</code></pre>
<p>Can you Please let me know, If I miss anything ?</p>

<p>Curious as to which version of k8s you're using. I was having the same issue using <code>v1.21.1</code>. I had to add an issuer to the per the docs (<a href="https://www.vaultproject.io/docs/auth/kubernetes" rel="nofollow noreferrer">https://www.vaultproject.io/docs/auth/kubernetes</a>)</p>
<blockquote>
<p>Kubernetes 1.21+ clusters may require setting the service account issuer to the same value as kube-apiserver's --service-account-issuer flag. This is because the service account JWTs for these clusters may have an issuer specific to the cluster itself, instead of the old default of kubernetes/serviceaccount</p>
</blockquote>
<p>like so</p>
<pre><code>vault write auth/kubernetes/config \
  token_reviewer_jwt=&quot;$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)&quot; \
  kubernetes_host=&quot;https://$KUBERNETES_PORT_443_TCP_ADDR:443&quot; \
  kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
  issuer=&quot;\&quot;test-aks-cluster-dns-d6cbb78e.hcp.uksouth.azmk8s.io\&quot;&quot;
</code></pre>
<p>and the issuer can be obtained by running <code>kubectl proxy &amp; curl --silent http://127.0.0.1:8001/.well-known/openid-configuration | jq -r .issuer </code></p>
