<p>I am using serviceAccount Token Volume projection as described <a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#service-account-token-volume-projection" rel="nofollow noreferrer">here</a>. This is the manifest file I'm using:</p>

<pre><code>kind: Pod
apiVersion: v1
metadata:
  name: pod-acr
spec:
  containers:
  - image: yorchaksacr.azurecr.io/busybox:1.0
    name: busybox
    command: ["/bin/sh","-c","sleep 36000"]
    volumeMounts:
    - mountPath: /var/run/secrets/tokens
      name: vault-token
  serviceAccountName: pull-acr-images
  volumes:
  - name: vault-token
    projected:
      sources:
      - serviceAccountToken:
          path: vault-token
          expirationSeconds: 7200
          audience: vault
</code></pre>

<p>As expected, the token is mounted to the container under <code>/var/run/secrets/tokens/vault-token</code>:</p>

<pre><code>/ # ls -la /var/run/secrets/tokens
total 4
drwxrwxrwt    3 root     root           100 Jul 24 21:35 .
drwxr-xr-x    4 root     root          4096 Jul 24 21:35 ..
drwxr-xr-x    2 root     root            60 Jul 24 21:35 ..2019_07_24_21_35_15.018111081
lrwxrwxrwx    1 root     root            31 Jul 24 21:35 ..data -&gt; ..2019_07_24_21_35_15.018111081
lrwxrwxrwx    1 root     root            18 Jul 24 21:35 vault-token -&gt; ..data/vault-token

</code></pre>

<p>Problem is if I try to authenticate to the API server using this token the API rejects the call with <code>401 Unauthorized</code>:</p>

<pre><code>/ # wget --header="Authorization: Bearer $(cat /var/run/secrets/tokens/vault-token)" --no-check-certificate https://10.2.1.19:6443
Connecting to 10.2.1.19:6443 (10.2.1.19:6443)
wget: server returned error: HTTP/1.1 401 Unauthorized

</code></pre>

<p>However if I use the default path and token where service account tokens are projected for all pods <code>/var/run/secrets/kubernetes.io/serviceacconts/token</code> that works:</p>

<pre><code>/ # wget --header="Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" --no-check-certificate https://10.2.1.19:6443
Connecting to 10.2.1.19:6443 (10.2.1.19:6443)
saving to 'index.html'
index.html           100% |************************************************************************************************************************************************************|  2738  0:00:00 ETA
'index.html' saved

</code></pre>

<p>If I <code>cat</code> both tokens I can see they are actually different:</p>

<pre><code># cat /var/run/secrets/tokens/vault-token
eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJhdWQiOlsidmF1bHQiXSwiZXhwIjoxNTY0MDEzMjcwLCJpYXQiOjE1NjQwMDYwNzAsImlzcyI6Imh0dHBzOi8vMTAuMi4xLjE5Iiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJkZWZhdWx0IiwicG9kIjp7Im5hbWUiOiJwb2QtYWNyIiwidWlkIjoiNThiNjI5YWEtZGU4Ni00YTAzLWI3YmQtMTI4ZGFiZWVkYmQ5In0sInNlcnZpY2VhY2NvdW50Ijp7Im5hbWUiOiJwdWxsLWFjci1pbWFnZXMiLCJ1aWQiOiJlZGE0NDlmYS1iODE2LTQ0ZGMtYTFjNi0yMWJhZWUwZmVkN2YifX0sIm5iZiI6MTU2NDAwNjA3MCwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmRlZmF1bHQ6cHVsbC1hY3ItaW1hZ2VzIn0.UjPaTgWHPwCeeh1ltBb64hv0yEjKkRxKw_BZ3PLDA3HsJK-keXN40Khp-8mNnLQ-uYIfMgW4FXwYIm0SVeQUhM4sh4rwjAYDEfEHDah9AvhEL8I65T_jhnhT10E1M7mzk1x0RFGvjZAECd1RlYM7IuXIkEfZCI_6GRVAbX3Vmk6XF0sRh2T8DZzw8kj_Z54J2gYCt2beBnn7hC9rOC9LW9J0AFEAAQQE_UJME5y4jZD6hfJMSGOouyQm70nVGytqKVsLbzyorH5pugEqrs1Z_dLx6E3Ta9kELRPvyDZgeNiS44fEYlRApn6fZawsppc1oRNoeyMqiIPRdgQekBVfTA/ #


# cat /var/run/secrets/kubernetes.io/serviceaccount/token
eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InB1bGwtYWNyLWltYWdlcy10b2tlbi1oYjU0NyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJwdWxsLWFjci1pbWFnZXMiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJlZGE0NDlmYS1iODE2LTQ0ZGMtYTFjNi0yMWJhZWUwZmVkN2YiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6ZGVmYXVsdDpwdWxsLWFjci1pbWFnZXMifQ.nqqhZVmBUuKVi6E3L9MEn8oW1dKd-DV4c9jcVy5mXAuEMZ1WgLlaaHFF1ibnVMjEK6VUJyJhp7w08hgSmyyh-KY4BQ5oJf1jmSySvmttJxjXW-KsMpf5rHF0ZDmgaqZwbi7FvowtoTECstFBVNoszKJUn1iV5mU_6MQkEtGTNyE4KuZ9LEvPuZxiNZ5UyW3UaHXLqF63-w_xlkfa_75E-cgXqvSSGTCb6RsTuOmVyCqganx5SpIb5EU-3Mu7hUWEhSRAh3tpcPIwjS7-NkuO0ReH7Z40rPHqkIokshUUO75WM_oPq7tlu6PSCTwOK-Jw66kzi-jqKNyKvMeWJUq4WQ/ #
</code></pre>

<p>Does anybody have an idea why am I seeing this behavior? I would expect both tokens to work, but apparently looks like it is not the case.</p>

<p>Configuration of the API Server:</p>

<pre><code>spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=10.2.1.19
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction
    - --enable-bootstrap-token-auth=true
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
    - --etcd-servers=https://127.0.0.1:2379
    - --insecure-port=0
    - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
    - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
    - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
    - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
    - --requestheader-allowed-names=front-proxy-client
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --requestheader-extra-headers-prefix=X-Remote-Extra-
    - --requestheader-group-headers=X-Remote-Group
    - --requestheader-username-headers=X-Remote-User
    - --secure-port=6443
    - --service-account-key-file=/etc/kubernetes/pki/sa.pub
    - --service-cluster-ip-range=10.96.0.0/12
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    - --basic-auth-file=/etc/kubernetes/pki/passwordfile
    - --service-account-issuer=https://10.2.1.19
    - --service-account-signing-key-file=/etc/kubernetes/pki/sa.key
    image: k8s.gcr.io/kube-apiserver:v1.15.1
    imagePullPolicy: IfNotPresent
</code></pre>

<p>Confusingly, 401 Unauthorized indicates an authentication problem rather than an authorization problem (see <a href="https://stackoverflow.com/questions/3297048/403-forbidden-vs-401-unauthorized-http-responses">here</a>). This means the Kubernetes service account token authenticator doesn't like the token at <code>/var/run/secrets/tokens/vault-token</code> (let's call it the <code>vault-token</code>).  The two tokens differ in a couple ways. Here's the decoded <code>vault-token</code>:</p>

<pre><code>{
  "aud": [
    "vault"
  ],
  "exp": 1564013270,
  "iat": 1564006070,
  "iss": "https://10.2.1.19",
  "kubernetes.io": {
    "namespace": "default",
    "pod": {
      "name": "pod-acr",
      "uid": "58b629aa-de86-4a03-b7bd-128dabeedbd9"
    },
    "serviceaccount": {
      "name": "pull-acr-images",
      "uid": "eda449fa-b816-44dc-a1c6-21baee0fed7f"
    }
  },
  "nbf": 1564006070,
  "sub": "system:serviceaccount:default:pull-acr-images"
}
</code></pre>

<p>Notice the audiences (<code>["vault"]</code>), issuer (<code>"https://10.2.1.19"</code>), and subject (<code>"system:serviceaccount:default:pull-acr-images"</code>).</p>

<p>Here's the <code>default-path-token</code>:</p>

<pre><code>{
  "iss": "kubernetes/serviceaccount",
  "kubernetes.io/serviceaccount/namespace": "default",
  "kubernetes.io/serviceaccount/secret.name": "pull-acr-images-token-hb547",
  "kubernetes.io/serviceaccount/service-account.name": "pull-acr-images",
  "kubernetes.io/serviceaccount/service-account.uid": "eda449fa-b816-44dc-a1c6-21baee0fed7f",
  "sub": "system:serviceaccount:default:pull-acr-images"
}
</code></pre>

<p>Same subject, but different issuer (<code>"kubernetes/serviceaccount"</code>) and no audiences.</p>

<p>I'm not sure why the <code>default-path-token</code> has a different issuer, or why it's authenticating correctly, but your <code>vault-token</code> is not authenticating correctly because the audiences don't match the issuer.</p>

<p>More specifically, the clue is in the doc you linked <a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#service-account-token-volume-projection" rel="nofollow noreferrer">here</a>. It says that this feature working correctly depends on how you set the following flags for the <code>kube-apiserver</code>:</p>

<ul>
<li><code>--service-account-issuer</code></li>
<li><code>--service-account-signing-key-file</code></li>
<li><code>--service-account-api-audiences</code></li>
</ul>

<p>Not sure if those docs are wrong or just out of date, but since you're using <code>v1.15.1</code> the flags are now called:</p>

<ul>
<li><code>--service-account-issuer</code></li>
<li><code>--service-account-signing-key-file string</code></li>
<li><code>--api-audiences</code></li>
</ul>

<p>The <a href="https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/" rel="nofollow noreferrer">flag documentation</a> says of the <code>--api-audiences</code> flag:</p>

<blockquote>
  <p>Identifiers of the API. The service account token authenticator will validate that tokens used against the API are bound to at least one of these audiences. If the --service-account-issuer flag is configured and this flag is not, this field defaults to a single element list containing the issuer URL.</p>
</blockquote>

<p>Since you <em>don't</em> have this flag set and you have <code>--service-account-issuer=https://10.2.1.19</code>, and you have <code>audience: vault</code> in your pod spec, your token is going to claim its bound to the <code>vault</code> audience, and the token authenticator is going to try to match this against the value of the <code>--service-account-issuer</code> flag, and clearly those don't match.</p>

<p>You can make these match by specifying <code>audience: https://10.2.1.19</code> in your pod spec instead of <code>audience: vault</code>. One caveat: this solution may technically work to ensure the token authenticates, but I'm not sure what the truly <em>right</em> answer is insofar as using this flags and fields in the pod spec as they're truly intended, and just making these strings match could be a bit hacky.</p>
