<p>I've recently been making use of the <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity" rel="noreferrer">GKE Workload Identity</a> feature. I'd be interested to know in more detail how the <code>gke-metadata-server</code> component works.</p>

<ol>
<li>GCP client code (<code>gcloud</code> or other language SDKs) falls through to the GCE metadata method</li>
<li>Request made to <code>http://metadata.google.internal/path</code></li>
<li>(guess) Setting <code>GKE_METADATA_SERVER</code> on my node pool configures this to resolve to the <code>gke-metadata-server</code> pod on that node.</li>
<li>(guess) the <code>gke-metadata-server</code> pod with --privileged and host networking has a means of determining the source (pod IP?) then looking up the pod and its service account to check for the <code>iam.gke.io/gcp-service-account</code> annotation.</li>
<li>(guess) the proxy calls the metadata server with the pods 'pseudo' identity set (e.g. <code>[PROJECT_ID].svc.id.goog[[K8S_NAMESPACE]/[KSA_NAME]]</code>) to get a token for the service account annotated on its Kubernetes service account.</li>
<li>If this account has token creator / workload ID user rights to the service account presumably the response from GCP is a success and contains a token, which is then packaged and set back to the calling pod for authenticated calls to other Google APIs.</li>
</ol>

<p>I guess the main puzzle for me right now is the verification of the calling pods identity. Originally I thought this would use the TokenReview API but now I'm not sure how the Google client tools would know to use the service account token mounted into the pod...</p>

<p><em>Edit</em> follow-up questions:</p>

<p>Q1: In between step 2 and 3, is the request to <code>metadata.google.internal</code> routed to the GKE metadata proxy by the setting <code>GKE_METADATA_SERVER</code> on the node pool? </p>

<p>Q2: Why does the metadata server pod need host networking?</p>

<p>Q3: In the video here: <a href="https://youtu.be/s4NYEJDFc0M?t=2243" rel="noreferrer">https://youtu.be/s4NYEJDFc0M?t=2243</a> it's taken as a given that the pod makes a GCP call. How does the GKE metadata server identify the pod making the call to start the process?</p>

Charlie Egan
<p>Before going into details, please familiarize yourself with these components:</p>

<p><em>OIDC provider</em>: Runs on Google’s infrastructure, provides cluster specific metadata and signs authorized JWTs.</p>

<p><em>GKE metadata server</em>: It runs as a DaemonSet meaning one instance on every node, exposes pod specific metadata server (it will provide backwards compatibility with old client libraries), emulates existing node metadata server.</p>

<p><em>Google IAM</em>: issues access token, validates bindings, validates OIDC signatures. </p>

<p><em>Google cloud</em>: accepts access tokens, does pretty much anything.</p>

<p><em>JWT</em>: JSON Web token</p>

<p><em>mTLS</em>: Mutual Transport Layer Security</p>

<p>The steps below explain how GKE metadata server components work:</p>

<p><strong>Step 1</strong>:  An authorized user binds the cluster to the namespace.</p>

<p><strong>Step 2</strong>: Workload tries to access Google Cloud service using client libraries.</p>

<p><strong>Step 3</strong>: GKE metadata server is going to request an OIDC signed JWT from the control plane. That connection is authenticated using mutual TLS (mTLS) connection with node credential. </p>

<p><strong>Step 4</strong>: Then the GKE metadata server is going use that OIDC signed JWT to request an access token for the <em>[identity namespace]/[Kubernetes service account]</em> from IAM. IAM is going to validate that the appropriate bindings exist on identity namespace and in the OIDC provider.</p>

<p><strong>Step 5</strong>: And then IAM validates that it was signed by the cluster’s correct OIDC provider. It will then return an access token for the <em>[identity namespace]/[kubernetes service account].</em></p>

<p><strong>Step 6</strong>: Then the metadata server sends the access token it just got back to IAM. IAM will then exchange that for a short lived GCP service account token after validating the appropriate bindings.</p>

<p><strong>Step 7</strong>: Then GKE metadata server returns the GCP service account token to the workload.</p>

<p><strong>Step 8</strong>: The workload can then use that token to make calls to any Google Cloud Service.</p>

<p>I also found a <a href="https://youtu.be/s4NYEJDFc0M?t=1243" rel="noreferrer">video</a> regarding Workload Identity which you will find useful.</p>

<p><strong><em>EDIT</em> Follow-up questions' answers:</strong></p>

<p>Below are answers to your follow-up questions: </p>

<p><strong>Q1</strong>: In between step 2 and 3, is the request to metadata.google.internal routed to the gke metadata proxy by the setting GKE_METADATA_SERVER on the node pool?</p>

<p>You are right, GKE_METADATA_SERVER is set on the node pool. This exposes a metadata API to the workloads that is compatible with the V1 Compute Metadata APIs. Once workload tries to access Google Cloud service, the GKE metadata server performs a lookup (the metadata server checks to see if a pod exists in the list whose IP matches the incoming IP of the request) before it goes on to request the OIDC token from the control plane.</p>

<p>Keep in mind that <a href="https://cloud.google.com/kubernetes-engine/docs/reference/rest/v1beta1/NodeConfig#nodemetadata" rel="noreferrer">GKE_METADATA_SERVER</a> enumeration feature can only be enabled if Workload Identity is enabled at the cluster level.</p>

<p><strong>Q2</strong>: Why does the metadata server pod need host networking?</p>

<p>The gke-metadata-server intercepts all GCE metadata server requests from pods, however pods using the host network are not intercepted.</p>

<p><strong>Q3</strong>: How does the GKE metadata server identify the pod making the call to start the process?</p>

<p>The pods are identified using iptables rules.</p>
