<p>We have a system designed to manage a large number of Kubernetes clusters housed in external customer accounts simultaneously. This system currently works by having the <code>kubeconfig</code> stored in a database that is queried at runtime, and then passed into the golang kube-client constructor like so:</p>

<pre><code>clientcmd.NewClientConfigFromBytes([]byte(kubeConfigFromDB))
</code></pre>

<p>For clusters using basic auth, this "just works".</p>

<p>For EKS clusters, this works as long as both the <code>aws-iam-authenticator</code> is installed on the machine that is running the golang code such that the kube-client can call out to it for authentication, and correct <code>API_AWS_ACCESS_KEY_ID</code> and <code>API_AWS_SECRET_ACCESS_KEY</code> are set within the <code>kubeconfig</code>'s <code>user.exec.env</code> key.</p>

<p>For GKE clusters, it's not clear what the best practice way of achieving this is, and I have not been able to get it to work yet despite trying a handful of different operations detailed below. The standard practice for generating a <code>kubeconfig</code> for a GKE cluster is very similar to EKS (detailed here <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl?authuser=1#generate_kubeconfig_entry" rel="nofollow noreferrer">https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl?authuser=1#generate_kubeconfig_entry</a>) which uses <code>gcloud config config-helper</code> to generate the authentication credentials.</p>

<p>One idea is to use the <code>GOOGLE_APPLICATION_CREDENTIALS</code> environment variable, however the problem with this is that it is <em>global</em> and thus we cannot have our system simultaneously talk to many remote GKE clusters because each needs a unique set of google credentials to authenticate.</p>

<p>My second idea was to use the <code>--impersonate-service-account</code> flag provided to <code>gcloud config config-helper</code>, however it crashes when I run it with the following error:</p>

<pre><code>$ gcloud config config-helper --format=json --impersonate-service-account=acct-with-gke-access@myorg.iam.gserviceaccount.com --project myproject
WARNING: This command is using service account impersonation. All API calls will be executed as [acct-with-gke-access@myorg.iam.gserviceaccount.com].
ERROR: gcloud crashed (AttributeError): 'unicode' object has no attribute 'utcnow'
</code></pre>

<p>My final idea is quite complicated. I would get the google-credentials-JSON and put it in the <code>kubeconfig</code> like so:</p>

<pre><code>  user:
    auth-provider:
      config:
        credentials: "&lt;google-credentials-JSON&gt;"
      name: my-custom-forked-gcp
</code></pre>

<p>And I would create my own copy of <a href="https://github.com/kubernetes/client-go/blob/master/plugin/pkg/client/auth/gcp/gcp.go#L156" rel="nofollow noreferrer">https://github.com/kubernetes/client-go/blob/master/plugin/pkg/client/auth/gcp/gcp.go#L156</a> and replace line 156 </p>

<pre><code>ts, err := google.DefaultTokenSource(context.Background(), scopes...)
</code></pre>

<p>with </p>

<pre><code>ts, err := tokenSourceFromJSON(context.Background(),  gcpConfig["credentials"], scopes...)
</code></pre>

<p>Where <code>tokenSourceFromJSON</code> is a new method that I add that looks like this:</p>

<pre><code>func tokenSourceFromJSON(ctx context.Context, jsonData string, scopes ...string) (oauth2.TokenSource, error) {
  creds, err := google.CredentialsFromJSON(ctx, []byte(jsonData), scopes...)
  if err != nil {
    return nil, err
  }
  return creds.TokenSource, nil
}
</code></pre>

<p>This last idea will probably work (hopefully! I'm working on it now) but it seems like a very complicated solution to a simple problem: to provide the <code>google-credentials-JSON</code> at runtime to the golang kubernetes client to authenticate using those credentials. Is there an easier way?</p>

Ben Sussman
<blockquote>
  <p>One idea is to use the GOOGLE_APPLICATION_CREDENTIALS environment variable, however, the problem with this is that it is global and thus we cannot have our system simultaneously talk to many remote GKE clusters because each needs a unique set of Google credentials to authenticate.</p>
</blockquote>

<p>You can override your env variable for the specific shell that is running your go program with something like:</p>

<pre><code>os.Setenv("GOOGLE_APPLICATION_CREDENTIALS", value)
</code></pre>

<blockquote>
  <p>My second idea was to use the --impersonate-service-account flag provided to gcloud config config-helper</p>
</blockquote>

<p>Looks like a bug in the python code of the gcloud app. As described <a href="https://stackoverflow.com/questions/19192209/attributeerror-module-object-has-no-attribute-utcnow">here</a> <code>utcnow</code> is only applicable to <code>datetime.datetime</code> object. You could check if that module works in python shell in your system.</p>

<blockquote>
  <p>My final idea is quite complicated.</p>
</blockquote>

<p>Seems like this would work as long as the value of <code>credentials: "&lt;google-credentials-JSON&gt;"</code> doesn't change between different sessions to the GCP API (The credentials value might have an expiration)</p>

<p>Note: <a href="https://github.com/kubernetes/kubernetes/pull/77223" rel="nofollow noreferrer">PR</a> for the final idea.</p>
