<p>I'm trying to construct a Kubernetes informer outside of the EKS cluster that it's watching. I'm using <a href="https://github.com/kubernetes-sigs/aws-iam-authenticator" rel="nofollow noreferrer">aws-iam-authenticator</a> plugin to provide the <a href="https://kubernetes.io/docs/reference/access-authn-authz/authentication/#client-go-credential-plugins" rel="nofollow noreferrer">exec-based credentials</a> to the EKS cluster. For the plugin to work, I'm assuming an IAM role and passing the AWS IAM credentials as environment variables.</p>
<p>The problem is that these credentials expire after an hour and cause the informer to fail with</p>
<blockquote>
<p>E0301 23:34:22.167817     582 runtime.go:79] Observed a panic: &amp;errors.StatusError{ErrStatus:v1.Status{TypeMeta:v1.TypeMeta{Kind:&quot;&quot;, APIVersion:&quot;&quot;}, ListMeta:v1.ListMeta{SelfLink:&quot;&quot;, ResourceVersion:&quot;&quot;, Continue:&quot;&quot;, RemainingItemCount:(*int64)(nil)}, Status:&quot;Failure&quot;, Message:&quot;the server has asked for the client to provide credentials (get pods)&quot;, Reason:&quot;Unauthorized&quot;, Details:(*v1.StatusDetails)(0xc0005b0300), Code:401}} (the server has asked for the client to provide credentials (get pods))</p>
</blockquote>
<p>Is there a better way of getting <code>ClientConfig</code> and <code>aws-iam-authenticator</code> to refresh the credentials?</p>
<p>Here's a rough skeleton of my code:</p>
<pre class="lang-golang prettyprint-override"><code>credentialsProvider := aws.NewCredentialsCache(stscreds.NewWebIdentityRoleProvider(...))
creds, err := credentialsProvider.Retrieve(ctx)

config := clientcmdapi.NewConfig()
// ...
config.AuthInfos[&quot;eks&quot;] = &amp;clientcmdapi.AuthInfo{
    Exec: &amp;clientcmdapi.ExecConfig{
        Command: &quot;aws-iam-authenticator&quot;,
        Args: []string{
            &quot;token&quot;,
            &quot;-i&quot;,
            clusterName,
        },
        // These env vars are static! :(
        Env: []clientcmdapi.ExecEnvVar{
            {
                Name:  &quot;AWS_ACCESS_KEY_ID&quot;,
                Value: creds.AccessKeyID,
            },
            {
                Name:  &quot;AWS_SECRET_ACCESS_KEY&quot;,
                Value: creds.SecretAccessKey,
            },
            {
                Name:  &quot;AWS_SESSION_TOKEN&quot;,
                Value: creds.SessionToken,
            },
        },
        APIVersion:      &quot;client.authentication.k8s.io/v1beta1&quot;,
        InteractiveMode: clientcmdapi.NeverExecInteractiveMode,
    },
}

restConfig, err := config.ClientConfig()
clientset, err = kubernetes.NewForConfig(restConfig)

informerFactory := informers.NewSharedInformerFactory(clientset, time.Second*30)
podInformer := cw.informerFactory.Core().V1().Pods().Informer()
</code></pre>
<p>Here are a couple similar threads I found:</p>
<ul>
<li><a href="https://stackoverflow.com/questions/74563117/kubernetes-client-go-informers-getting-unauthorized-error-after-15-mins">Kubernetes client-go informers getting &quot;Unauthorized&quot; error after 15 mins</a></li>
<li><a href="https://github.com/kubernetes/client-go/issues/1189" rel="nofollow noreferrer">https://github.com/kubernetes/client-go/issues/1189</a></li>
</ul>

<p>My solution was to create write the credentials to a file and create a background thread to refresh that file. I can then pass tell <code>aws-iam-authenticator</code> to read the credentials from the file via the <code>AWS_SHARED_CREDENTIALS_FILE</code> environment variable.</p>
<p>This might also be possible using <code>AWS_WEB_IDENTITY_TOKEN_FILE</code> to save some steps, but I didn't look further.</p>
<p>The updated code looks like this</p>
<pre><code>func updateCredentials(ctx context.Context) {
    creds, err := c.credentialsProvider.Retrieve(ctx)
    s := fmt.Sprintf(`[default]
aws_access_key_id=%s
aws_secret_access_key=%s
aws_session_token=%s`, creds.AccessKeyID, creds.SecretAccessKey, creds.SessionToken)
    err = os.WriteFile(credentialsFile.Name(), []byte(s), 0666)
    return nil
}

func updateCredentialsLoop(ctx context.Context) {
    for {
        err := updateCredentials(ctx)
        time.Sleep(5*time.Minute)
    }
}

credentialsProvider := aws.NewCredentialsCache(stscreds.NewWebIdentityRoleProvider(...))

credentialsFile, err := os.CreateTemp(&quot;&quot;, &quot;credentials&quot;)
updateCredentials(ctx)
go updateCredentialsLoop(ctx)

config := clientcmdapi.NewConfig()
// ...
config.AuthInfos[&quot;eks&quot;] = &amp;clientcmdapi.AuthInfo{
    Exec: &amp;clientcmdapi.ExecConfig{
        Command: &quot;aws-iam-authenticator&quot;,
        Args: []string{
            &quot;token&quot;,
            &quot;-i&quot;,
            clusterName,
        },
        Env: []clientcmdapi.ExecEnvVar{
            {
                Name:  &quot;AWS_SHARED_CREDENTIALS_FILE&quot;,
                Value: credentialsFile.Name(),
            },
        },
        APIVersion:      &quot;client.authentication.k8s.io/v1beta1&quot;,
        InteractiveMode: clientcmdapi.NeverExecInteractiveMode,
    },
}

restConfig, err := config.ClientConfig()
clientset, err = kubernetes.NewForConfig(restConfig)

informerFactory := informers.NewSharedInformerFactory(clientset, time.Second*30)
podInformer := cw.informerFactory.Core().V1().Pods().Informer()
</code></pre>
