<p>I'm using Amazon EKS for Kubernetes deployment (initially created by an AWS admin user), and currently having difficulty to use the AWS credentials from AWS STS assume-role to execute <code>kubectl</code> commands to interact with the stack</p>

<p>I have 2 EKS stacks on 2 different AWS accounts (PROD &amp; NONPROD), and I'm trying to get the CI/CD tool to deploy to both kubernetes stacks with the credentials provided by AWS STS assume-role but I'm constantly getting error such as <code>error: You must be logged in to the server (the server has asked for the client to provide credentials)</code>.  </p>

<p>I have followed the following link to add additional AWS IAM role to the config:</p>

<ul>
<li><a href="https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html" rel="noreferrer">https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html</a></li>
</ul>

<p>But I'm not sure what I'm not doing right.</p>

<p>I ran "aws eks update-kubeconfig" to update the local .kube/config file, contents populated as below:</p>

<pre><code>apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: [hidden]
    server: https://[hidden].eu-west-1.eks.amazonaws.com
  name: arn:aws:eks:eu-west-1:[hidden]:cluster/demo-eks
contexts:
- context:
    cluster: arn:aws:eks:eu-west-1:[hidden]:cluster/demo-eks
    user: arn:aws:eks:eu-west-1:[hidden]:cluster/demo-eks
  name: arn:aws:eks:eu-west-1:[hidden]:cluster/demo-eks
current-context: arn:aws:eks:eu-west-1:[hidden]:cluster/demo-eks
kind: Config
preferences: {}
users:
- name: arn:aws:eks:eu-west-1:[hidden]:cluster/demo-eks
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      args:
      - token
      - -i
      - triage-eks
      command: aws-iam-authenticator
</code></pre>

<p>and had previously updated Kubernetes aws-auth ConfigMap with an additional role as below:</p>

<pre><code>data:
  mapRoles: |
    - rolearn: arn:aws:iam::[hidden]:role/ci_deployer
      username: system:node:{{EC2PrivateDNSName}}
      groups:
        - system:masters
</code></pre>

<p>My CI/CD EC2 instance can assume the <code>ci_deployer</code> role for either AWS accounts.</p>

<p>Expected: I can call "kubectl version" to see both Client and Server versions</p>

<p>Actual: but I get "the server has asked for the client to provide credentials"</p>

<p>What is still missing?</p>

<p>After further testing, I can confirm kubectl will only work from an environment (e.g. my CI EC2 instance with an AWS instance role) of the same AWS account where the EKS stack is created.  This means that my CI instance from account A will not be able to communicate with EKS from account B, even if the CI instance can assume a role from account B, and the account B role is included in the aws-auth of the kube config of account B EKS.  I hope its due to missing configuration as I find this rather undesirable if a CI tool can't deploy to multiple EKS's from multiple AWS accounts using role assumption. </p>

<p>Look forward to further @Kubernetes support on this</p>

Raymond Ng
<blockquote>
  <p>Can kubectl work from an assumed role from AWS</p>
</blockquote>

<p>Yes, it can work. A good way to troubleshoot it is to run from the same command line where you are running kubectl:</p>

<pre><code>$ aws sts get-caller-identity
</code></pre>

<p>You can see the <a href="https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html" rel="noreferrer"><code>Arn</code></a> for the role (or user) and then make sure there's a <a href="https://docs.aws.amazon.com/directoryservice/latest/admin-guide/edit_trust.html" rel="noreferrer">trust relationship</a> in <a href="https://aws.amazon.com/iam/" rel="noreferrer">IAM</a> between that and the role that you specify here in your <a href="https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/" rel="noreferrer">kubeconfig</a>:</p>

<pre><code>command: aws-iam-authenticator
args:
   - "token"
   - "-i"
   - "&lt;cluster-name&gt;"
   - "-r"
   - "&lt;role-you-want-to-assume-arn&gt;"
</code></pre>

<p>or with the newer option:</p>

<pre><code>command: aws
args:
- eks
- get-token
- --cluster-name
- &lt;cluster-name&gt;
- --role
- &lt;role-you-want-to-assume-arn&gt;
</code></pre>

<p>Note that if you are using <code>aws eks update-kubeconfig</code> you can pass in the <code>--role-arn</code> flag to generate the above in your kubeconfig.</p>

<p>In your case, some things that you can look at:</p>

<ul>
<li><p>The credential environment variables are not set in your CI?:</p>

<pre><code>AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
</code></pre></li>
<li><p>Your ~/.aws/credentials file is not populated correctly in your CI. With something like this:</p>

<pre><code>[default]
aws_access_key_id = xxxx
aws_secret_access_key = xxxx
</code></pre></li>
<li><p>Generally, the environment variables take precedence so it could be that you could have different credentials altogether in those environment variables too.</p></li>
<li><p>It could also be the <code>AWS_PROFILE</code> env variable or the <code>AWS_PROFILE</code> config in <code>~/.kube/config</code></p>

<pre><code>users:
- name: aws
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      command: aws-iam-authenticator
      args:
        - "token"
        - "-i"
        - "&lt;cluster-name&gt;"
        - "-r"
        - "&lt;role-arn&gt;"
      env:
        - name: AWS_PROFILE &lt;== is this value set
          value: "&lt;aws-profile&gt;"
</code></pre></li>
<li><p>Is the profile set correctly under <code>~/.aws/config</code>?</p></li>
</ul>
