<p>I have a python script in our K8s cluster that is run as a k8s Cronjob every few minutes. The script checks the nodes in the cluster and if a node is unhealthy for more than 5 minutes, it terminates the node.
To connect to AWS I use Boto3.
requirement.txt</p>
<pre><code>boto3==1.16.11
botocore==1.19.11
</code></pre>
<p>and the permissions are passed as pod annotations.</p>
<pre><code>Annotations:  iam.amazonaws.com/role: arn:aws:iam::123456789:role/k8s-nodes-monitoring-role
</code></pre>
<p>The IAM role has <code>arn:aws:iam::aws:policy/AmazonEC2FullAccess</code> policy and a valid trust policy.</p>
<pre><code>{
   &quot;Version&quot;: &quot;2012-10-17&quot;,
   &quot;Statement&quot;: [
{
  &quot;Effect&quot;: &quot;Allow&quot;,
  &quot;Principal&quot;: {
    &quot;Service&quot;: &quot;ec2.amazonaws.com&quot;
  },
  &quot;Action&quot;: &quot;sts:AssumeRole&quot;
},
{
  &quot;Sid&quot;: &quot;&quot;,
  &quot;Effect&quot;: &quot;Allow&quot;,
  &quot;Principal&quot;: {
    &quot;AWS&quot;: &quot;arn:aws:iam::123456789:role/nodes.my-domain.com&quot;
  },
  &quot;Action&quot;: &quot;sts:AssumeRole&quot;
}
]
}
</code></pre>
<p>The problem that I facing is that on some occasions the script throws <code>NoCredentialsError('Unable to locate credentials')</code> error. This behaviour is not consistent as on most occasions the script has successfully terminates the unhealthy node and I can cross-check it against AWS CloudTrail events.
I can see in kub2iam logs that the Get request receives 200 but the Put request receives 403.</p>
<pre><code>ime=&quot;2020-12-21T12:50:16Z&quot; level=info msg=&quot;GET /latest/meta-data/iam/security- 
credentials/k8s-nodes-monitoring-role (200) took 47918.000000 ns&quot; req.method=GET 
req.path=/latest/meta-data/iam/security-credentials/k8s-nodes-monitoring-role 
req.remote=100.116.203.13 res.duration=47918 res.status=200
time=&quot;2020-12-21T12:52:16Z&quot; level=info msg=&quot;PUT /latest/api/token (403) took 19352999.000000 
ns&quot; req.method=PUT req.path=/latest/api/token req.remote=100.116.203.14 
res.duration=1.9352999e+07 res.status=40
</code></pre>
<p>Any help or idea about how to debug this will be highly appreciated.</p>

Asim
<p>I dont know kube2iam in detail, but maybe you should switch to a AWS native way called IRSA (IAM Roles for Service Accounts). You can find all necessary information in this blog post: <a href="https://aws.amazon.com/de/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/" rel="nofollow noreferrer">https://aws.amazon.com/de/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/</a></p>
