<p>I have a service account which I am trying to use across multiple pods installed in the same namespace.</p>
<p>One of the pods is created by Airflow KubernetesPodOperator.
The other is created via Helm through Kubernetes deployment.</p>
<p>In the Airflow deployment, I see the IAM role being assigned and DynamoDB tables are created, listed etc however in the second helm chart deployment (or) in a test pod (<a href="https://aws.amazon.com/premiumsupport/knowledge-center/eks-pods-iam-role-service-accounts/" rel="nofollow noreferrer">created as shown here</a>), I keep getting <code>AccessDenied</code> error for <code>CreateTable</code> in DynamoDB.</p>
<p>I can see the AWS Role ARN being assigned to the service account and the service account being applied to the pod and the corresponding token file also being created, but I see <code>AccessDenied</code> exception.</p>
<pre><code>arn:aws:sts::1234567890:assumed-role/MyCustomRole/aws-sdk-java-1636152310195 is not authorized to perform: dynamodb:CreateTable on resource
</code></pre>
<p><strong>ServiceAccount</strong></p>
<pre><code>Name:                mypipeline-service-account
Namespace:           abc-qa-daemons
Labels:              app.kubernetes.io/managed-by=Helm
                     chart=abc-pipeline-main.651
                     heritage=Helm
                     release=ab-qa-pipeline
                     tier=mypipeline
Annotations:         eks.amazonaws.com/role-arn: arn:aws:iam::1234567890:role/MyCustomRole
                     meta.helm.sh/release-name: ab-qa-pipeline
                     meta.helm.sh/release-namespace: abc-qa-daemons
Image pull secrets:  &lt;none&gt;
Mountable secrets:   mypipeline-service-account-token-6gm5b
Tokens:              mypipeline-service-account-token-6gm5b
</code></pre>
<p>P.S: Both the client code created using <code>KubernetesPodOperator</code> and through Helm chart deployment is same i.e. same docker image. Other attributes like <code>nodeSelector</code>, <code>tolerations</code> etc, volume mounts are also same.</p>
<p>The <code>describe pod</code> output for both of them is similar with just some name and label changes.
The <code>KubernetesPodOperator</code> pod has QoS class as <code>Burstable</code> while the Helm chart ones is <code>BestEffort</code>.</p>
<p>Why do I get <code>AccessDenied</code> in Helm deployment but not in <code>KubernetesPodOperator</code>? How to debug this issue?</p>

JavaTechnical
<p>Whenever we get an <code>AccessDenied</code> exception, there can be two possible reasons:</p>
<ol>
<li>You have assigned the <em>wrong role</em></li>
<li>The assigned role doesn't have necessary <em>permissions</em></li>
</ol>
<p>In my case, latter is the issue. The permissions assigned to particular role can be sophisticated i.e. they can be more granular.</p>
<p>For example, in my case, the DynamoDB tables which the role can create/describe is limited to only those that are starting with a specific <em>prefix</em> but not all the DynamoDB tables.</p>
<blockquote>
<p>So, it is always advisable to check the IAM role permissions whenever
you get this error.</p>
</blockquote>
<p>As stated in the question, be sure to check the service account using the <code>awscli</code> image.</p>
<hr />
<p>Keep in mind that, there is a credential provider chain used in AWS SDKs which determines the credentials to be used by the application. In most cases, the <code>DefaultAWSCredentialsProviderChain</code> is used and its order is given below. Ensure that the SDK is picking up the intended provider (in our case it is <code>WebIdentityTokenCredentialsProvider</code>)</p>
<pre><code> super(new EnvironmentVariableCredentialsProvider(),
              new SystemPropertiesCredentialsProvider(),
              new ProfileCredentialsProvider(),
              WebIdentityTokenCredentialsProvider.create(),
              new EC2ContainerCredentialsProviderWrapper());
</code></pre>
<p>Additionally, you might also want to set the AWS SDK classes to DEBUG mode in your logger to see which credentials provider is being picked up and why.</p>
<hr />
<p>To check if the service account is applied to a pod, describe it and check if the AWS environment variables are set to it like <code>AWS_REGION</code>, <code>AWS_DEFAULT_REGION</code>, <code>AWS_ROLE_ARN</code> and <code>AWS_WEB_IDENTITY_TOKEN_FILE</code>.</p>
<p>If not, then check your service account if it has the AWS annotation <code>eks.amazonaws.com/role-arn</code> by describing that service account.</p>
