<p>I am trying to run spark from jupyterhub against a eks cluster that uses IRSA.I followed examples presented in
<a href="https://stackoverflow.com/questions/64625111/aws-eks-spark-3-0-hadoop-3-2-error-noclassdeffounderror-com-amazonaws-servic">AWS EKS Spark 3.0, Hadoop 3.2 Error - NoClassDefFoundError: com/amazonaws/services/s3/model/MultiObjectDeleteException</a>
and <a href="https://medium.com/swlh/how-to-perform-a-spark-submit-to-amazon-eks-cluster-with-irsa-50af9b26cae" rel="nofollow noreferrer">https://medium.com/swlh/how-to-perform-a-spark-submit-to-amazon-eks-cluster-with-irsa-50af9b26cae</a> for code and IRSA role examples. However I am getting an unable to find a region via the region provider chain error. I have tried using different spark versions and aws sdk versions and hardcoded AWS_DEFAULT_REGION values, it did not resolve the issue. Appreciate any advise on resolving the issue.</p>
<pre><code>SPARK_HADOOP_VERSION=&quot;3.2&quot;
HADOOP_VERSION=&quot;3.2.0&quot;   
SPARK_VERSION=&quot;3.0.1&quot;
AWS_VERSION=&quot;1.11.874&quot;
TINI_VERSION=&quot;0.18.0&quot;
</code></pre>
<p>I am adding below jars in my spark jars folder,</p>
<pre><code>&quot;https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/${HADOOP_VERSION}/hadoop-aws-${HADOOP_VERSION}.jar&quot;
&quot;https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/${AWS_VERSION}/aws-java-sdk-bundle-${AWS_VERSION}.jar&quot;
&quot;https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk/${AWS_VERSION}/aws-java-sdk-${AWS_VERSION}.jar&quot;
&quot;https://s3.amazonaws.com/redshift-downloads/drivers/jdbc/1.2.51.1078/RedshiftJDBC42-no-awssdk-1.2.51.1078.jar&quot;
</code></pre>
<p>Sample spark session builder code</p>
<pre><code>SPARK_DRIVER_PACKAGES = ['org.apache.spark:spark-core_2.12:3.0.1',
                         'org.apache.spark:spark-avro_2.12:3.0.1',
                         'org.apache.spark:spark-sql_2.12:3.0.1',
                         'io.github.spark-redshift-community:spark-redshift_2.12:4.2.0',
                         'org.postgresql:postgresql:42.2.14',
                         'mysql:mysql-connector-java:8.0.22',
                         'org.apache.hadoop:hadoop-aws:3.2.0',
                         'com.amazonaws:aws-java-sdk-bundle:1.11.874']

spark_session = SparkSession.builder.master(master_host)\
    .appName(&quot;pyspark_session_app_1&quot;)\
    .config('spark.driver.host', local_ip)\
    .config('spark.kubernetes.authenticate.driver.serviceAccountName', 'spark')\
    .config('spark.kubernetes.authenticate.executor.serviceAccountName', 'spark')\
    .config(&quot;spark.kubernetes.executor.annotation.eks.amazonaws.com/role-arn&quot;,&quot;arn:aws:iam::xxxxxxxxx:role/spark-irsa&quot;) \
    .config('spark.kubernetes.driver.limit.cores', 0.2)\
    .config('spark.hadoop.fs.s3a.aws.credentials.provider','com.amazonaws.auth.WebIdentityTokenCredentialsProvider')\
    .config(&quot;spark.kubernetes.authenticate.submission.caCertFile&quot;, &quot;/var/run/secrets/kubernetes.io/serviceaccount/ca.crt&quot;) \
    .config(&quot;spark.kubernetes.authenticate.submission.oauthTokenFile&quot;, &quot;/var/run/secrets/kubernetes.io/serviceaccount/token&quot;)\
    .config('spark.kubernetes.executor.request.cores', executor_cores)\
    .config('spark.executor.instances', executor_instances)\
    .config('spark.executor.memory', executor_memory)\
    .config('spark.driver.memory', driver_memory)\
    .config('spark.kubernetes.executor.limit.cores', 1)\
    .config('spark.scheduler.mode', 'FAIR')\
    .config('spark.submit.deployMode', 'client')\
    .config('spark.kubernetes.container.image', SPARK_IMAGE)\
    .config('spark.kubernetes.container.image.pullPolicy', 'Always')\
    .config('spark.kubernetes.namespace', 'prod-data-science')\
    .config('spark.sql.execution.arrow.pyspark.enabled', 'true')\
    .config('spark.sql.execution.arrow.pyspark.fallback.enabled', 'true')\
    .config('spark.executorEnv.ARROW_PRE_0_15_IPC_FORMAT', '1')\
    .config('spark.jars.packages', ','.join(SPARK_DRIVER_PACKAGES))\
    .config(&quot;spark.hadoop.fs.s3a.multiobjectdelete.enable&quot;, &quot;false&quot;) \
    .config(&quot;spark.hadoop.fs.s3a.fast.upload&quot;,&quot;true&quot;) \
    .config(&quot;spark.hadoop.fs.s3a.impl&quot;, &quot;org.apache.hadoop.fs.s3a.S3AFileSystem&quot;) \
    .config('spark.eventLog.enabled','true')\
    .config('spark.eventLog.dir','s3a://spark-logs-xxxx/')\
    .getOrCreate()

</code></pre>
<p>error message received</p>
<pre><code>Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.
: java.nio.file.AccessDeniedException: spark-logs-xxxx: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials provided by WebIdentityTokenCredentialsProvider : com.amazonaws.SdkClientException: Unable to find a region via the region provider chain. Must provide an explicit region in the builder or setup environment to supply a region.
    at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:187)
    at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:111)
    at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$3(Invoker.java:265)
    at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:322)
    at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:261)
    at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:236)
    at org.apache.hadoop.fs.s3a.S3AFileSystem.verifyBucketExists(S3AFileSystem.java:375)
    at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:311)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3303)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3352)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3320)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479)
    at org.apache.spark.util.Utils$.getHadoopFileSystem(Utils.scala:1853)
    at org.apache.spark.deploy.history.EventLogFileWriter.&lt;init&gt;(EventLogFileWriters.scala:60)
    at org.apache.spark.deploy.history.SingleEventLogFileWriter.&lt;init&gt;(EventLogFileWriters.scala:213)
    at org.apache.spark.deploy.history.EventLogFileWriter$.apply(EventLogFileWriters.scala:181)
    at org.apache.spark.scheduler.EventLoggingListener.&lt;init&gt;(EventLoggingListener.scala:64)
    at org.apache.spark.SparkContext.&lt;init&gt;(SparkContext.scala:576)
    at org.apache.spark.api.java.JavaSparkContext.&lt;init&gt;(JavaSparkContext.scala:58)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:238)
    at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
    at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
    at py4j.GatewayConnection.run(GatewayConnection.java:238)
    at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials provided by WebIdentityTokenCredentialsProvider : com.amazonaws.SdkClientException: Unable to find a region via the region provider chain. Must provide an explicit region in the builder or setup environment to supply a region.
    at org.apache.hadoop.fs.s3a.AWSCredentialProviderList.getCredentials(AWSCredentialProviderList.java:159)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1257)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:833)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:783)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5212)
    at com.amazonaws.services.s3.AmazonS3Client.getBucketRegionViaHeadRequest(AmazonS3Client.java:6013)
    at com.amazonaws.services.s3.AmazonS3Client.fetchRegionFromCache(AmazonS3Client.java:5986)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5196)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5158)
    at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1421)
    at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:1357)
    at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$verifyBucketExists$1(S3AFileSystem.java:376)
    at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:109)
    ... 29 more
Caused by: com.amazonaws.SdkClientException: Unable to find a region via the region provider chain. Must provide an explicit region in the builder or setup environment to supply a region.
    at com.amazonaws.client.builder.AwsClientBuilder.setRegion(AwsClientBuilder.java:462)
    at com.amazonaws.client.builder.AwsClientBuilder.configureMutableProperties(AwsClientBuilder.java:424)
    at com.amazonaws.client.builder.AwsSyncClientBuilder.build(AwsSyncClientBuilder.java:46)
    at com.amazonaws.auth.STSAssumeRoleWithWebIdentitySessionCredentialsProvider.buildStsClient(STSAssumeRoleWithWebIdentitySessionCredentialsProvider.java:125)
    at com.amazonaws.auth.STSAssumeRoleWithWebIdentitySessionCredentialsProvider.&lt;init&gt;(STSAssumeRoleWithWebIdentitySessionCredentialsProvider.java:97)
    at com.amazonaws.auth.STSAssumeRoleWithWebIdentitySessionCredentialsProvider.&lt;init&gt;(STSAssumeRoleWithWebIdentitySessionCredentialsProvider.java:40)
    at com.amazonaws.auth.STSAssumeRoleWithWebIdentitySessionCredentialsProvider$Builder.build(STSAssumeRoleWithWebIdentitySessionCredentialsProvider.java:226)
    at com.amazonaws.services.securitytoken.internal.STSProfileCredentialsService.getAssumeRoleCredentialsProvider(STSProfileCredentialsService.java:40)
    at com.amazonaws.auth.profile.internal.securitytoken.STSProfileCredentialsServiceProvider.getProfileCredentialsProvider(STSProfileCredentialsServiceProvider.java:39)
    at com.amazonaws.auth.profile.internal.securitytoken.STSProfileCredentialsServiceProvider.getCredentials(STSProfileCredentialsServiceProvider.java:71)
    at com.amazonaws.auth.WebIdentityTokenCredentialsProvider.getCredentials(WebIdentityTokenCredentialsProvider.java:76)
    at org.apache.hadoop.fs.s3a.AWSCredentialProviderList.getCredentials(AWSCredentialProviderList.java:137)
    ... 47 more

</code></pre>

<p>Identified the issue was the aws region associated with spark pods.Included was region variable on the base spark docker image and the issue got resolved</p>
