title: 403 Error while accessing s3a using Spark
body: <p><strong>ISSUE:</strong></p>
  
  <p>Able to successfully download the file using AWS CLI as well as boto 3.
  However, while using the S3A connector of Hadoop/Spark , receiving the below error:</p>
  
  <pre><code>py4j.protocol.Py4JJavaError: An error occurred while calling o24.parquet.
  : com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 403, AWS Service: Amazon S3, AWS Request ID: BCFFD14CB2939D68, AWS Error Code: null, AWS Error Message: Forbidden, S3 Extended Request ID: MfT8J6ZPlJccgHBXX+tX1fpX47V7dWCP3Dq+W9+IBUfUhsD4Nx+DcyqsbgbKsPn8NZzjc2U
  </code></pre>
  
  <p><strong>Configuration:</strong>
  Running this on my Local machine</p>
  
  <ol>
  <li><p>Spark Version 2.4.4</p></li>
  <li><p>Hadoop Version 2.7</p></li>
  </ol>
  
  <p>Jars added: </p>
  
  <ol>
  <li><p>hadoop-aws-2.7.3.jar</p></li>
  <li><p>aws-java-sdk-1.7.4.jar</p></li>
  </ol>
  
  <p><strong>Hadoop Config:</strong></p>
  
  <pre><code>hadoop_conf.set("fs.s3a.access.key", access_key)
  hadoop_conf.set("fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")
  hadoop_conf.set("fs.s3a.secret.key", secret_key)
  hadoop_conf.set("fs.s3a.aws.credentials.provider","org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider")
  hadoop_conf.set("fs.s3a.session.token", session_key)
  hadoop_conf.set("fs.s3a.endpoint", "s3-us-west-2.amazonaws.com") # yes, I am using central eu server.
  hadoop_conf.set("com.amazonaws.services.s3.enableV4", "true")
  </code></pre>
  
  <p><strong>Code to Read the file:</strong></p>
  
  <pre><code>from pyspark import SparkConf, SparkContext, SQLContext
  sc = SparkContext.getOrCreate()
  hadoop_conf=sc._jsc.hadoopConfiguration()
  sqlContext = SQLContext(sc)
  df = sqlContext.read.parquet(path)
  print(df.head())
  </code></pre>