title: spark cannot access customer's s3 bucket from emr
body: <p>I am struggling to connect to another account's s3 bucket. The problem is I can connect to the bucket via s3 API</p>
  
  <pre><code>def listFilesAndRename(key : String) = {
      val clientBucket = "test"
      val credentials = new BasicAWSCredentials("access key", "secret key")
      val s3 = AmazonS3ClientBuilder.standard.withRegion(Regions.US_EAST_1).withCredentials(new AWSStaticCredentialsProvider(credentials)).build
  
    val result = s3.listObjectsV2(clientBucket, key)
    val objects = result.getObjectSummaries.toArray
    for {
      os &lt;- objects
      fileName = os.asInstanceOf[S3ObjectSummary].getKey
      if !fileName.endsWith("_SUCCESS")
         } {
      System.out.println("* " + os.asInstanceOf[S3ObjectSummary].getKey)
    }
  }
  </code></pre>
  
  <p>but when i want to connect using spark, it does not work. I tried the following settings</p>
  
  <pre><code>spark.conf.set("spark.hadoop.fs.s3a.access.key", "access key")
  spark.conf.set("spark.hadoop.fs.s3a.secret.key", "secret key")
  spark.conf.set("spark.hadoop.fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")
  </code></pre>
  
  <p>Caused by: </p>
  
  <pre><code>com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 799F611A3D7BB5E1; S3 Extended Request ID: USTzJmBtWt27ccvF+YuTogwDSknuvB2tBLHxqPySLwftHXfmPpPYVjqVubzeVnqjJS89fzIjJw8=)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1712)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1367)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1113)
    at com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:770)
  </code></pre>
  
  <p>So clearly this is not a connection error, but looks like spark is unable to connect to the s3 bucket</p>