title: reading text file from Amazon S3 with PySpark
body: <p>I am trying to get a Spark cluster to read data sources from Amazon S3 cloud storage. This results in the following error, for which I need some help diagnosing the problem: </p>
  
  <pre><code>&gt;&gt;&gt; sc.textFile("s3a://storage-bucket/s3test.txt").collect()
  
  py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
  : com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 400, AWS Service: Amazon S3, AWS Request ID: D47397DA8BCB4669, AWS Error Code: null, AWS Error Message: Bad Request, S3 Extended Request ID: /aBi99tozgFEsdRGubDwhriMsNQvl1jLOf8AJquA8VXxzkpPL/LLCWDFQQvYn4snHx5gx66/pXo=
  </code></pre>
  
  <p>This works fine by the way:</p>
  
  <pre><code>$ aws s3 cp s3://storage-bucket/s3test.txt ./s3text.txt
  download: s3://storage-bucket/s3test.txt to ./s3text.txt
  $ cat s3text.txt 
  Hello S3
  </code></pre>
  
  <p>Some more detail from the error message:</p>
  
  <pre><code>Caused by: org.jets3t.service.S3ServiceException: Service Error Message. -- ResponseCode: 403, ResponseStatus: Forbidden, XML Error Message: &lt;?xml version="1.0" encoding="UTF-8"?&gt;&lt;Error&gt;&lt;Code&gt;SignatureDoe
  sNotMatch&lt;/Code&gt;&lt;Message&gt;The request signature we calculated does not match the signature you provided. Check your key and signing method.&lt;/Message&gt;&lt;AWSAccessKeyId&gt;xxxxxxxxxxxxxxxxxx&lt;/AWSAccessKeyId&gt;&lt;St
  </code></pre>