title: PySpark s3 Access with Multiple AWS Credential Profiles?
body: <p>I'm writing a pyspark job that needs to read out of <strong>two</strong> different s3 buckets. Each bucket has different credentials, which are stored on my machine as separate profiles in <code>~/.aws/credentials</code>.</p>
  
  <p>Is there a way to tell pyspark which profile to use when connecting to s3?</p>
  
  <p>When using a single bucket, I had set the <code>AWS_ACCESS_KEY_ID</code> and <code>AWS_SECRET_ACCESS_KEY</code> environment variables in <code>conf/spark-env.sh</code>. Naturally, this only works for accessing 1 of the 2 buckets.</p>
  
  <p>I am aware that I could set these values manually in pyspark when required, using:</p>
  
  <pre><code>sc._jsc.hadoopConfiguration().set("fs.s3n.awsAccessKeyId", "ABCD")
  sc._jsc.hadoopConfiguration().set("fs.s3n.awsSecretAccessKey", "EFGH")
  </code></pre>
  
  <p>But would prefer a solution where these values were not hard-coded in. Is that possible?</p>