title: Write Pyspark dataframe to S3
body: <p>Hello I am new to pyspark and I have a dataframe that I formed using the following method:</p>
  <pre><code>spark = SparkSession.builder \
      .appName(&quot;Python Spark SQL basic example&quot;) \
      .config(&quot;spark.some.config.option&quot;, &quot;some-value&quot;) \
      .getOrCreate()
  
  df = spark.read.option(&quot;header&quot;,True).csv(&quot;input.csv&quot;)
  </code></pre>
  <p>I now want to write this df to s3 but I have tried everything available online with no help.</p>
  <p>I first tried to set this up</p>
  <pre><code>spark.sparkContext.hadoopConfiguration.set(&quot;fs.s3n.access.key&quot;, &quot;my access key&quot;)
  spark.sparkContext.hadoopConfiguration.set(&quot;fs.s3n.secret.key&quot;, &quot;my secret key&quot;)
  spark.sparkContext.hadoopConfiguration.set(&quot;fs.s3n.endpoint&quot;, &quot;s3.amazonaws.com&quot;)
  </code></pre>
  <p>But for this I get the error:</p>
  <pre><code>AttributeError: 'SparkContext' object has no attribute 'hadoopConfiguration'
  </code></pre>
  <p>I also tried the following different methods to write:</p>
  <pre><code>df.write.option(&quot;header&quot;,&quot;true&quot;).csv(&quot;s3://mypath&quot;)
  df.write.parquet(&quot;s3://mypath&quot;, mode=&quot;overwrite&quot;)
  df.coalesce(1).write.format('csv').mode('overwrite').option(&quot;header&quot;, &quot;false&quot;)\
  .save(&quot;s3://mypath&quot;)
  </code></pre>
  <p>But for all these I get the same error:</p>
  <pre><code>: java.io.IOException: No FileSystem for scheme: s3
  </code></pre>
  <p>I am new to this and I really dont know what to do. Can anyone help me out?</p>