<p>I'm trying to launch a Spark application which should be able to read and write to S3, using Spark Operator on Kubernetes and pySpark version 3.0.0. The Spark Operator is wworking nicely, but I soon realized that the application launched can't read files from S3 properly.</p>
<p>This command:</p>
<p><code>spark.read.json(&quot;s3a://bucket/path/to/data.json&quot;)</code></p>
<p>is throwing this exception:</p>
<pre><code>py4j.protocol.Py4JJavaError: An error occurred while calling o58.json.
java.lang.RuntimeException: java.lang.ClassNotFoundException: 
Class org.apache.hadoop.fs.s3a.S3AFileSystem not found
</code></pre>
<p>I tried this first using <code>gcr.io/spark-operator/spark-py:v3.0.0</code> as Spark image, and then tried adding some <code>.jars</code> to it with no success:</p>
<pre><code>ADD https://repo1.maven.org/maven2/org/apache/httpcomponents/httpclient/4.5.9/httpclient-4.5.9.jar $SPARK_HOME/jars
ADD https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/2.7.4/hadoop-aws-2.7.4.jar $SPARK_HOME/jars
ADD https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk/1.7.4/aws-java-sdk-1.7.4.jar $SPARK_HOME/jars
</code></pre>
<p>Here's my spark conf:</p>
<pre><code>    &quot;spark.hadoop.fs.s3a.endpoint&quot;: &quot;S3A_ENDPOINT&quot;
    &quot;spark.hadoop.fs.s3a.access.key&quot;: &quot;ACCESS_KEY&quot;
    &quot;spark.hadoop.fs.s3a.secret.key&quot;: &quot;SECRET_KEY&quot;
    &quot;spark.hadoop.fs.s3a.connection.ssl.enabled&quot;: &quot;false&quot;
    &quot;spark.hadoop.fs.s3a.impl&quot;: &quot;org.apache.hadoop.fs.s3a.S3AFileSystem&quot;
    &quot;spark.hadoop.fs.s3a.path.style.access&quot;: &quot;true&quot;
    &quot;spark.driver.extraClassPath&quot;: &quot;/opt/spark/jars/*&quot;
    &quot;spark.executor.extraClassPath&quot;: &quot;/opt/spark/jars/*&quot;
</code></pre>
<p>And my <code>$SPARK_HOME</code> is <code>/opt/spark</code>.</p>
<p>Is anyone able to read/write from S3 using Spark 3.0.0? Is this an issue with pyspark, exclusively? How can I &quot;fix&quot; this?
Thanks in advance!</p>

<p>I figured out how to do it:
Here’s a fork with the changes I made to the base docker image (just a few changes):</p>
<p><a href="https://github.com/Coqueiro/spark/tree/branch-3.0-s3" rel="nofollow noreferrer">https://github.com/Coqueiro/spark/tree/branch-3.0-s3</a>
I created a Makefile to aid distribution creation, but I basically just followed the official doc:</p>
<p><a href="http://spark.apache.org/docs/latest/building-spark.html" rel="nofollow noreferrer">http://spark.apache.org/docs/latest/building-spark.html</a></p>
<p>Also, here’s the image, already built and pushed to Docker Hub:
<a href="https://hub.docker.com/repository/docker/coqueirotree/spark-py" rel="nofollow noreferrer">https://hub.docker.com/repository/docker/coqueirotree/spark-py</a></p>
<p>It has Spark 3.0.0, Hadoop 3.2.0, S3A and Kubernetes support.</p>
