<p>I'm trying to submit my Pyspark application to a Kubernetes cluster (Minikube) using spark-submit:</p>
<pre><code>./bin/spark-submit \
   --master k8s://https://192.168.64.4:8443 \
   --deploy-mode cluster \
   --packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.0.1 \
   --conf spark.kubernetes.container.image='pyspark:dev' \
   --conf spark.kubernetes.container.image.pullPolicy='Never' \
   local:///main.py
</code></pre>
<p>The application try to reach a Kafka instance deployed inside the cluster, so I specified the jar dependency:</p>
<pre><code>--packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.0.1
</code></pre>
<p>The container image I'm using is based on the one I've built with the utility script. I've packed all my python dependencies that my app need inside it.</p>
<p>The driver correctly deploy and get the Kafka package (I can provide the logs if needed) and launch the executor in a new pod.</p>
<p>But then the executor pod crash:</p>
<pre><code>ERROR Executor: Exception in task 0.0 in stage 1.0 (TID 1)
java.lang.ClassNotFoundException: org.apache.spark.sql.kafka010.KafkaBatchInputPartition
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:68)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1986)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1850)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2160)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2405)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2405)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2329)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2187)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1667)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:503)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:461)
at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:407)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
</code></pre>
<p>So I did my investigations on the executor pod and found that the jar is not present (as stated in the stack trace) in the $SPARK_CLASSPATH folder (which is set to ':/opt/spark/jars/*')</p>
<p>Do I need to also get and include the dependency in the spark jars folder when building the docker image? (I thought the '--packages' option would also make the executor retrieve the specified jar)</p>

<p>Did you start out with the official Dockerfile (kubernetes/dockerfiles/spark/bindings/python/Dockerfile) as described in the <a href="https://spark.apache.org/docs/latest/running-on-kubernetes.html#docker-images" rel="nofollow noreferrer">Docker images section</a> of the documentation?
You also need to specify an upload location on a Hadoop-compatible filesystem and make sure that the specified Ivy home and cache directories have the correct permissions, as described in the <a href="https://spark.apache.org/docs/latest/running-on-kubernetes.html#dependency-management" rel="nofollow noreferrer">Dependency Management section</a>.</p>
<p>Example from the docs:</p>
<pre><code>...
--packages com.amazonaws:aws-java-sdk:1.7.4,org.apache.hadoop:hadoop-aws:2.7.6
--conf spark.kubernetes.file.upload.path=s3a://&lt;s3-bucket&gt;/path
--conf spark.hadoop.fs.s3a.access.key=...
--conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem
--conf spark.hadoop.fs.s3a.fast.upload=true
--conf spark.hadoop.fs.s3a.secret.key=....
--conf spark.driver.extraJavaOptions=&quot;-Divy.cache.dir=/tmp -Divy.home=/tmp&quot;
...
</code></pre>
