<p>I have spent days now trying to figure out a dependency issue I'm experiencing with (Py)Spark running on Kubernetes. I'm using the <a href="https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/" rel="noreferrer">spark-on-k8s-operator</a> and Spark's Google Cloud connector.</p>
<p>When I try to submit my spark job <strong>without</strong> a dependency using <code>sparkctl create sparkjob.yaml ...</code> with below .yaml file, it works like a charm.</p>
<pre><code>apiVersion: &quot;sparkoperator.k8s.io/v1beta2&quot;
kind: SparkApplication
metadata:
  name: spark-job
  namespace: my-namespace
spec:
  type: Python
  pythonVersion: &quot;3&quot;
  hadoopConf:
    &quot;fs.gs.impl&quot;: &quot;com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem&quot;
    &quot;fs.AbstractFileSystem.gs.impl&quot;: &quot;com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS&quot;
    &quot;fs.gs.project.id&quot;: &quot;our-project-id&quot;
    &quot;fs.gs.system.bucket&quot;: &quot;gcs-bucket-name&quot;
    &quot;google.cloud.auth.service.account.enable&quot;: &quot;true&quot;
    &quot;google.cloud.auth.service.account.json.keyfile&quot;: &quot;/mnt/secrets/keyfile.json&quot;
  mode: cluster
  image: &quot;image-registry/spark-base-image&quot;
  imagePullPolicy: Always
  mainApplicationFile: ./sparkjob.py
  deps:
    jars:
      - https://repo1.maven.org/maven2/org/apache/spark/spark-sql-kafka-0-10_2.11/2.4.5/spark-sql-kafka-0-10_2.11-2.4.5.jar
  sparkVersion: &quot;2.4.5&quot;
  restartPolicy:
    type: OnFailure
    onFailureRetries: 3
    onFailureRetryInterval: 10
    onSubmissionFailureRetries: 5
    onSubmissionFailureRetryInterval: 20
  driver:
    cores: 1
    coreLimit: &quot;1200m&quot;
    memory: &quot;512m&quot;
    labels:
      version: 2.4.5
    serviceAccount: spark-operator-spark
    secrets:
    - name: &quot;keyfile&quot;
      path: &quot;/mnt/secrets&quot;
      secretType: GCPServiceAccount
    envVars:
      GCS_PROJECT_ID: our-project-id
  executor:
    cores: 1
    instances: 1
    memory: &quot;512m&quot;
    labels:
      version: 2.4.5
    secrets:
    - name: &quot;keyfile&quot;
      path: &quot;/mnt/secrets&quot;
      secretType: GCPServiceAccount
    envVars:
      GCS_PROJECT_ID: our-project-id
</code></pre>
<p>The Docker image <code>spark-base-image</code> is built with Dockerfile</p>
<pre><code>FROM gcr.io/spark-operator/spark-py:v2.4.5

RUN rm $SPARK_HOME/jars/guava-14.0.1.jar
ADD https://repo1.maven.org/maven2/com/google/guava/guava/28.0-jre/guava-28.0-jre.jar $SPARK_HOME/jars

ADD https://repo1.maven.org/maven2/com/google/cloud/bigdataoss/gcs-connector/hadoop2-2.0.1/gcs-connector-hadoop2-2.0.1-shaded.jar $SPARK_HOME/jars

ENTRYPOINT [ &quot;/opt/entrypoint.sh&quot; ]
</code></pre>
<p>the main application file is uploaded to GCS when submitting the application and subsequently fetched from there and copied into the driver pod upon starting the application. The problem starts whenever I want to supply my own Python module <code>deps.zip</code> as a dependency to be able to use it in my main application file <code>sparkjob.py</code>.</p>
<p>Here's what I have tried so far:</p>
<p>1</p>
<p>Added the following lines to spark.deps in sparkjob.yaml</p>
<pre><code>pyFiles:
   - ./deps.zip
</code></pre>
<p>which resulted in the operator not being able to even submit the Spark application with error</p>
<pre><code>java.lang.RuntimeException: java.lang.ClassNotFoundException: Class com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem not found
</code></pre>
<p><code>./deps.zip</code> is successfully uploaded to the GCS bucket along with the main application file but while the main application file can be successfully fetched from GCS (I see this in the logs in jobs with no dependencies as defined above), <code>./deps.zip</code> can somehow not be fetched from there. I also tried adding the gcs-connector jar to the spark.deps.jars list explicitly - nothing changes.</p>
<p>2</p>
<p>I added <code>./deps.zip</code> to the base docker image used for starting up the driver and executor pods by adding <code>COPY ./deps.zip /mnt/</code> to the above Dockerfile and adding the dependency in the sparkjob.yaml via</p>
<pre><code>pyFiles:
    - local:///mnt/deps.zip
</code></pre>
<p>This time the spark job can be submitted and the driver pod is started, however I get a <code>file:/mnt/deps.zip not found</code> error when the Spark context is being initialized
I also tried to additionally set <code>ENV SPARK_EXTRA_CLASSPATH=/mnt/</code> in the Dockerfile but without any success. I even tried to explicitly mount the whole <code>/mnt/</code> directory into the driver and executor pods using volume mounts, but that also didn't work.</p>
<hr />
<p>edit:</p>
<p>My workaround (2), adding dependencies to the Docker image and setting <code>ENV SPARK_EXTRA_CLASSPATH=/mnt/</code> in the Dockerfile actually worked! Turns out the tag didn't update and I've been using an old version of the Docker image all along. Duh.</p>
<p>I still don't know why the (more elegant) solution 1 via the gcs-connector isn't working, but it might be related to <a href="https://stackoverflow.com/questions/62408395/mountvolume-setup-failed-for-volume-spark-conf-volume">MountVolume.Setup failed for volume &quot;spark-conf-volume&quot;</a></p>

<p>If the zip files contain jar which you always shall require while running your spark job, facing a similar issue I just added</p>
<pre><code>FROM gcr.io/spark-operator/spark-py:v2.4.5
COPY mydepjars/ /opt/spark/jars/
</code></pre>
<p>And everything is getting loaded within my spark session. Could be one way to do it.</p>
