<p>I have spent days now trying to figure out a dependency issue I'm experiencing with (Py)Spark running on Kubernetes. I'm using the <a href="https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/" rel="noreferrer">spark-on-k8s-operator</a> and Spark's Google Cloud connector.</p>
<p>When I try to submit my spark job <strong>without</strong> a dependency using <code>sparkctl create sparkjob.yaml ...</code> with below .yaml file, it works like a charm.</p>
<pre><code>apiVersion: &quot;sparkoperator.k8s.io/v1beta2&quot;
kind: SparkApplication
metadata:
  name: spark-job
  namespace: my-namespace
spec:
  type: Python
  pythonVersion: &quot;3&quot;
  hadoopConf:
    &quot;fs.gs.impl&quot;: &quot;com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem&quot;
    &quot;fs.AbstractFileSystem.gs.impl&quot;: &quot;com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS&quot;
    &quot;fs.gs.project.id&quot;: &quot;our-project-id&quot;
    &quot;fs.gs.system.bucket&quot;: &quot;gcs-bucket-name&quot;
    &quot;google.cloud.auth.service.account.enable&quot;: &quot;true&quot;
    &quot;google.cloud.auth.service.account.json.keyfile&quot;: &quot;/mnt/secrets/keyfile.json&quot;
  mode: cluster
  image: &quot;image-registry/spark-base-image&quot;
  imagePullPolicy: Always
  mainApplicationFile: ./sparkjob.py
  deps:
    jars:
      - https://repo1.maven.org/maven2/org/apache/spark/spark-sql-kafka-0-10_2.11/2.4.5/spark-sql-kafka-0-10_2.11-2.4.5.jar
  sparkVersion: &quot;2.4.5&quot;
  restartPolicy:
    type: OnFailure
    onFailureRetries: 3
    onFailureRetryInterval: 10
    onSubmissionFailureRetries: 5
    onSubmissionFailureRetryInterval: 20
  driver:
    cores: 1
    coreLimit: &quot;1200m&quot;
    memory: &quot;512m&quot;
    labels:
      version: 2.4.5
    serviceAccount: spark-operator-spark
    secrets:
    - name: &quot;keyfile&quot;
      path: &quot;/mnt/secrets&quot;
      secretType: GCPServiceAccount
    envVars:
      GCS_PROJECT_ID: our-project-id
  executor:
    cores: 1
    instances: 1
    memory: &quot;512m&quot;
    labels:
      version: 2.4.5
    secrets:
    - name: &quot;keyfile&quot;
      path: &quot;/mnt/secrets&quot;
      secretType: GCPServiceAccount
    envVars:
      GCS_PROJECT_ID: our-project-id
</code></pre>
<p>The Docker image <code>spark-base-image</code> is built with Dockerfile</p>
<pre><code>FROM gcr.io/spark-operator/spark-py:v2.4.5

RUN rm $SPARK_HOME/jars/guava-14.0.1.jar
ADD https://repo1.maven.org/maven2/com/google/guava/guava/28.0-jre/guava-28.0-jre.jar $SPARK_HOME/jars

ADD https://repo1.maven.org/maven2/com/google/cloud/bigdataoss/gcs-connector/hadoop2-2.0.1/gcs-connector-hadoop2-2.0.1-shaded.jar $SPARK_HOME/jars

ENTRYPOINT [ &quot;/opt/entrypoint.sh&quot; ]
</code></pre>
<p>the main application file is uploaded to GCS when submitting the application and subsequently fetched from there and copied into the driver pod upon starting the application. The problem starts whenever I want to supply my own Python module <code>deps.zip</code> as a dependency to be able to use it in my main application file <code>sparkjob.py</code>.</p>
<p>Here's what I have tried so far:</p>
<p>1</p>
<p>Added the following lines to spark.deps in sparkjob.yaml</p>
<pre><code>pyFiles:
   - ./deps.zip
</code></pre>
<p>which resulted in the operator not being able to even submit the Spark application with error</p>
<pre><code>java.lang.RuntimeException: java.lang.ClassNotFoundException: Class com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem not found
</code></pre>
<p><code>./deps.zip</code> is successfully uploaded to the GCS bucket along with the main application file but while the main application file can be successfully fetched from GCS (I see this in the logs in jobs with no dependencies as defined above), <code>./deps.zip</code> can somehow not be fetched from there. I also tried adding the gcs-connector jar to the spark.deps.jars list explicitly - nothing changes.</p>
<p>2</p>
<p>I added <code>./deps.zip</code> to the base docker image used for starting up the driver and executor pods by adding <code>COPY ./deps.zip /mnt/</code> to the above Dockerfile and adding the dependency in the sparkjob.yaml via</p>
<pre><code>pyFiles:
    - local:///mnt/deps.zip
</code></pre>
<p>This time the spark job can be submitted and the driver pod is started, however I get a <code>file:/mnt/deps.zip not found</code> error when the Spark context is being initialized
I also tried to additionally set <code>ENV SPARK_EXTRA_CLASSPATH=/mnt/</code> in the Dockerfile but without any success. I even tried to explicitly mount the whole <code>/mnt/</code> directory into the driver and executor pods using volume mounts, but that also didn't work.</p>
<hr />
<p>edit:</p>
<p>My workaround (2), adding dependencies to the Docker image and setting <code>ENV SPARK_EXTRA_CLASSPATH=/mnt/</code> in the Dockerfile actually worked! Turns out the tag didn't update and I've been using an old version of the Docker image all along. Duh.</p>
<p>I still don't know why the (more elegant) solution 1 via the gcs-connector isn't working, but it might be related to <a href="https://stackoverflow.com/questions/62408395/mountvolume-setup-failed-for-volume-spark-conf-volume">MountVolume.Setup failed for volume &quot;spark-conf-volume&quot;</a></p>

denise
<p>I had same problems with s3a (org.apache.hadoop.fs.s3a.S3AFileSystem not found). When I tried:</p>
<pre><code>  deps:
    files:
      - &quot;s3a://k8s-3c172e28d7da2e-bucket/test.jar&quot;
</code></pre>
<p>Even added jars files inside image: &quot;image-registry/spark-base-image&quot; did not work. But I fixed this problem when I added necessary jars inside spark-operator pod. You can rebuild you Docker image by adding jars. In may case with s3, I rebuild it:</p>
<pre><code>FROM ghcr.io/googlecloudplatform/spark-operator:v1beta2-1.3.7-3.1.1

ENV SPARK_HOME /opt/spark

RUN curl https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/2.7.4/hadoop-aws-2.7.4.jar -o ${SPARK_HOME}/jars/hadoop-aws-2.7.4.jar
RUN curl https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk/1.7.4/aws-java-sdk-1.7.4.jar -o ${SPARK_HOME}/jars/aws-java-sdk-1.7.4.jar
</code></pre>
<p>In spark-operator inside has hadoop version 2.7 and we need use all dependencies exactly for this version on <a href="https://mvnrepository.com/" rel="nofollow noreferrer">https://mvnrepository.com/</a></p>
<p>First for tests I went to inside spark-operator pod by command</p>
<pre><code>kubectl exec -it spark-operator-fb8f779cb-gt657 -n spark-operator -- bash 
</code></pre>
<p>where pod name is</p>
<blockquote>
<p>spark-operator-fb8f779cb-gt657</p>
</blockquote>
<p>and -n spark-operator it is name of namespace</p>
<p>you can get all you pods and find pod name just use</p>
<p>kubectl get po -A</p>
<p>Then inside my spark-operator pod I go to /opt/spark/jars and upload jars (for example curl <a href="https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/2.7.4/hadoop-aws-2.7.4.jar" rel="nofollow noreferrer">https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/2.7.4/hadoop-aws-2.7.4.jar</a>)</p>
<p>Then I tried apply my manifest with deps.files and it is worked.</p>
