<p>I'm trying to run a PySpark Job using Kubernetes. Both the main script and the py-files are hosted on Google Cloud storage.
If I launch the Job using the standalone resource manager:</p>

<pre><code>spark-submit \
--master local \
--deploy-mode client \
--repositories "http://central.maven.org/maven2/" \
--packages "org.postgresql:postgresql:42.2.2" \
--py-files https://storage.googleapis.com/foo/some_dependencies.zip \
https://storage.googleapis.com/foo/script.py some args
</code></pre>

<p>It works fine.
But if I try the same using Kubernetes:</p>

<pre><code>spark-submit \
--master k8s://https://xx.xx.xx.xx  \
--deploy-mode cluster \
--conf spark.kubernetes.container.image=gcr.io/my-spark-image \
--repositories "http://central.maven.org/maven2/" \
--packages "org.postgresql:postgresql:42.2.2" \
--py-files https://storage.googleapis.com/foo/some_dependencies.zip \
https://storage.googleapis.com/foo/script.py  some args
</code></pre>

<p>Then the main script runs, but it can't find the modules in the dependencies files.
I know I can copy all the files inside the Docker image but I would prefer doing it this way.</p>

<p>Is this possible? Am I missing something?</p>

<p>Thanks</p>

<p>So the idea behind the k8s scheduler is to put absolutely everything in the container.</p>
<p>So your CI/CD would build a Dockerfile with the Apache Spark kubernetes Docker as its base and then have a zipped copy of your python repo and driver python script inside the docker image. Like this:</p>
<pre><code>$ bin/spark-submit \
    --master k8s://&lt;k8s-apiserver-host&gt;:&lt;k8s-apiserver-port&gt; \
    --deploy-mode cluster \
    --py-files local:///path/to/repo/in/container/pyspark-repo.zip \
    --conf spark.kubernetes.container.image=pyspark-repo-docker-image:1.0.0 \
    local:///path/to/repo/in/container/pyspark-driver.py
</code></pre>
<p>Your <code>spark.kubernetes.container.image</code> should be your full application complete with a</p>
<ul>
<li>zip of the repo for <code>--py-files</code> (ex: repo.zip)</li>
<li>your <code>requirements.txt</code> installed to the container's version of python (done in your repo's Dockerfile)</li>
<li>driver script (ex: driver.py)</li>
</ul>
