<p>I am running pyspark application, v2.4.0, on Kubernetes, my spark application depends on numpy and tensorflow modules, please suggest the way to add these dependencies to Spark executors.</p>

<p>I have checked the documentation, we can include the remote dependencies using --py-files, --jars etc. but nothing mentioned about library dependencies.</p>

Lakshman Battini
<p>Found the way to add the library dependencies to Spark applications on K8S, thought of sharing it here.</p>

<p>Mention the required dependencies installation commands in Dockerfile and rebuild the spark image, when we submit the spark job, new container will be instantiated with the dependencies as well.</p>

<p>Dokerfile (/{spark_folder_path}/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/bindings/python/Dockerfile) contents:</p>

<pre><code>RUN apk add --no-cache python &amp;&amp; \
    apk add --no-cache python3 &amp;&amp; \
    python -m ensurepip &amp;&amp; \
    python3 -m ensurepip &amp;&amp; \
    # We remove ensurepip since it adds no functionality since pip is
    # installed on the image and it just takes up 1.6MB on the image
    rm -r /usr/lib/python*/ensurepip &amp;&amp; \
    pip install --upgrade pip setuptools &amp;&amp; \
    # You may install with python3 packages by using pip3.6
    pip install numpy &amp;&amp; \
    # Removed the .cache to save space
    rm -r /root/.cache
</code></pre>
