<p>I am new to running Spark on Kubernetes and I have a very simple app I am trying to package to run in the Spark on K8s cluster that I have setup. The challenge I am facing is how to package my app to run in Spark? I installed Spark on K8's from the <a href="https://github.com/GoogleCloudPlatform/spark-on-k8s-operator" rel="nofollow noreferrer">Spark Operator</a> and I notice that the examples use the image <code>gcr.io/spark-operator/spark:v3.0.0</code>.</p>
<p>I noticed the Spark <a href="https://spark.apache.org/docs/latest/running-on-kubernetes.html#docker-images" rel="nofollow noreferrer">documentation</a> mentions this <code>docker-image-tool.sh</code> script to generate an app. but this looks like it would be for custom tailoring the environment. I would just like a lightweight image my app can use to run on the Spark cluster. Not sure how to tie it all together.</p>
<p>So I see 2 options from the documentation</p>
<ol>
<li>Use <a href="https://github.com/apache/spark/blob/master/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile" rel="nofollow noreferrer">this docker file</a> as a base image.</li>
<li>use the <code>docker-image-tool.sh</code> script.</li>
</ol>
<p>But I'm not sure which option I need to go for or when to use either option? Why use one over the other? Is there another option? Is there a prebuilt image that I can just copy my application into and use that to run?</p>

<p>Spark's <code>docker-image-tool.sh</code> is a tool script to create Spark's image. If you want a lightweight docker image, you can just tweak the Dockerfile that comes with the project or write your own - don't forget to edit the <code>entrypoint.sh</code> script as well.</p>
<p>Generally the steps for getting your Spark App to Kubernetes goes like this:</p>
<ol>
<li>Create a base spark image for you to use.</li>
<li>Push it to a docker registry - if you are using minikube, the  you can use the <code>-m</code> flag to move it to the minikube env.</li>
<li>Write a Spark Application, Package it and then run the <code>spark-submit</code> command.</li>
</ol>
<p>Note:
If you are not invested so much in Kubernetes and just want to quickly try this whole platform out, you can just proxy the <code>kube-api-server</code> by running the following command:</p>
<pre><code>kubectl proxy
</code></pre>
<p>It would start serving the api server on <code>localhost:8001</code>, then you can submit your spark app by running a command like</p>
<pre><code>bin/spark-submit \
--master k8s://http:localhost:8001 \ #If you don't specify the protocol here, it'd default to https
--deploy-mode client \ #if you want to go for `cluster mode` then you'd have to set up a service account
--name sparkle \
--class edu.girdharshubham \ 
--conf spark.executor.instances=1 \ #Number of pods
--conf spark.kubernetes.container.image=&lt;your-spark-image&gt;
--conf spark.kubernetes.driver.pod.name=&quot;sparkle&quot;
--conf spark.kubernetes.hadoop.configMapName=&quot;HADOOP_CONF_DIR&quot;
--conf spark.kubernetes.executor.deleteOnTermination
path/to/jar
</code></pre>
<p>Consideration for running in client mode:</p>
<ul>
<li>Make sure that the node - VM - you are running the aforementioned command from is network addressable from the pods</li>
<li>executor(pod) will run a JAVA process</li>
<li>The default Dockerfile uses tini -  tiny but valid <code>init</code> for containers.</li>
<li><code>spark.kubernetes.executor.deleteOnTermination</code> - This conf should be your go to conf if you are just starting out, by default pods are deleted in case of failure or normal termination. This would help you debug rather quickly about what's going on with your executor pods - whether they are failing or not.</li>
</ul>
