<p>I am new to <a href="https://spark.apache.org/" rel="nofollow noreferrer">spark</a>. I am trying to get <a href="https://spark.apache.org/" rel="nofollow noreferrer">spark</a> running on <a href="https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/" rel="nofollow noreferrer">k8s</a> using <a href="https://github.com/helm/charts/tree/master/stable/spark" rel="nofollow noreferrer">helm chart: stable/spark</a>. I can see that it spins up the 1 <strong>master</strong> and 2 <strong>executer</strong> by default and exposes port: <strong>8080</strong> on <code>ClusterIP</code>. </p>

<p>Now what I have done is to expose the Port: <code>8080</code> via <code>elb</code> so I can see the <code>UI</code>.</p>

<p>Question is do I always have to bake the <code>jar</code> or <code>pySpark</code> code in the image I am using to spin the master or do I have other option as well ?</p>

<p>I don't want to use <code>k8s</code> as Cluster Manager for <code>spark</code>. I am trying to see if there is a way to host <code>spark</code> as an application on <code>k8s</code> and submit jobs to it as it is a standalone cluster with worker nodes.</p>

<p>so instead of using:</p>

<pre><code>spark-submit \
...
--master k8s://https://KUBECLUSTER-DNS-ADDRESS
</code></pre>

<p>I want to do:</p>

<pre><code>spark-submit \
...
--master spark://SPARK-MASTER-ELB-DNS
</code></pre>

<p>Also, I am trying to avoid baking the <code>job</code> in the <strong>spark docker image</strong></p>

devnull
<blockquote>
  <p>I don't want to use k8s as Cluster Manager for spark. I am trying to see if there is a way to host spark as an application on k8s and submit jobs to it as it is a standalone cluster with worker nodes.</p>
</blockquote>

<p>You can use <code>client</code> or <code>cluster</code> <a href="https://spark.apache.org/docs/latest/submitting-applications.html" rel="nofollow noreferrer">mode</a>. </p>

<p><strong><em>client:</em></strong></p>

<pre><code># Run on a Spark standalone cluster in client deploy mode
./bin/spark-submit \
  --class org.apache.spark.examples.SparkPi \
  --master spark://IP-ADDRESS-OF-MASTER:7077 \
  --executor-memory 20G \
  --total-executor-cores 100 \
  /path/to/examples.jar \
  1000
</code></pre>

<p><strong><em>cluster:</em></strong></p>

<pre><code>./bin/spark-submit \
  --class org.apache.spark.examples.SparkPi \
  --master spark://IP-ADDRESS-OF-MASTER:7077 \
  --deploy-mode cluster \
  --supervise \
  --executor-memory 20G \
  --total-executor-cores 100 \
  /path/to/examples.jar \
  1000
</code></pre>

<blockquote>
  <p>Also, I am trying to avoid baking the job in the spark docker image.</p>
</blockquote>

<p>The only way is to use <code>client</code> mode. Basically, your driver will be in whatever machine where you run <code>spark-submit</code> from and that will need to have all the bits you need to execute your job. The only downside is that you might be susceptible to network latency if the client is not co-located with your Kubernetes cluster.</p>

<p>With <code>cluster</code> mode you will have bake stuff into your container image because your driver can start on any of the containers/pods that are slaves in your cluster.</p>
