<p><strong>In a k8s cluster. How do you configure zeppelin to run spark jobs in an existing spark cluster instead of spinning up a new pod?</strong></p>
<p>I've got a k8s cluster up and running in which I want to run Spark with Zeppelin.</p>
<p>Spark is deployed using the official bitnami/spark helm chart (v 3.0.0). I got one Master and two Worker pods running fine, everything good.</p>
<p>Zeppelin is deployed with the <code>zeppelin-server.yaml</code> from the official apache-zeppelin github.</p>
<p>I've build my own zeppelin container without much modification from apache/zeppelin:0.9.0..</p>
<p>Short pseudo Dockerfile:</p>
<pre><code>FROM bitnami/spark:3.0.0 AS spark 
FROM apache/zeppelin:0.9-0 AS Zeppelin 
COPY --from spark /opt/btinami/spark/ /opt/bitnami/spark  
RUN Install kubectl 
END
</code></pre>
<p>I modified <code>zeppelin-server.yaml</code> slightly. (Image, imagePullSecret, setting spark master to the headless Service DNS of spark master)</p>
<p>Now I want my zeppelin jobs to run on my existing spark cluster --- with no success.</p>
<p>When I'm submitting zeppelin jobs (for the spark interpreter), zeppelin fires up a new spark pod and solely works with this one. Spark interpreter settings are like they should be. spark master url is set (<code>spark://\&lt;master-url\&gt;:\&lt;master-port\&gt;</code>), spark home as well.</p>
<p>While this is kind of a sweet behaviour, it's not what I want.</p>
<p>What I want (and what my question is) is: I want my zeppelin pod to submit the spark jobs to the existing cluster - not fire up a new pod. I am PRETTY sure that there has to be some <code>config/env/whatever</code> that I have to set but I simply can't find it.</p>
<p>So, I wanna ask: Is there anyone out there, who knows how to run zeppelin spark jobs on an existing spark cluster? I thought setting the spark master should do the job...</p>
<p>Kind regards
Bob</p>

<p>Answering myself after it has been a while...</p>
<p>For anyone running into the same problem:</p>
<ol>
<li><p>Go into the spark interpreter Settings</p>
</li>
<li><p>(optional, if you haven't already got the property) Press &quot;edit&quot;, scroll down and add the property SPARK_SUBMIT_OPTIONS</p>
</li>
<li><p>Edit SPARK_SUBMIT_OPTIONS value and add &quot;--master spark://&lt;ENDPOINT OF YOUR SPARK MASTER&gt; &quot;</p>
</li>
<li><p>Save settings and done...</p>
</li>
</ol>
<p>This threw me off massively, as there's already an option to set the spark master itself.</p>
<p>What solved the problem entering the spark master two times.</p>
<ol>
<li>Under key &quot;master&quot;</li>
<li>The edit to SPARK_SUBMIT_OPTIONS described above.</li>
</ol>
