<p>The official <a href="https://spark.apache.org/docs/latest/running-on-kubernetes.html#cluster-mode" rel="nofollow noreferrer">spark documentation</a> only has information on the <code>spark-submit</code> method for deploying code to a spark cluster. It mentions we must prefix the address from kubernetes api server with <code>k8s://</code>. What should we do when deploying through <a href="https://github.com/GoogleCloudPlatform/spark-on-k8s-operator" rel="nofollow noreferrer">Spark Operator</a>?</p>
<p>For instance if I have a basic pyspark application that starts up like this how do I set the master:</p>
<pre><code>from pyspark import SparkConf, SparkContext
from pyspark.sql import SQLContext
from pyspark.sql.types import *
from pyspark import SparkConf, SparkContext

sc = SparkContext(&quot;local&quot;, &quot;Big data App&quot;)
spark = SQLContext(sc)
spark_conf = SparkConf().setMaster('local').setAppName('app_name')
</code></pre>
<p>Here I have <code>local</code>, where if I was running on a non-k8's cluster I would mention the master address with <code>spark://</code> prefix or <code>yarn</code>. Must I also use the <code>k8s://</code> prefix if deploying through the Spark Operator?
If not what should be used for master parameter?</p>

<p>It's better not to use <code>setMaster</code> in the code, but instead specify it when running the code via spark-submit, something like this (see <a href="https://spark.apache.org/docs/latest/running-on-kubernetes.html#submitting-applications-to-kubernetes" rel="nofollow noreferrer">documentation for details</a>):</p>
<pre class="lang-sh prettyprint-override"><code>./bin/spark-submit \
    --master k8s://https://&lt;k8s-apiserver-host&gt;:&lt;k8s-apiserver-port&gt; \
    --deploy-mode cluster \
    your_script.py
</code></pre>
<p>I haven't used Spark operator, but it should set master automatically, as I understand from the documentation.</p>
<p>you also need to get convert this code:</p>
<pre class="lang-py prettyprint-override"><code>sc = SparkContext(&quot;local&quot;, &quot;Big data App&quot;)
spark = SQLContext(sc)
spark_conf = SparkConf().setMaster('local').setAppName('app_name')
</code></pre>
<p>to more modern (see <a href="https://spark.apache.org/docs/latest/sql-getting-started.html#starting-point-sparksession" rel="nofollow noreferrer">doc</a>):</p>
<pre class="lang-py prettyprint-override"><code>from pyspark.sql import SparkSession

spark = SparkSession \
    .builder \
    .appName(&quot;Python Spark SQL basic example&quot;) \
    .config(&quot;spark.some.config.option&quot;, &quot;some-value&quot;) \
    .getOrCreate()
</code></pre>
<p>as <code>SQLContext</code> is deprecated.</p>
<p>P.S. I recommend to get through first chapters of Learning Spark, 2ed that is <a href="https://databricks.com/p/ebook/learning-spark-from-oreilly" rel="nofollow noreferrer">freely available from the Databricks site</a>.</p>
