<p>I recently saw a <a href="https://github.com/apache/spark/pull/21092" rel="noreferrer">pull request</a> that was merged to the Apache/Spark repository that apparently adds initial Python bindings for PySpark on K8s. I posted a comment to the PR asking a question about how to use spark-on-k8s in a Python Jupyter notebook, and was told to ask my question here. </p>

<p>My question is:</p>

<p>Is there a way to create SparkContexts using PySpark's <code>SparkSession.Builder</code> with master set to <code>k8s://&lt;...&gt;:&lt;...&gt;</code>, and have the resulting jobs run on <code>spark-on-k8s</code>, instead of on <code>local</code>?</p>

<p>E.g.:</p>

<pre><code>from pyspark.sql import SparkSession
spark = SparkSession.builder.master('k8s://https://kubernetes:443').getOrCreate()
</code></pre>

<p>I have an interactive Jupyter notebook running inside a Kubernetes pod, and I'm trying to use PySpark to create a <code>SparkContext</code> that runs on spark-on-k8s instead of resorting to using <code>local[*]</code> as <code>master</code>.</p>

<p>Till now, I've been getting an error saying that:</p>

<blockquote>
  <p>Error: Python applications are currently not supported for Kubernetes.</p>
</blockquote>

<p>whenever I set <code>master</code> to <code>k8s://&lt;...&gt;</code>.</p>

<p>It seems like PySpark always runs in <code>client</code> mode, which doesn't seem to be supported for <code>spark-on-k8s</code> at the moment -- perhaps there's some workaround that I'm not aware of.</p>

<p>Thanks in advance!</p>

<p>pyspark client mode works on Spark's latest version 2.4.0</p>

<p>This is how I did it (in Jupyter lab):</p>

<pre><code>import os
os.environ['PYSPARK_PYTHON']="/usr/bin/python3.6"
os.environ['PYSPARK_DRIVER_PYTHON']="/usr/bin/python3.6"

from pyspark import SparkContext, SparkConf
from pyspark.sql import SparkSession

sparkConf = SparkConf()
sparkConf.setMaster("k8s://https://localhost:6443")
sparkConf.setAppName("KUBERNETES-IS-AWESOME")
sparkConf.set("spark.kubernetes.container.image", "robot108/spark-py:latest")
sparkConf.set("spark.kubernetes.namespace", "playground")

spark = SparkSession.builder.config(conf=sparkConf).getOrCreate()
sc = spark.sparkContext
</code></pre>

<p>Note: I am running kubernetes locally on Mac with Docker Desktop.</p>
