<p>I am deploying pyspark in my aks Kubernetes cluster using this guides:</p>
<ul>
<li><a href="https://towardsdatascience.com/ignite-the-spark-68f3f988f642" rel="nofollow noreferrer">https://towardsdatascience.com/ignite-the-spark-68f3f988f642</a></li>
<li><a href="http://blog.brainlounge.de/memoryleaks/getting-started-with-spark-on-kubernetes/" rel="nofollow noreferrer">http://blog.brainlounge.de/memoryleaks/getting-started-with-spark-on-kubernetes/</a></li>
</ul>
<p>I have deployed my driver pod as is explained in the links above:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: spark
  name: my-notebook-deployment
  labels:
    app: my-notebook
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-notebook
  template:
    metadata:
      labels:
        app: my-notebook
    spec:
      serviceAccountName: spark
      containers:
      - name: my-notebook
        image: pidocker-docker-registry.default.svc.cluster.local:5000/my-notebook:latest
        ports:
          - containerPort: 8888
        volumeMounts:
          - mountPath: /root/data
            name: my-notebook-pv
        workingDir: /root
        resources:
          limits:
            memory: 2Gi
      volumes:
        - name: my-notebook-pv
          persistentVolumeClaim:
            claimName: my-notebook-pvc
---
apiVersion: v1
kind: Service
metadata:
  namespace: spark
  name: my-notebook-deployment
spec:
  selector:
    app: my-notebook
  ports:
    - protocol: TCP
      port: 29413
  clusterIP: None
</code></pre>
<p>Then I can create the spark cluster using the following code:</p>
<pre class="lang-py prettyprint-override"><code>import os
from pyspark import SparkContext, SparkConf
from pyspark.sql import SparkSession
# Create Spark config for our Kubernetes based cluster manager
sparkConf = SparkConf()
sparkConf.setMaster(&quot;k8s://https://kubernetes.default.svc.cluster.local:443&quot;)
sparkConf.setAppName(&quot;spark&quot;)
sparkConf.set(&quot;spark.kubernetes.container.image&quot;, &quot;&lt;MYIMAGE&gt;&quot;)
sparkConf.set(&quot;spark.kubernetes.namespace&quot;, &quot;spark&quot;)
sparkConf.set(&quot;spark.executor.instances&quot;, &quot;7&quot;)
sparkConf.set(&quot;spark.executor.cores&quot;, &quot;2&quot;)
sparkConf.set(&quot;spark.driver.memory&quot;, &quot;512m&quot;)
sparkConf.set(&quot;spark.executor.memory&quot;, &quot;512m&quot;)
sparkConf.set(&quot;spark.kubernetes.pyspark.pythonVersion&quot;, &quot;3&quot;)
sparkConf.set(&quot;spark.kubernetes.authenticate.driver.serviceAccountName&quot;, &quot;spark&quot;)
sparkConf.set(&quot;spark.kubernetes.authenticate.serviceAccountName&quot;, &quot;spark&quot;)
sparkConf.set(&quot;spark.driver.port&quot;, &quot;29413&quot;)
sparkConf.set(&quot;spark.driver.host&quot;, &quot;my-notebook-deployment.spark.svc.cluster.local&quot;)
# Initialize our Spark cluster, this will actually
# generate the worker nodes.
spark = SparkSession.builder.config(conf=sparkConf).getOrCreate()
sc = spark.sparkContext
</code></pre>
<p>It works. But I would want to run spark jobs in another pod, but if I do that I have the following error:</p>
<pre class="lang-py prettyprint-override"><code>python-input-1-ca9882ef9eeb&gt; in &lt;module&gt;
     25 # Initialize our Spark cluster, this will actually
     26 # generate the worker nodes.
---&gt; 27 spark = SparkSession.builder.config(conf=sparkConf).getOrCreate()
     28 sc = spark.sparkContext

/usr/local/spark/python/pyspark/sql/session.py in getOrCreate(self)
    171                     for key, value in self._options.items():
    172                         sparkConf.set(key, value)
--&gt; 173                     sc = SparkContext.getOrCreate(sparkConf)
    174                     # This SparkContext may be an existing one.
    175                     for key, value in self._options.items():

/usr/local/spark/python/pyspark/context.py in getOrCreate(cls, conf)
    365         with SparkContext._lock:
    366             if SparkContext._active_spark_context is None:
--&gt; 367                 SparkContext(conf=conf or SparkConf())
    368             return SparkContext._active_spark_context
    369 

/usr/local/spark/python/pyspark/context.py in __init__(self, master, appName, sparkHome, pyFiles, environment, batchSize, serializer, conf, gateway, jsc, profiler_cls)
    134         try:
    135             self._do_init(master, appName, sparkHome, pyFiles, environment, batchSize, serializer,
--&gt; 136                           conf, jsc, profiler_cls)
    137         except:
    138             # If an error occurs, clean up in order to allow future SparkContext creation:

/usr/local/spark/python/pyspark/context.py in _do_init(self, master, appName, sparkHome, pyFiles, environment, batchSize, serializer, conf, jsc, profiler_cls)
    196 
    197         # Create the Java SparkContext through Py4J
--&gt; 198         self._jsc = jsc or self._initialize_context(self._conf._jconf)
    199         # Reset the SparkConf to the one actually used by the SparkContext in JVM.
    200         self._conf = SparkConf(_jconf=self._jsc.sc().conf())

/usr/local/spark/python/pyspark/context.py in _initialize_context(self, jconf)
    304         Initialize SparkContext in function to allow subclass specific initialization
    305         &quot;&quot;&quot;
--&gt; 306         return self._jvm.JavaSparkContext(jconf)
    307 
    308     @classmethod

/usr/local/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py in __call__(self, *args)
   1523         answer = self._gateway_client.send_command(command)
   1524         return_value = get_return_value(
-&gt; 1525             answer, self._gateway_client, None, self._fqn)
   1526 
   1527         for temp_arg in temp_args:

/usr/local/spark/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
    326                 raise Py4JJavaError(
    327                     &quot;An error occurred while calling {0}{1}{2}.\n&quot;.
--&gt; 328                     format(target_id, &quot;.&quot;, name), value)
    329             else:
    330                 raise Py4JError(

Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.
: java.net.BindException: Cannot assign requested address: Service 'sparkDriver' failed after 16 retries (starting from 29414)! Consider explicitly setting the appropriate port for the service 'sparkDriver' (for example spark.ui.port for SparkUI) to an available port or increasing spark.port.maxRetries.
    at sun.nio.ch.Net.bind0(Native Method)
    at sun.nio.ch.Net.bind(Net.java:433)
    at sun.nio.ch.Net.bind(Net.java:425)
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:220)
    at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:128)
    at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:558)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1283)
    at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:501)
    at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:486)
    at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:989)
    at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:254)
    at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:364)
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
    at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
    at java.lang.Thread.run(Thread.java:748)

</code></pre>
<p>As far I understand I am trying to run my spark cluster in client node, the jupyter pod acts as a master node and creates the slaves, it works when I run the code inside the jupyter pod but when other pod tries to connect it can.</p>
<p>How could I solve this?</p>

J.C Guzman
<p>I had a similiar issue and I was at the end creating the services needed for the client pod manually. In my case I wanted to deploy the spark-thrift server which didn't support cluster mode.</p>
<p>First of all you need to create the service needed for the spark blockManager and the driver itself</p>
<pre><code>apiVersion: v1
kind: Service
metadata:
  name: spark-thrift
spec:
  type: ClusterIP
  ports:
    - protocol: TCP
      port: 4000
      name: driver
    - protocol: TCP
      port: 4001
      name: block-manager
</code></pre>
<p>Now you can start your driver something like that:</p>
<pre><code>apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: spark-thrift
  labels:
    app: spark-thrift
spec:
  template:
    spec:
      containers:
        - name: spark-thrift-driver
          image: image:latest
          command:
            - /opt/spark/bin/spark-submit
          args:
            - &quot;--name&quot;
            - &quot;spark-thrift&quot;
            - &quot;--class&quot;
            - &quot;org.apache.spark.sql.hive.thriftserver.HiveThriftServer2&quot;
            - &quot;--conf&quot;
            - &quot;spark.driver.port=4000&quot;
            - &quot;--conf&quot;
            - &quot;spark.driver.host=spark-thrift&quot;
            - &quot;--conf&quot;
            - &quot;spark.driver.bindAddress=0.0.0.0&quot;
            - &quot;--conf&quot;
            - &quot;spark.driver.blockManager.port=4001&quot;
          imagePullPolicy: Always
          ports:
            - name: driver
              containerPort: 4000
            - name: blockmanager
              containerPort: 4001
</code></pre>
<p>The important arguments here are</p>
<ul>
<li>&quot;spark.driver.host=spark-thrift&quot; - points to the host (so the service name)</li>
<li>&quot;spark.driver.port=4000&quot; - points to the driver port</li>
<li>&quot;spark.driver.bindAddress=0.0.0.0&quot; - this is necessary for spark not to use spark-thrift as a hostname which would result in an error</li>
<li>&quot;spark.driver.blockManager.port=4001&quot; - points to the blockManager port</li>
</ul>
<p>obviously this is not a complete example of a working pod you still need to configure your own ports and volumes in the spec.</p>
