<p>I have been trying to deploy spark and jupyter note on minikube.  I used helm charts for deploying both</p>
<p>Jupyter notebook - <a href="https://artifacthub.io/packages/helm/pyspark-notebook-helm/pyspark-notebook" rel="nofollow noreferrer">https://artifacthub.io/packages/helm/pyspark-notebook-helm/pyspark-notebook</a></p>
<p>Spark - <a href="https://bitnami.com/stack/spark/helm" rel="nofollow noreferrer">https://bitnami.com/stack/spark/helm</a></p>
<p>While able to establish to the master using</p>
<pre><code>spark = SparkSession.builder.master(&quot;spark://my-release-spark-master-0.my-release-spark-headless.default.svc.cluster.local:7077&quot;).getOrCreate() 
</code></pre>
<p>When running the following snippet</p>
<pre><code>nums= sc.parallelize([1,2,3,4])
squared = nums.map(lambda x: x*x).collect()
for num in squared:
    print('%i ' % (num))
</code></pre>
<p>The execution takes a long time and never completes when it runs the collect() method</p>

You can check the <a href="https://spark.apache.org/docs/latest/running-on-kubernetes.html#prerequisites" rel="nofollow noreferrer">prerequisites</a> when running Spark in Kubernetes to determine if it will increase its performance</p>


A running Kubernetes cluster at version &gt;= 1.22 with access configured to it using <a href="https://kubernetes.io/docs/user-guide/prereqs/" rel="nofollow noreferrer">kubectl</a>. If you do not already have a working Kubernetes cluster, you may set up a test cluster on your local machine using <a href="https://kubernetes.io/docs/getting-started-guides/minikube/" rel="nofollow noreferrer">minikube</a>.</p>
</li>


We recommend using the latest release of minikube with the DNS addon enabled.</p>
</li>

Be aware that the default minikube configuration is not enough for running Spark applications. We recommend 3 CPUs and 4g of memory to be able to start a simple Spark application with a single executor.</p>
</li>

Check <a href="https://github.com/fabric8io/kubernetes-client" rel="nofollow noreferrer">kubernetes-client library</a>&rsquo;s version of your Spark environment, and its compatibility with your Kubernetes cluster&rsquo;s version.</p>
</li>


You must have appropriate permissions to list, create, edit and delete <a href="https://kubernetes.io/docs/user-guide/pods/" rel="nofollow noreferrer">pods</a> in your cluster. You can verify that you can list these resources by running&nbsp;</p>
</li>

 &nbsp;&nbsp;&nbsp; kubectl auth can-i &lt;list|create|edit|delete&gt; pods</p>


The service account credentials used by the driver pods must be allowed to create pods, services and configmaps.</p>
</li>

You must have <a href="https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/" rel="nofollow noreferrer">Kubernetes DNS</a> configured in your cluster.</p>
