<p>I want to install Apache Spark <strong>v2.4</strong> on my Kubernetes cluster, but there does not seem to be a stable helm chart for this version. An older/stable chart (for v1.5.1) exists at</p>

<p><a href="https://github.com/helm/charts/tree/master/stable/spark" rel="nofollow noreferrer">https://github.com/helm/charts/tree/master/stable/spark</a></p>

<p>How can I create/find a v2.4 chart?</p>

<p>Then: The reason for needing v2.4 is to enable client-mode, because I would like to be able to submit (PySpark/Jupyter notebook) jobs to the cluster from my laptop's dev environment. What extra steps are required to enable client-mode (including exposing the service)?</p>

<p>The closest attempt so far (but for Spark v2.0.0) that I have found, but which I haven't yet got working, is at</p>

<p><a href="https://github.com/Uninett/kubernetes-apps/tree/master/spark" rel="nofollow noreferrer">https://github.com/Uninett/kubernetes-apps/tree/master/spark</a></p>

<p>At <a href="https://github.com/phatak-dev/kubernetes-spark" rel="nofollow noreferrer">https://github.com/phatak-dev/kubernetes-spark</a> (also two years old), there is nothing about jupyter deployment.</p>

<p>Pangeo-specific: <a href="https://discourse.jupyter.org/t/spark-integration-documentation/243" rel="nofollow noreferrer">https://discourse.jupyter.org/t/spark-integration-documentation/243</a></p>

<p>SO thread: <a href="https://github.com/jupyterhub/zero-to-jupyterhub-k8s/issues/1030" rel="nofollow noreferrer">https://github.com/jupyterhub/zero-to-jupyterhub-k8s/issues/1030</a></p>

<p>I have searched for up-to-date resources on this but have found nothing that has everything in one place. I will update this question with other relevant links if and when people are able to point them out to me. Hopefully it will be possible to cobble together an answer.</p>

<p>As ever, huge thanks in advance.</p>

<p><strong>Update:</strong></p>

<p><a href="https://github.com/SnappyDataInc/spark-on-k8s" rel="nofollow noreferrer">https://github.com/SnappyDataInc/spark-on-k8s</a> for v2.2 is <em>extremely</em> easy to deploy - looks promising...</p>

<p>see <a href="https://hub.helm.sh/charts/microsoft/spark" rel="nofollow noreferrer">https://hub.helm.sh/charts/microsoft/spark</a> this is based off <a href="https://github.com/helm/charts/tree/master/stable/spark" rel="nofollow noreferrer">https://github.com/helm/charts/tree/master/stable/spark</a> and uses spark 2.4.6 with hadoop 3.1. You can check the source for this chat at <a href="https://github.com/dbanda/charts" rel="nofollow noreferrer">https://github.com/dbanda/charts</a>. The Livy service makes it easy to submit spark jobs via REST API. You can also submit jobs using Zeppelin. We made this chart as alternative way to run spark on K8s without using the spark-submit k8s mode. I hope it helps.</p>
