<p>I am trying to run jupyter for spark on k8s. </p>

<p>After little experiment :) 
I found that </p>

<p>k8s does not support client mode </p>

<pre><code>$ bin/spark-shell --master k8s://192.168.99.100:8443
...
...
Error: Client mode is currently not supported for Kubernetes.
</code></pre>

<p>And that</p>

<p>spark-shell can't run in cluster mode </p>

<pre><code>$ bin/spark-shell --master k8s://192.168.99.100:8443 --deploy-mode cluster
...
...
Error: Cluster deploy mode is not applicable to Spark shells.
</code></pre>

<p>So I am guessing that <em>jupyter spark kernel</em> or <em>zeppelin</em> will not run for k8s. (not sure) </p>

<p>As a workaround I am thinking about using livy. </p>

<p>Does apache livy 0.5.0 supports spark on k8s?
Any advice for setting up spark cluster would be appreciated.</p>

<p>P.S. I tried to use GCP dataproc but after lost GCS shared note once (during 7 days period evaluation, no clue about reason). I am sure it's my fault (maybe on configuration?) but didn't dive deep to dive deep in k8s. :)</p>

ruseel
<p>You can try customized livy-0.6.0 for that: <a href="https://github.com/apache/incubator-livy/pull/167" rel="nofollow noreferrer">https://github.com/apache/incubator-livy/pull/167</a>
We've required a solution for the project to run Spark on Kubernetes and modified Livy to serve that purpose.
You can be interested in Helm chart for Livy setup either: <a href="https://github.com/jahstreet/spark-on-kubernetes-helm" rel="nofollow noreferrer">https://github.com/jahstreet/spark-on-kubernetes-helm</a></p>
