<p>I've used the <a href="https://github.com/helm/charts/tree/master/stable/spark" rel="nofollow noreferrer">helm chart</a> to deploy Spark to Kubernetes in GCE. According to default configuration in values.yaml the Spark is deployed to the path <code>/opt/spark</code>. I've checked that Spark has deployed successfully by running <code>kubectl --namespace=my-namespace get pods -l "release=spark"</code>. There is 1 master and 3 workers running. </p>

<p>However when I've tried to check Spark version by executing <code>spark-submit --version</code> from the Google cloud console it returned <code>-bash: spark-submit: command not found</code>. </p>

<p>I've navigated to the <code>/opt</code> directory and the <code>/spark</code> folder is missing. What should I do to be able to open Spark shell Terminal and to execute Spark commands?</p>

<p>You can verify by checking service</p>

<p><code>kubectl get services -n &lt;namespace&gt;</code></p>

<p>you can port-forward particular service and try running locally to check</p>

<p><code>kubectl port-forward svc/&lt;service name&gt; &lt;external port&gt;:&lt;internal port or spark running port&gt;</code></p>

<p>Locally you can try running spark terminal it will be connected to spark running on GCE instance.</p>

<p>If you check the helm chart document there is also options for UI you can also do same to access UI via port-forward</p>

<blockquote>
  <p><strong>Access via SSH inside  pod</strong></p>
</blockquote>

<pre><code>Kubectl exec -it &lt;spark pod name&gt; -- /bin/bash
</code></pre>

<p>here you can directly run spark commands. <code>spark-submit --version</code></p>

<blockquote>
  <p><strong>Access UI</strong></p>
</blockquote>

<p>Access UI via port-forwarding if you have enable UI in helm chart.</p>

<p><code>kubectl port-forward svc/&lt;spark service name&gt; &lt;external port&gt;:&lt;internal port or spark running port&gt;</code></p>

<blockquote>
  <p><strong>External Load balancer</strong></p>
</blockquote>

<p>This particular helm chart also creating <code>External Load balancer</code> you can also get <code>External IP</code> using</p>

<p><code>Kubectl get svc -n &lt;namespace&gt;</code></p>

<blockquote>
  <p><strong>Access Shell</strong></p>
</blockquote>

<p><strong>If want to connect via LB IP &amp; port</strong></p>

<pre><code>./bin/spark-shell --conf spark.cassandra.connection.host=&lt;Load balancer IP&gt; spark.cassandra-connection.native.port=&lt;Port&gt;
</code></pre>

<p><strong>Creating connection using port-forward</strong></p>

<p><code>kubectl port-forward svc/&lt;spark service name&gt; &lt;external(local) port&gt;:&lt;internal port or spark running port&gt;</code></p>

<pre><code>./bin/spark-shell --conf spark.cassandra.connection.host=localhost spark.cassandra-connection.native.port=&lt;local Port&gt;
</code></pre>
