<p>I have a k8s cluster. Now i want to deploy a spark job on the k8s cluster and i'm wondering whether i need to install and configure spark on all the worker machines or not.</p>

semenchukou
<p>As I understand it depends which mode you will use.</p>

<p>You can use <a href="https://github.com/apache/spark/blob/master/docs/running-on-kubernetes.md#cluster-mode" rel="nofollow noreferrer"><code>Cluster Mode</code></a> to launch Spark Pi .</p>

<pre><code>$ ./bin/spark-submit \
    --master k8s://https://&lt;k8s-apiserver-host&gt;:&lt;k8s-apiserver-port&gt; \
    --deploy-mode cluster \
    --name spark-pi \
    --class org.apache.spark.examples.SparkPi \
    --conf spark.executor.instances=5 \
    --conf spark.kubernetes.container.image=&lt;spark-image&gt; \
    local:///path/to/examples.jar

</code></pre>

<blockquote>
  <p>The Spark master, specified either via passing the <code>--master</code> command line argument to <code>spark-submit</code> or by setting <code>spark.master</code> in the application's configuration, must be a URL with the format <code>k8s://&lt;api_server_host&gt;:&lt;k8s-apiserver-port&gt;</code>. The port must always be specified, even if it's the HTTPS port 443. Prefixing the master string with <code>k8s://</code> will cause the Spark application to launch on the Kubernetes cluster, with the API server being contacted at <code>api_server_url</code>. If no HTTP protocol is specified in the URL, it defaults to <code>https</code>. For example, setting the master to <code>k8s://example.com:443</code> is equivalent to setting it to <code>k8s://https://example.com:443</code>, but to connect without TLS on a different port, the master would be set to <code>k8s://http://example.com:8080</code>.</p>
  
  <p>In Kubernetes mode, the Spark application name that is specified by <code>spark.app.name</code> or the <code>--name</code> argument to <code>spark-submit</code> is used by default to name the Kubernetes resources created like drivers and executors. So, application names must consist of lower case alphanumeric characters, <code>-</code>, and <code>.</code> and must start and end with an alphanumeric character.</p>
</blockquote>

<p>You can also setup a <a href="https://github.com/apache/spark/blob/master/docs/running-on-kubernetes.md#client-mode" rel="nofollow noreferrer"><code>Client Mode</code></a></p>

<blockquote>
  <p>Starting with Spark 2.4.0, it is possible to run Spark applications on Kubernetes in client mode. When your application runs in client mode, the driver can run inside a pod or on a physical host. When running an application in client mode, it is recommended to account for the following factors:</p>
  
  <h3>Client Mode Networking</h3>
  
  <p>Spark executors must be able to connect to the Spark driver over a hostname and a port that is routable from the Spark executors. The specific network configuration that will be required for Spark to work in client mode will vary per setup. If you run your driver inside a Kubernetes pod, you can use a  <a href="https://kubernetes.io/docs/concepts/services-networking/service/#headless-services" rel="nofollow noreferrer">headless service</a>  to allow your driver pod to be routable from the executors by a stable hostname. When deploying your headless service, ensure that the service's label selector will only match the driver pod and no other pods; it is recommended to assign your driver pod a sufficiently unique label and to use that label in the label selector of the headless service. Specify the driver's hostname via  <code>spark.driver.host</code>  and your spark driver's port to  <code>spark.driver.port</code>.
  ...</p>
</blockquote>

<p>The whole documentation regarding running Spark on Kubernetes is available <a href="https://github.com/apache/spark/blob/master/docs/running-on-kubernetes.md" rel="nofollow noreferrer">here</a>.</p>

<p>There is also a nice explanation given by <a href="https://www.gigaspaces.com/" rel="nofollow noreferrer">Gigaspaces</a> about <a href="https://docs.gigaspaces.com/latest/admin/kubernetes-spark-job.html" rel="nofollow noreferrer">Running a Spark Job in Kubernetes</a></p>
