title: Running Spark on Kubernetes with Dynamic Allocation
body: <p>I am trying to use spark on Kubernetes cluster (existing setup: emr + yarn). Our use case is to handle too many jobs including short lived ones (few seconds to 15 minutes). Also, we have peak hours when many workers needs to run to handle 100s of jobs running concurrently.</p>
  <p>So what I want to achieve, running master and fixed few workers (say 5) all time and increase workers to 40-50 at peak time. Also, I will prefer to use dynamic allocation.</p>
  <p>I am setting it as below</p>
  <p>Master image (spark-master:X)</p>
  <pre><code>    FROM &lt;BASE spark 3.1 Image build using dev/make-distribution.sh -Pkubernetes in spark&gt;
      ENTRYPOINT [&quot;/opt/spark/sbin/start-master.sh&quot;, &quot;-p&quot;, &quot;8081&quot;, &quot;&lt;A long running server command that can accept get traffic on 8080 to submit jobs&gt;&quot;]
  </code></pre>
  <p>Worker worker image (spark-worker:X)</p>
  <pre><code>    FROM &lt;BASE spark 3.1 Image build using dev/make-distribution.sh -Pkubernetes in spark&gt;
      ENTRYPOINT [&quot;/opt/spark/sbin/start-worker.sh&quot;, &quot;spark//spark-master:8081&quot; ,&quot;-p&quot;, &quot;8081&quot;, &quot;&lt;A long running server command to keep up the worker&gt;&quot;]
  </code></pre>
  <p>Deplyments</p>
  <pre><code>apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: spark-master-server
  spec:
    replicas: 1
    selector:
      matchLabels:
        component: spark-master-server
    template:
      metadata:
        labels:
          component: spark-master-server
      spec:
        containers:
          - name:  spark-master-server
            image: spark-master:X
            imagePullPolicy: IfNotPresent 
            ports:
              - containerPort: 8081
  ---
  apiVersion: v1
  kind: Service
  metadata:
    name: spark-master
  
  spec:
    type: ClusterIP
    ports:
      - port: 8081
        targetPort: 8081
    selector:
      component: spark-master-server
  ---
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: spark-worker-instance
  spec:
    replicas: 3
    selector:
      matchLabels:
        component: spark-worker-instance
    template:
      metadata:
        labels:
          component: spark-worker-instance
      spec:
        containers:
          - name:  spark-worker-server
            image: spark-worker:X
            imagePullPolicy: IfNotPresent 
            ports:
              - containerPort: 8081
  </code></pre>
  <p>Questions</p>
  <ul>
  <li>Is this setup recommended ?</li>
  <li>How can we submit new job from within Kubernetes cluster in absence of yarn (k8s) ?</li>
  </ul>
  <p>Reason we are trying not create master and driver dynamically per job (as given in example - <a href="http://spark.apache.org/docs/latest/running-on-kubernetes.html" rel="nofollow noreferrer">http://spark.apache.org/docs/latest/running-on-kubernetes.html</a>) is that it may be an overhead for large no. of small jobs.</p>