<p>I am trying to use spark on Kubernetes cluster (existing setup: emr + yarn). Our use case is to handle too many jobs including short lived ones (few seconds to 15 minutes). Also, we have peak hours when many workers needs to run to handle 100s of jobs running concurrently.</p>
<p>So what I want to achieve, running master and fixed few workers (say 5) all time and increase workers to 40-50 at peak time. Also, I will prefer to use dynamic allocation.</p>
<p>I am setting it as below</p>
<p>Master image (spark-master:X)</p>
<pre><code>    FROM &lt;BASE spark 3.1 Image build using dev/make-distribution.sh -Pkubernetes in spark&gt;
    ENTRYPOINT [&quot;/opt/spark/sbin/start-master.sh&quot;, &quot;-p&quot;, &quot;8081&quot;, &quot;&lt;A long running server command that can accept get traffic on 8080 to submit jobs&gt;&quot;]
</code></pre>
<p>Worker worker image (spark-worker:X)</p>
<pre><code>    FROM &lt;BASE spark 3.1 Image build using dev/make-distribution.sh -Pkubernetes in spark&gt;
    ENTRYPOINT [&quot;/opt/spark/sbin/start-worker.sh&quot;, &quot;spark//spark-master:8081&quot; ,&quot;-p&quot;, &quot;8081&quot;, &quot;&lt;A long running server command to keep up the worker&gt;&quot;]
</code></pre>
<p>Deplyments</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: spark-master-server
spec:
  replicas: 1
  selector:
    matchLabels:
      component: spark-master-server
  template:
    metadata:
      labels:
        component: spark-master-server
    spec:
      containers:
        - name:  spark-master-server
          image: spark-master:X
          imagePullPolicy: IfNotPresent 
          ports:
            - containerPort: 8081
---
apiVersion: v1
kind: Service
metadata:
  name: spark-master

spec:
  type: ClusterIP
  ports:
    - port: 8081
      targetPort: 8081
  selector:
    component: spark-master-server
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: spark-worker-instance
spec:
  replicas: 3
  selector:
    matchLabels:
      component: spark-worker-instance
  template:
    metadata:
      labels:
        component: spark-worker-instance
    spec:
      containers:
        - name:  spark-worker-server
          image: spark-worker:X
          imagePullPolicy: IfNotPresent 
          ports:
            - containerPort: 8081
</code></pre>
<p>Questions</p>
<ul>
<li>Is this setup recommended ?</li>
<li>How can we submit new job from within Kubernetes cluster in absence of yarn (k8s) ?</li>
</ul>
<p>Reason we are trying not create master and driver dynamically per job (as given in example - <a href="http://spark.apache.org/docs/latest/running-on-kubernetes.html" rel="nofollow noreferrer">http://spark.apache.org/docs/latest/running-on-kubernetes.html</a>) is that it may be an overhead for large no. of small jobs.</p>

<blockquote>
<p>Is this setup recommended ?</p>
</blockquote>
<p>Don't think so.</p>
<hr />
<p><a href="http://spark.apache.org/docs/latest/job-scheduling.html#dynamic-resource-allocation" rel="nofollow noreferrer">Dynamic Resource Allocation</a> is a property of a single Spark application <em>&quot;to dynamically adjust the resources your application occupies based on the workload.&quot;</em></p>
<p>Dynamic Resource Allocation spans its resource requirements regardless of available nodes in a cluster. As long as there are resources available and a cluster manager could assign them to a Spark application these resources are free to go.</p>
<p>What you seem to be trying to set up is how to scale the cluster itself up and down. In your case it's Spark Standalone and although technically it's possible with ReplicaSets (just a guess) I've never heard any earlier attempts at it. You're on your own as Spark Standalone does not support it out of the box.</p>
<hr />
<p>That I think is an overkill since you're building a multi-layer cluster environment: using a cluster manager (Kubernetes) to host another cluster manager (Spark Standalone) for Spark applications. Given Spark on Kubernetes supports Dynamic Allocation out of the box the only worry of yours should simply be how to &quot;throw in&quot; more CPU and memory on demand while resizing Kubernetes cluster. You should rely on the capabilities of Kubernetes to resize itself up and down rather than Spark Standalone on Kubernetes.</p>
