<p>I have currently set up the spark standalone cluster in k8s using helm. It has a fixed number of workers (5 workers) and a single master. Currently doing spark-submit as below</p>
<pre><code>spark-submit --class LogParser.LogBundleConfigFetcher --conf spark.submit.deployMode=cluster --conf spark.network.timeout=300 --conf spark.scheduler.mode=FAIR --conf **spark.master=spark://sm-care-alice-staging:7077** --conf spark.executor.cores=5 --conf spark.executor.memory=20g --conf spark.dynamicAllocation.maxExecutors=3 --conf spark.driver.memory=16g --conf spark.dynamicAllocation.enabled=false --conf spark.cores.max=15 http://minio.platform.svc.cluster.local:9000/alice-care/staging/config/spark/aliceparser.jar
</code></pre>
<p>But it has some limitations like we can not use cluster mode for pyspark- <strong>Cluster deploy mode is currently not supported for python applications on standalone clusters</strong></p>
<p>So, I would like to use k8s as a cluster manager. But we have a huge k8s cluster and limited resources are allocated to each project (assigned by namespaces). Using k8s as cluster manager would spawn the pods dynamically so we can't control the resources (I know we can control the instances but in case of multiple spark-submit we have to restrict it to a limit of cores and memory).</p>
<blockquote>
</blockquote>
<pre><code>spark-submit \
    --master k8s://https://$(k8s_ip):8443 \
    --deploy-mode cluster \
    --name spark-example \
    --conf spark.executor.instances=5 \
    --conf spark.kubernetes.container.image=pyspark-k8s-example:2.4.1 \
    --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
    --conf spark.kubernetes.pyspark.pythonVersion=3 \
    /usr/bin/run.py
</code></pre>
<p>I would like to know how can we control the k8s cluster manager to limit the resources i.e limit the cores to 100 irrespective of how many spark-submit being called.</p>
<p>Or is there any other way that can run pyspark in cluster mode in the same setup?</p>

<p>You can use <a href="https://kubernetes.io/docs/concepts/policy/resource-quotas/#compute-resource-quota" rel="nofollow noreferrer">compute resource quota</a> to limit how much cpu and memory can be consumed by pods running in a namespace.</p>
<ul>
<li><code>limits.cpu</code> : Across all pods in a non-terminal state, the sum of CPU
limits cannot exceed this value.</li>
<li><code>limits.memory</code> : Across all pods in a
non-terminal state, the sum of memory limits cannot exceed this
value.</li>
<li><code>requests.cpu</code>   : Across all pods in a non-terminal state, the sum
of CPU requests cannot exceed this value.</li>
<li><code>requests.memory</code> : Across all
pods in a non-terminal state, the sum of memory requests cannot
exceed this value.</li>
</ul>
<hr />
<pre><code>apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute-resources
  namespace: spark-ns
spec:
  hard:
    requests.cpu: &quot;1&quot;
    requests.memory: 1Gi
    limits.cpu: &quot;2&quot;
    limits.memory: 2Gi
</code></pre>
