<p>I tried to use K8s to setup spark cluster (I use standalone deployment mode, and I cannot use k8s deployment mode for some reason)</p>

<p>I didn't set any cpu related arguments.</p>

<p>for spark, that means:</p>

<blockquote>
  <p>Total CPU cores to allow Spark applications to use on the machine (default: all available); only on worker</p>
  
  <p><a href="http://spark.apache.org/docs/latest/spark-standalone.html" rel="nofollow noreferrer">http://spark.apache.org/docs/latest/spark-standalone.html</a></p>
</blockquote>

<p>for k8s pods, that means:</p>

<blockquote>
  <p>If you do not specify a CPU limit for a Container, then one of these situations applies:</p>
  
  <ul>
  <li><p>The Container has no upper bound on the CPU resources it can use. The Container could use all of the CPU resources available on the Node where it is running.</p></li>
  <li><p>The Container is running in a namespace that has a default CPU limit, and the Container is automatically assigned the default limit. Cluster administrators can use a LimitRange to specify a default value for the CPU limit.</p></li>
  </ul>
  
  <p><a href="https://kubernetes.io/docs/tasks/configure-pod-container/assign-cpu-resource/" rel="nofollow noreferrer">https://kubernetes.io/docs/tasks/configure-pod-container/assign-cpu-resource/</a></p>
</blockquote>

<pre><code>...
Addresses:
  InternalIP:  172.16.197.133
  Hostname:    ubuntu
Capacity:
 cpu:     4
 memory:  3922Mi
 pods:    110
Allocatable:
 cpu:     4
 memory:  3822Mi
 pods:    110
...
</code></pre>

<p>But my spark worker only use 1 core (I have 4 cores on the worker node and the namespace has no resource limits).</p>

<p>That means the spark worker pod only used 1 core of the node (which should be 4).</p>

<p>How can I write yaml file to set the pod to use all available cpu cores?</p>

<p>Here is my yaml file:</p>

<pre><code>---

apiVersion: v1
kind: Namespace
metadata:
  name: spark-standalone

---

kind: DaemonSet
apiVersion: apps/v1
metadata:
  name: spark-slave
  namespace: spark-standalone
  labels:
    k8s-app: spark-slave
spec:
  selector:
    matchLabels:
      k8s-app: spark-slave
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
  template:
    metadata:
      name: spark-slave
      namespace: spark-standalone
      labels:
        k8s-app: spark-slave
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: node-role.kubernetes.io/edge
                operator: Exists
      hostNetwork: true
      containers:
        - name: spark-slave
          image: spark:2.4.3
          command: ["/bin/sh","-c"]
          args: 
            - "
              ${SPARK_HOME}/sbin/start-slave.sh
              spark://$(SPARK_MASTER_IP):$(SPARK_MASTER_PORT)
              --webui-port $(SPARK_SLAVE_WEBUI_PORT)
              &amp;&amp;
              tail -f ${SPARK_HOME}/logs/*
              "
          env:
            - name: SPARK_MASTER_IP
              value: "10.4.20.34"
            - name: SPARK_MASTER_PORT
              value: "7077"
            - name: SPARK_SLAVE_WEBUI_PORT
              value: "8081"

---
</code></pre>

Green
<h2>Kubernetes - No upper bound</h2>

<blockquote>
  <p>The Container has no upper bound on the CPU resources it can use. The Container could use all of the CPU resources available on the Node where it is running</p>
</blockquote>

<p>Unless you confgure a <code>limit</code> on CPU for your pod, it <em>can</em> use all available CPU resources on the node.</p>

<p><strong>Consider dedicated nodes</strong></p>

<p>If you are running other workload on the same node, they also consume CPU resources, and may be guaranteed CPU resources if they have configured <code>request</code> for CPU. Consider use a dedicated node for your workload using <code>NodeSelector</code> and <a href="https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/" rel="nofollow noreferrer">Taints and Tolerations</a>.</p>

<h2>Spark - No upper bound</h2>

<p>You <a href="https://spark.apache.org/docs/latest/spark-standalone.html" rel="nofollow noreferrer">configure the slave</a> with parameters to the <code>start-slave.sh</code> e.g. <code>--cores X</code> to <em>limit</em> CPU core usage.</p>

<blockquote>
  <p>Total CPU cores to allow Spark applications to use on the machine (<strong>default: all available</strong>); only on worker</p>
</blockquote>

<h2>Multithreaded workload</h2>

<p>In the end, if pod can use multiple CPU cores depends on how your application uses threads. Some things only uses a single thread, so the application must be designed for <strong>multithreading</strong> and have something <strong>parallelized</strong> to do.</p>
