<p>I am running a Spark Job in Kubernetes cluster using spark-submit command as below,</p>

<pre><code>bin/spark-submit \
    --master k8s://https://api-server-host:443 \
    --deploy-mode cluster \
    --name spark-job-name \
    --conf spark.kubernetes.namespace=spark \
    --conf spark.kubernetes.container.image=docker-repo/pyspark:55 \
    --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark-submit \
    --conf spark.kubernetes.pyspark.pythonVersion=3 \
    --conf spark.executor.memory=4G \
    --files local:///mnt/conf.json \
    local:///mnt/ingest.py
</code></pre>

<p>and when I check the request and limit for the executor pod, it shows below. There is almost 1700 MB extra got allocated for the pod.</p>

<pre><code>Limits:
  memory:  5734Mi
Requests:
  cpu:     4
  memory:  5734Mi
</code></pre>

<p>Why is that?</p>

karthikeayan
<p>In addition to @CptDolphin 's answer, be aware that Spark <em>always</em> allocates <a href="https://spark.apache.org/docs/latest/configuration.html#application-properties" rel="nofollow noreferrer"><code>spark.executor.memoryOverhead</code></a> extra memory (max of 10% of <code>spark.executor.memory</code> or 384MB, unless explicitly configured), and <em>may</em> allocate additional <code>spark.executor.pyspark.memory</code> if you defined that in your configuration.</p>
