<p>I am using Spark <code>3.1.2</code> and have created a cluster with 4 executors each with 15 cores.</p>
<p>My total number of partitions therefore should be 60, yet only 30 are assigned.</p>
<p>The job starts as follows, requesting 4 executors</p>
<pre><code>21/12/23 23:51:11 DEBUG ExecutorPodsAllocator: Set total expected execs to {0=4}
</code></pre>
<p>A few mins later, it is still waiting for them</p>
<pre><code>21/12/23 23:53:13 DEBUG ExecutorPodsAllocator: ResourceProfile Id: 0 pod allocation status: 0 running, 4 unknown pending, 0 scheduler backend known pending, 0 unknown newly created, 0 scheduler backend known newly created.
21/12/23 23:53:13 DEBUG ExecutorPodsAllocator: Still waiting for 4 executors for ResourceProfile Id 0 before requesting more.
</code></pre>
<p>then finally 2 come up</p>
<pre><code>21/12/23 23:53:14 DEBUG ExecutorPodsWatchSnapshotSource: Received executor pod update for pod named io-getspectrum-data-acquisition-modelscoringprocessor-8b92877de9b4ab13-exec-1, action MODIFIED
21/12/23 23:53:14 DEBUG ExecutorPodsWatchSnapshotSource: Received executor pod update for pod named io-getspectrum-data-acquisition-modelscoringprocessor-8b92877de9b4ab13-exec-3, action MODIFIED
21/12/23 23:53:15 DEBUG ExecutorPodsAllocator: ResourceProfile Id: 0 pod allocation status: 2 running, 2 unknown pending, 0 scheduler backend known pending, 0 unknown newly created, 0 scheduler backend known newly created.
</code></pre>
<p>then a third</p>
<pre><code>21/12/23 23:53:17 DEBUG ExecutorPodsWatchSnapshotSource: Received executor pod update for pod named io-getspectrum-data-acquisition-modelscoringprocessor-8b92877de9b4ab13-exec-2, action MODIFIED
21/12/23 23:53:18 DEBUG ExecutorPodsAllocator: ResourceProfile Id: 0 pod allocation status: 3 running, 1 unknown pending, 0 scheduler backend known pending, 0 unknown newly created, 0 scheduler backend known newly created.
</code></pre>
<p>...and then finally the job proceeds</p>
<pre><code>21/12/23 23:53:30 DEBUG KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Launching task 0 on executor id: 1 hostname: 10.128.35.137.
21/12/23 23:53:33 INFO MyProcessor: Calculated partitions are read 45 write 1
</code></pre>
<p>I don't understand why it suddenly decides to proceed when we have 3 executors as opposed to waiting for the 4th.</p>
<p>I have gone through the Spark and Spark K8s configs I don't see an appropriate config to influence this behavior</p>
<p>Why does it proceed when we have 3 executors?</p>

<p>Per <a href="https://spark.apache.org/docs/latest/configuration.html#scheduling" rel="nofollow noreferrer">Spark docs</a>, scheduling is controlled by these settings</p>
<blockquote>
<p><code>spark.scheduler.maxRegisteredResourcesWaitingTime</code><br>default=30s<br>
Maximum amount
of time to wait for resources to register before scheduling
begins. <br><br>
<code>spark.scheduler.minRegisteredResourcesRatio</code><br> default=0.8 for
KUBERNETES mode; 0.8 for YARN mode; 0.0 for standalone mode and Mesos
coarse-grained mode<br>   The minimum ratio of registered resources
(registered resources / total expected resources) (resources are
executors in yarn mode and Kubernetes mode, CPU cores in standalone
mode and Mesos coarse-grained mode ['spark.cores.max' value is total
expected resources for Mesos coarse-grained mode] ) to wait for before
scheduling begins. Specified as a double between 0.0 and 1.0.
Regardless of whether the minimum ratio of resources has been reached,
the maximum amount of time it will wait before scheduling begins is
controlled by config
spark.scheduler.maxRegisteredResourcesWaitingTime.</p>
</blockquote>
<p>In your case, looks like the <code>WaitingTime</code> has been reached.</p>
