<p>I used spark on Bitnami (<a href="https://bitnami.com/stack/spark/helm" rel="nofollow noreferrer">https://bitnami.com/stack/spark/helm</a>) and I was able to port my current yarn application to this. This uses (or I think it uses) the default k8s scheduler to schedule the pods.</p>
<p>However, I see schedulers like yunikorn (<a href="https://yunikorn.apache.org" rel="nofollow noreferrer">https://yunikorn.apache.org</a>) and volcano (volcano.sh) which say they provide schedulers for spark.</p>
<p>What I am not clear about is --the pros and cons of using k8s default scheduler for spark. I searched the web, but couldn't get an answer. What makes somebody not use k8s scheduler but use volcano.sh/ yunikorn? What use cases does k8s default scheduler not solve?</p>

chendu
<p>Originally Spark used Yarn scheduler to manage the underlaying cluster and schedule jobs on top of it. In the last period YARN is replaced more and more ofter with Kubernetes that fits much better into cloud and distributed environments, usually based on containers, and overcome YARN in terms of containers management, multi-tenancy isolation, cluster elasticity, dependencies managements, flexibility to connect multi-cloud infrastructures, etc.</p>
<p>On the other side, probably, if you don't need to have multi-cloud configuration or if you don't have a containeraized application, YARN is a better choice.</p>
<p>In the meantime, other scheduler has come up, like Yunikorn. This scheduler can be seen as an higher layer of Kubernetes/Yarn schedulers, able to speak with both with a central management console.</p>
<pre><code>      Yuvicorn
      /       \
    YARN  |   K8s
  -----------------
 one or more cluster
</code></pre>
<p>Yuvikorn scheduler collects resources from underneath resource management platforms (like YARN/K8s), and is responsible for container allocation requests. To give some usecase examples, you can use it if you need to manage a large cluster with multiple schedulers or if you have multiple resource managers to run on the same cluster and you need to handle those managers.</p>
