<p>There are two ways to run spark application on Kubernetes.</p>
<ol>
<li>Use spark-submit with <code>k8s://</code> master URL.</li>
<li>Use <a href="https://github.com/GoogleCloudPlatform/spark-on-k8s-operator" rel="nofollow noreferrer">spark-on-k8s-operator</a>. With the benefit of CRD in spark operator, we could have declarative management of spark pods using configuration files (YAML).</li>
</ol>
<p>How do I choose between them in production that running 1000+ spark workloads on Kubernetes?</p>
<p>Any pros and cons of using a spark operator?</p>

Kevin Su
<p>Pros of <code>spark-on-k8s-operator</code>:</p>
<ul>
<li>creating a k8s deployment of the Spark Operator will spawn pods of <code>SparkApplication</code> type and spawn driver and executor pods.</li>
<li>Includes a Prometheus metrics exporter</li>
<li>Simplifies application development and management to a YAML file   * It can be scheduled using <code>ScheduledSparkApplication</code></li>
</ul>
<p>Cons of <code>spark-submit</code></p>
<ul>
<li>creates naked pods, more <a href="https://kubernetes.io/docs/concepts/configuration/overview/#naked-pods-vs-replicasets-deployments-and-jobs" rel="nofollow noreferrer">here</a></li>
<li>configurations able to be scattered throughout:
<code>spark-submit</code>, <code>SessionBuilder</code>, Scala or python code, and k8s.</li>
<li>requires additional scheduler</li>
</ul>
<p>TL;DR - Use <code>spark-on-k8s-operator</code></p>
