<p>I have a very simple Spark job, but I can't get it to work in Kubernetes. The error I get is:</p>

<pre><code>&gt;     19/10/03 14:59:51 WARN DependencyUtils: Local jar /opt/spark/work-dir/target/scala-2.11/ScalaTest-assembly-1.0.jar does
&gt; not exist, skipping.
&gt;     19/10/03 14:59:51 WARN SparkSubmit$$anon$2: Failed to load ScalaTest.
&gt;     java.lang.ClassNotFoundException: ScalaTest
&gt;       at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
&gt;       at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
&gt;       at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
&gt;       at java.lang.Class.forName0(Native Method)
&gt;       at java.lang.Class.forName(Class.java:348)
&gt;       at org.apache.spark.util.Utils$.classForName(Utils.scala:238)
&gt;       at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:806)
&gt;       at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
&gt;       at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
&gt;       at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
&gt;       at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
&gt;       at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
&gt;       at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
</code></pre>

<p><strong>Project structure:</strong></p>

<pre><code>project/build.properties
project/plugins.sbt
src/main/scala/ScalaTest.scala
Dockerfile
build.sbt
</code></pre>

<p><strong><code>build.properties</code></strong></p>

<pre><code>sbt.version=1.2.8
</code></pre>

<p><strong><code>plugins.sbt</code></strong></p>

<pre><code>addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.6")
addSbtPlugin("net.virtual-void" % "sbt-dependency-graph" % "0.10.0-RC1")
</code></pre>

<p><strong><code>ScalaTest.scala</code></strong></p>

<pre><code>import org.apache.spark.sql.SparkSession
import org.apache.spark.SparkContext

object ScalaTest {
  def main(args: Array[String]) {
    val spark = SparkSession.builder.appName("ScalaTest").config("spark.master", "local[*]").getOrCreate()

    import spark.implicits._

    println("hello")

  }
}
</code></pre>

<p><strong><code>Dockerfile</code></strong> This is just a wrapper image based on the one built from the kubernetes folder within Spark binaries. Before building this image I make sure I run <strong><code>sbt assembly</code></strong>, which generates the Uber jar.</p>

<pre><code>FROM spark:latest

WORKDIR /opt/spark/work-dir

COPY target/scala-2.11/ScalaTest-assembly-1.0.jar target/scala-2.11/ScalaTest-assembly-1.0.jar
</code></pre>

<p><strong><code>build.sbt</code></strong></p>

<pre><code>name := "ScalaTest"

version := "1.0"

scalaVersion := "2.11.12"

val sparkVersion = "2.4.4"

libraryDependencies ++= Seq(
    "org.apache.spark" % "spark-core_2.11" % sparkVersion % "provided",
    "org.apache.spark" % "spark-sql_2.11" % sparkVersion % "provided"
)
</code></pre>

<p>Finally my <strong><code>spark-submit</code></strong>. Before I do this I pushed the image to ECR's registry, so EKS can pull that image. I also point to the location of the uber jar <strong>within</strong> my image.</p>

<pre><code>~/spark-2.4.4-bin-hadoop2.7/bin/spark-submit \
    --master k8s://{K8S_ENDPOINT}:443 \
    --deploy-mode cluster \
    --name test-job \
    --conf spark.kubernetes.container.image={ECR_IMAGE}:latest \
    --conf spark.kubernetes.submission.waitAppCompletion=false \
    --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
    --conf spark.kubernetes.driver.pod.name=test-job \
    --class ScalaTest \
    local:///opt/spark/work-dir/target/scala-2.11/ScalaTest-assembly-1.0.jar
</code></pre>

<p>Also note, when I run the command below (spark-submit within my container, locally) it works as expected:</p>

<pre><code>docker run --rm -it my-custom-image ../bin/spark-submit target/scala-2.11/ScalaTest-assembly-1.0.jar
</code></pre>

<p><strong>UPDATE</strong>
Inspecting the assembled uber jar, I can see ScalaTest's class is there.</p>

<pre><code>jar tf target/scala-2.11/ScalaTest-assembly-1.0.jar

...
ScalaTest$.class
ScalaTest.class
...
</code></pre>

<p>The solution to this is to leave the jar not in the working dir but in the jars folders. I haven't looked at the docs but probably this is an environment variable that can be changed. In any case, the Dockerfile should look like this:</p>

<pre><code>FROM spark:latest

COPY target/scala-2.11/ScalaTest-assembly-1.0.jar /ops/spark/jars/ScalaTest-assembly-1.0.jar
</code></pre>

<p>And then change the <code>spark-submit</code> accordingly:</p>

<pre><code>~/spark-2.4.4-bin-hadoop2.7/bin/spark-submit \
    --master k8s://{K8S_ENDPOINT}:443 \
    --deploy-mode cluster \
    --name test-job \
    --conf spark.kubernetes.container.image={ECR_IMAGE}:latest \
    --conf spark.kubernetes.submission.waitAppCompletion=false \
    --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
    --conf spark.kubernetes.driver.pod.name=test-job \
    --class ScalaTest \
    local:///opt/spark/jars/ScalaTest-assembly-1.0.jar
</code></pre>
