<p>I've seen others have been dealing with this same issue, but since none of the proposed solutions or workarounds worked for me and I've already spent hours on this, I figured I would share my specific case in detail in hope someone could point out what I'm missing.</p>
<p>I wanted to experiment with running a very simple Spark Spring-Boot application on a Minikube k8s cluster. When I run the app locally (using <code>SparkSession.builder().master(&quot;local&quot;)</code>) everything works as expected. However, when I deploy my app to minikube, I manage to get my driver pod to spin up the executor pods when the job is triggered, but then I get this exception on my executor pods:</p>
<pre><code>ERROR Executor: Exception in task 0.1 in stage 0.0 (TID 1)
cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.sql.execution.MapPartitionsExec.func of type scala.Function1 in instance of org.apache.spark.sql.execution.MapPartitionsExec
</code></pre>
<p>Here is my spring-boot app. For the sake of simplicity of sharing this, I kept all the logic on the controller:</p>
<p><strong>WordcountController</strong></p>
<pre class="lang-java prettyprint-override"><code>@RestController
public class WordCountController implements Serializable {
    @PostMapping(&quot;/wordcount&quot;)
    public ResponseEntity&lt;String&gt; handleFileUpload(@RequestParam(&quot;file&quot;) MultipartFile file) throws IOException {
        String hostIp;
        try {
            hostIp = InetAddress.getLocalHost().getHostAddress();
        } catch (UnknownHostException e) {
            throw new RuntimeException(e);
        }

        SparkConf conf = new SparkConf();
        conf.setAppName(&quot;count.words.in.file&quot;)
                .setMaster(&quot;k8s://https://kubernetes.default.svc:443&quot;)
                .setJars(new String[]{&quot;/app/wordcount.jar&quot;})
                .set(&quot;spark.driver.host&quot;, hostIp)
                .set(&quot;spark.driver.port&quot;, &quot;8080&quot;)
                .set(&quot;spark.kubernetes.namespace&quot;, &quot;default&quot;)
                .set(&quot;spark.kubernetes.container.image&quot;, &quot;spark:3.3.2h.1&quot;)
                .set(&quot;spark.executor.cores&quot;, &quot;2&quot;)
                .set(&quot;spark.executor.memory&quot;, &quot;1g&quot;)
                .set(&quot;spark.kubernetes.authenticate.executor.serviceAccountName&quot;, &quot;spark&quot;)
                .set(&quot;spark.kubernetes.dynamicAllocation.deleteGracePeriod&quot;, &quot;20&quot;)
                .set(&quot;spark.cores.max&quot;, &quot;4&quot;)
                .set(&quot;spark.executor.instances&quot;, &quot;2&quot;);

        SparkSession spark = SparkSession.builder()
                .config(conf)
                .getOrCreate();

        byte[] byteArray = file.getBytes();
        String contents = new String(byteArray, StandardCharsets.UTF_8);
        Dataset&lt;String&gt; text = spark.createDataset(Arrays.asList(contents), Encoders.STRING());

        Dataset&lt;String&gt; wordsDataset = text.flatMap((FlatMapFunction&lt;String, String&gt;) line -&gt; {
            List&lt;String&gt; words = new ArrayList&lt;&gt;();
            for (String word : line.split(&quot; &quot;)) {
                words.add(word);
            }
            return words.iterator();
        }, Encoders.STRING());

        // Count the number of occurrences of each word
        Dataset&lt;Row&gt; wordCounts = wordsDataset.groupBy(&quot;value&quot;)
                .agg(count(&quot;*&quot;).as(&quot;count&quot;))
                .orderBy(desc(&quot;count&quot;));

        // Convert the word count results to a List of Rows
        List&lt;Row&gt; wordCountsList = wordCounts.collectAsList();

        StringBuilder resultStringBuffer = new StringBuilder();

        // Build the final string representation
        for (Row row : wordCountsList) {
            resultStringBuffer.append(row.getString(0)).append(&quot;: &quot;).append(row.getLong(1)).append(&quot;\n&quot;);
        }
        return ResponseEntity.ok(resultStringBuffer.toString());

    }
</code></pre>
<p>Here is my maven <code>pom.xml</code>:</p>
<pre class="lang-xml prettyprint-override"><code>&lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt;
&lt;project xmlns=&quot;http://maven.apache.org/POM/4.0.0&quot; xmlns:xsi=&quot;http://www.w3.org/2001/XMLSchema-instance&quot;
         xsi:schemaLocation=&quot;http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd&quot;&gt;
    &lt;modelVersion&gt;4.0.0&lt;/modelVersion&gt;
    &lt;parent&gt;
        &lt;groupId&gt;org.springframework.boot&lt;/groupId&gt;
        &lt;artifactId&gt;spring-boot-starter-parent&lt;/artifactId&gt;
        &lt;version&gt;2.7.8&lt;/version&gt;
        &lt;relativePath/&gt; &lt;!-- lookup parent from repository --&gt;
    &lt;/parent&gt;
    &lt;groupId&gt;com.example&lt;/groupId&gt;
    &lt;artifactId&gt;wordcount&lt;/artifactId&gt;
    &lt;version&gt;0.0.1-SNAPSHOT&lt;/version&gt;
    &lt;name&gt;wordcount&lt;/name&gt;
    &lt;description&gt;wordcount&lt;/description&gt;
    &lt;properties&gt;
        &lt;java.version&gt;11&lt;/java.version&gt;
        &lt;spark.version&gt;3.3.2&lt;/spark.version&gt;
        &lt;scala.version&gt;2.12&lt;/scala.version&gt;
    &lt;/properties&gt;
    &lt;dependencyManagement&gt;
        &lt;dependencies&gt;
            &lt;!--Spark java.lang.NoClassDefFoundError: org/codehaus/janino/InternalCompilerException--&gt;
            &lt;dependency&gt;
                &lt;groupId&gt;org.codehaus.janino&lt;/groupId&gt;
                &lt;artifactId&gt;commons-compiler&lt;/artifactId&gt;
                &lt;version&gt;3.0.8&lt;/version&gt;
            &lt;/dependency&gt;
            &lt;dependency&gt;
                &lt;groupId&gt;org.codehaus.janino&lt;/groupId&gt;
                &lt;artifactId&gt;janino&lt;/artifactId&gt;
                &lt;version&gt;3.0.8&lt;/version&gt;
            &lt;/dependency&gt;
        &lt;/dependencies&gt;
    &lt;/dependencyManagement&gt;
    &lt;dependencies&gt;
        &lt;dependency&gt;
            &lt;groupId&gt;org.codehaus.janino&lt;/groupId&gt;
            &lt;artifactId&gt;commons-compiler&lt;/artifactId&gt;
            &lt;version&gt;3.0.8&lt;/version&gt;
        &lt;/dependency&gt;
        &lt;dependency&gt;
            &lt;groupId&gt;org.codehaus.janino&lt;/groupId&gt;
            &lt;artifactId&gt;janino&lt;/artifactId&gt;
            &lt;version&gt;3.0.8&lt;/version&gt;
        &lt;/dependency&gt;
        &lt;dependency&gt;
            &lt;groupId&gt;org.springframework.boot&lt;/groupId&gt;
            &lt;artifactId&gt;spring-boot-starter-web&lt;/artifactId&gt;
        &lt;/dependency&gt;

        &lt;dependency&gt;
            &lt;groupId&gt;org.apache.spark&lt;/groupId&gt;
            &lt;artifactId&gt;spark-core_${scala.version}&lt;/artifactId&gt;
            &lt;version&gt;${spark.version}&lt;/version&gt;
        &lt;/dependency&gt;
        &lt;dependency&gt; &lt;!-- Spark dependency --&gt;
            &lt;groupId&gt;org.apache.spark&lt;/groupId&gt;
            &lt;artifactId&gt;spark-sql_${scala.version}&lt;/artifactId&gt;
            &lt;version&gt;${spark.version}&lt;/version&gt;
            &lt;scope&gt;provided&lt;/scope&gt;
        &lt;/dependency&gt;
        &lt;dependency&gt;
            &lt;groupId&gt;org.apache.spark&lt;/groupId&gt;
            &lt;artifactId&gt;spark-kubernetes_${scala.version}&lt;/artifactId&gt;
            &lt;version&gt;${spark.version}&lt;/version&gt;
        &lt;/dependency&gt;

        &lt;dependency&gt;
            &lt;groupId&gt;org.springframework.boot&lt;/groupId&gt;
            &lt;artifactId&gt;spring-boot-starter-test&lt;/artifactId&gt;
            &lt;scope&gt;test&lt;/scope&gt;
        &lt;/dependency&gt;
    &lt;/dependencies&gt;
    &lt;build&gt;
        &lt;plugins&gt;
            &lt;plugin&gt;
                &lt;groupId&gt;org.springframework.boot&lt;/groupId&gt;
                &lt;artifactId&gt;spring-boot-maven-plugin&lt;/artifactId&gt;
            &lt;/plugin&gt;
        &lt;/plugins&gt;
    &lt;/build&gt;
&lt;/project&gt;
</code></pre>
<p>and here is the <strong>Dockerfile</strong> I'm using to package my spring-boot application before deploying it to minikube:</p>
<pre><code># Use an existing image as the base image
FROM openjdk:11-jdk

# Set the working directory
WORKDIR /app

# Copy the compiled JAR file to the image
COPY target/wordcount-0.0.1-SNAPSHOT.jar /app/wordcount.jar

RUN useradd -u 185 sparkuser

# Set the entrypoint command to run the JAR file
ENTRYPOINT [&quot;java&quot;, &quot;-jar&quot;, &quot;wordcount.jar&quot;]
</code></pre>
<p>For the <code>spark.kubernetes.container.image</code> I built a docker image using the Dockerfile which is shipped with my local Spark bin (spark-3.3.2-bin-hadoop3 - same Spark version used by my spring-boot app) following <a href="https://spark.apache.org/docs/latest/running-on-kubernetes.html#docker-images" rel="nofollow noreferrer">these instructions</a> and loaded it to minikube.</p>
<p>Here are some of the things I tried with no luck so far:</p>
<ul>
<li>Share my app's jar with Spark using <code>setJars(new String[]{&quot;/app/wordcount.jar&quot;})</code> as suggested <a href="https://stackoverflow.com/questions/28186607/java-lang-classcastexception-using-lambda-expressions-in-spark-job-on-remote-ser/28367602#28367602">here</a> - this absolute file-path is where my app's jar lives on my driver image</li>
<li>use maven-shade-plugin as suggested <a href="https://stackoverflow.com/a/58098501/10185326">here</a> to change the way my app's jar distributes its dependencies - this resulted in a <code>ClassNotFoundException: SparkSession</code> exception on my driver pod.</li>
<li>Refactor my controller's code to not use lambda functions (didn't make a difference):</li>
</ul>
<pre class="lang-java prettyprint-override"><code>public static class SplitLine implements FlatMapFunction&lt;String, String&gt; {
   @Override
   public Iterator&lt;String&gt; call(String line) throws Exception {
       List&lt;String&gt; words = new ArrayList&lt;&gt;();
       for (String word : line.split(&quot; &quot;)) {
           words.add(word);
       }
       return words.iterator();
   }

...

Dataset&lt;String&gt; wordsDataset = text.flatMap(new SplitLine(), Encoders.STRING());
</code></pre>
<p>Any tips or hints regarding my setup or suggestions on how I can refactor my code to get it to work with the existing setup would be greatly appreciated.</p>

itaydafna
<p>In the end I managed to get around this issue by turning this example into a <a href="https://www.baeldung.com/maven-multi-module" rel="nofollow noreferrer">maven multi-module project</a>.</p>
<p>This allowed creating a separate jar for the executor code and then letting Spark know about it using spark-config:</p>
<pre><code>.setJars(new String[]{&quot;word-count-spark-job.jar&quot;})
</code></pre>
<p>More on this solution in <a href="https://www.itaydafna.dev/blog/spring-boot-spark-on-minikube/#project-structure" rel="nofollow noreferrer">this blog-post</a>.</p>
