<p>I am trying to merge some data to delta table in a streaming application in k8s using spark submit in cluster mode</p>
<p>Getting the below error, But its works fine in k8s local mode and in my laptop, none of the operations related to delta lake is working in k8s cluster mode,</p>
<p>Below is the library versions i am using , is it some compatibility issue,</p>
<pre><code>SPARK_VERSION_DEFAULT=3.3.0
HADOOP_VERSION_DEFAULT=3
HADOOP_AWS_VERSION_DEFAULT=3.3.1
AWS_SDK_BUNDLE_VERSION_DEFAULT=1.11.974
</code></pre>
<p>below is the error message</p>
<blockquote>
<p>py4j.protocol.Py4JJavaError: An error occurred while calling o128.saveAsTable. : java.util.concurrent.ExecutionException: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 4) (192.168.15.250 executor 2): java.lang.ClassCastException: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.sql.catalyst.expressions.ScalaUDF.f of type scala.Function1 in instance of org.apache.spark.sql.catalyst.expressions.ScalaUDF</p>
</blockquote>

<p>Finaly able to resolve this issue , issue was due to some reason dependant jars like delta, kafka are not available in executor , as per the below SO response</p>
<p><a href="https://stackoverflow.com/questions/73473309/cannot-assign-instance-of-scala-collection-immutable-listserializationproxy-to">cannot assign instance of scala.collection.immutable.List$SerializationProxy to field org.apache.spark.sql.execution.datasources.v2.DataSourceRDD</a></p>
<p>i have added the jars in spark/jars folder using docker image and issue got resolved ,</p>
