<p>Have a simple program as shown below</p>
<pre><code>import pyspark

builder = (
    pyspark.sql.SparkSession.builder.appName(&quot;MyApp&quot;)
    .config(&quot;spark.sql.extensions&quot;, &quot;io.delta.sql.DeltaSparkSessionExtension&quot;)
    .config(
        &quot;spark.sql.catalog.spark_catalog&quot;,
        &quot;org.apache.spark.sql.delta.catalog.DeltaCatalog&quot;,
    )
)

spark = builder.getOrCreate()

spark._jsc.hadoopConfiguration().set(
    &quot;fs.gs.impl&quot;, &quot;com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem&quot;
)
spark._jsc.hadoopConfiguration().set(
    &quot;fs.AbstractFileSystem.gs.impl&quot;, &quot;com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS&quot;
)

df = spark.read.format(&quot;delta&quot;).load(
    &quot;gs://org/delta/bronze/mongodb/registration/audits&quot;
)
print(df.show())
</code></pre>
<p>This is packaged into a container using the below Dockerfile</p>
<pre><code>FROM varunmallya/spark-pi:3.2.1
USER root
ADD gcs-connector-hadoop2-latest.jar $SPARK_HOME/jars
WORKDIR /app
COPY main.py .
</code></pre>
<p>This app is then deployed as a SparkApplication on k8s using the <a href="https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/docs/user-guide.md#specifying-deployment-mode" rel="nofollow noreferrer">spark-on-k8s</a> operator</p>
<p>I expected to see 20 rows of data but instead got this exception</p>
<pre><code>java.lang.ClassCastException: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.sql.catalyst.expressions.ScalaUDF.f of type scala.Function1 in instance of org.apache.spark.sql.catalyst.expressions.ScalaUDF
</code></pre>
<p>However when I run this in local jupyter notebook I can see the desired. I have added the necessary package - <em>io.delta:delta-core_2.12:1.2.0</em> via the crd and have also ensured the <em>gcs-connector-hadoop2-latest.jar</em> is made available.</p>
<p>What could the issue be?</p>

<p>Could you try the following <code>Dockerfile</code>:</p>
<pre><code>FROM datamechanics/spark:3.1.1-hadoop-3.2.0-java-8-scala-2.12-python-3.8-dm17
USER root
WORKDIR /app
COPY main.py .
</code></pre>
<p>And then try deploying the <code>SparkApplication</code>:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
  name: sparky-pi
  namespace: spark
spec:
  type: Python
  mode: cluster
  pythonVersion: &quot;3&quot;
  image: &lt;YOUR_IMAGE_GOES_HERE&gt;:latest
  mainApplicationFile: local:///app/main.py
  sparkVersion: &quot;3.1.1&quot;
  restartPolicy:
    type: OnFailure
    onFailureRetries: 3
    onFailureRetryInterval: 10
    onSubmissionFailureRetries: 5
    onSubmissionFailureRetryInterval: 20
  driver:
    cores: 1
    coreLimit: &quot;1200m&quot;
    memory: &quot;512m&quot;
    labels:
      version: 3.1.1
    serviceAccount: spark
  executor:
    serviceAccount: spark
    cores: 1
    instances: 1
    memory: &quot;512m&quot;
    labels:
      version: 3.1.1
</code></pre>
<p>I ran this on my Kubernetes cluster and was able to get:</p>
<p><a href="https://i.stack.imgur.com/Iq5pE.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Iq5pE.png" alt="" /></a></p>
<p>I think here the base image <code>datamechanics/spark:3.1.1-hadoop-3.2.0-java-8-scala-2.12-python-3.8-dm17</code> is key. Props to the folks who put it together!</p>
<p>Source: <a href="https://towardsdatascience.com/optimized-docker-images-for-apache-spark-now-public-on-dockerhub-1f9f8fed1665" rel="nofollow noreferrer">https://towardsdatascience.com/optimized-docker-images-for-apache-spark-now-public-on-dockerhub-1f9f8fed1665</a></p>
