<p>I'm trying to upgrade from Spark 3.0.1 to 3.1.1.</p>
<p>I am running PySpark 3.1.1 in client mode on Jupyter notebook.</p>
<p>The following ran on 3.0.1 but fails after upgrading spark:</p>
<pre><code>from pyspark import SparkContext, SparkConf
from pyspark.sql import SparkSession



def get_spark_session(app_name: str, conf: SparkConf):
    conf.setMaster(&quot;k8s://https://kubernetes.default.svc.cluster.local&quot;)
    conf \
      .set(&quot;spark.kubernetes.namespace&quot;, &quot;spark-ml&quot;) \
      .set(&quot;spark.kubernetes.container.image&quot;, &quot;itayb/spark:3.1.1-hadoop-3.2.0-python-3.8.6-aws&quot;) \
      .set(&quot;spark.kubernetes.authenticate.driver.serviceAccountName&quot;, &quot;spark-executor&quot;) \
      .set(&quot;spark.executor.instances&quot;, &quot;2&quot;) \
      .set(&quot;spark.executor.memory&quot;, &quot;2g&quot;) \
      .set(&quot;spark.executor.cores&quot;, &quot;2&quot;) \
      .set(&quot;spark.driver.memory&quot;, &quot;1G&quot;) \
      .set(&quot;spark.driver.port&quot;, &quot;2222&quot;) \
      .set(&quot;spark.driver.blockManager.port&quot;, &quot;7777&quot;) \
      .set(&quot;spark.driver.host&quot;, &quot;jupyter.recs.svc.cluster.local&quot;) \
      .set(&quot;spark.driver.bindAddress&quot;, &quot;0.0.0.0&quot;) \
      .set(&quot;spark.ui.port&quot;, &quot;4040&quot;) \
      .set(&quot;spark.network.timeout&quot;, &quot;240&quot;) \
      .set(&quot;spark.hadoop.fs.s3a.endpoint&quot;, &quot;localstack.kube-system.svc.cluster.local:4566&quot;) \
      .set(&quot;spark.hadoop.fs.s3a.connection.ssl.enabled&quot;, &quot;false&quot;) \
      .set(&quot;spark.hadoop.fs.s3a.path.style.access&quot;, &quot;true&quot;) \
      .set(&quot;spark.hadoop.fs.s3a.impl&quot;, &quot;org.apache.hadoop.fs.s3a.S3AFileSystem&quot;) \
      .set(&quot;spark.hadoop.com.amazonaws.services.s3.enableV4&quot;, &quot;true&quot;) \
      .set(&quot;spark.hadoop.fs.s3a.aws.credentials.provider&quot;, &quot;org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider&quot;)
    return SparkSession.builder.appName(app_name).config(conf=conf).getOrCreate()


spark = get_spark_session(&quot;aws_localstack&quot;, SparkConf())
try:
    df = spark.read.csv('s3a://my-bucket/stocks.csv',header=True)
    df.printSchema()
    print(df.count())
except Exception as exp:
    print(exp)
    
spark.stop()
</code></pre>
<p>The executors logs:</p>
<pre><code>+ SPARK_CLASSPATH=':/opt/spark/jars/*'                                                                                                                                    │
+ env                                                                                                                                                                     │
+ grep SPARK_JAVA_OPT_                                                                                                                                                    │
+ sort -t_ -k4 -n                                                                                                                                                         │
+ sed 's/[^=]*=\(.*\)/\1/g'                                                                                                                                               │
+ readarray -t SPARK_EXECUTOR_JAVA_OPTS                                                                                                                                   │
+ '[' -n '' ']'                                                                                                                                                           │
+ '[' -z ']'                                                                                                                                                              │
+ '[' -z ']'                                                                                                                                                              │
+ '[' -n '' ']'                                                                                                                                                           │
+ '[' -z ']'                                                                                                                                                              │
+ '[' -z x ']'                                                                                                                                                            │
+ SPARK_CLASSPATH='/opt/spark/conf::/opt/spark/jars/*'                                                                                                                    │
+ case &quot;$1&quot; in                                                                                                                                                            │
+ shift 1                                                                                                                                                                 │
+ CMD=(${JAVA_HOME}/bin/java &quot;${SPARK_EXECUTOR_JAVA_OPTS[@]}&quot; -Xms$SPARK_EXECUTOR_MEMORY -Xmx$SPARK_EXECUTOR_MEMORY -cp &quot;$SPARK_CLASSPATH:$SPARK_DIST_CLASSPATH&quot; org.apac │
+ exec /usr/bin/tini -s -- /usr/local/openjdk-11/bin/java -Dspark.network.timeout=240 -Dspark.driver.port=2222 -Dspark.ui.port=4040 -Dspark.driver.blockManager.port=7777 │
Unrecognized options: --resourceProfileId                                                                                                                                 │
Usage: org.apache.spark.executor.CoarseGrainedExecutorBackend [options]                                                                                                   │
 Options are:                                                                                                                                                             │
   --driver-url &lt;driverUrl&gt;                                                                                                                                               │
   --executor-id &lt;executorId&gt;                                                                                                                                             │
   --bind-address &lt;bindAddress&gt;                                                                                                                                           │
   --hostname &lt;hostname&gt;                                                                                                                                                  │
   --cores &lt;cores&gt;                                                                                                                                                        │
   --resourcesFile &lt;fileWithJSONResourceInformation&gt;                                                                                                                      │
   --app-id &lt;appid&gt;                                                                                                                                                       │
   --worker-url &lt;workerUrl&gt;                                                                                                                                               │
   --user-class-path &lt;url&gt;                                                                                                                                                │
   --resourceProfileId &lt;id&gt;                                                                                                                                               │
stream closed
</code></pre>
<p>seems like <code>--resourceProfileId</code> doesn't have value, but I'm not sure why.</p>

ItayB
<p>It was my bad :(</p>
<p>I used different Spark versions between the driver (3.1.1) and executors (3.0.1).</p>
<p>The mistake happened because I tried to build the executor image from the <a href="https://github.com/apache/spark" rel="nofollow noreferrer">official repo</a> by simply checking out to the <code>v3.1.1</code> tag and use the</p>
<pre><code>./bin/docker-image-tool.sh
</code></pre>
<p>I guess that I had to compile or something before because the version was older.
I verified that by <code>exec</code> to the worker container and run:</p>
<pre><code>spark-submit --version
</code></pre>
<p>The fast solution was to download the right compiled version (instead of using the source), extract and build:</p>
<pre><code>wget https://archive.apache.org/dist/spark/spark-3.1.1/spark-3.1.1-bin-hadoop3.2.tgz
tar xvzf spark-3.1.1-bin-hadoop3.2.tgz
cd spark-3.1.1-bin-hadoop3.2.tgz
./bin/docker-image-tool.sh -u root -r itayb -t 3.1.1-hadoop-3.2.0 -p kubernetes/dockerfiles/spark/bindings/python/Dockerfile build
</code></pre>
