<p>I am new to the world of Spark and Kubernetes. I built a Spark docker image using the official Spark 3.0.1 bundled with Hadoop 3.2 using the docker-image-tool.sh utility.</p>
<p>I have also created another docker image for Jupyter notebook and am trying to run spark on Kubernetes in client mode. I first run my Jupyter notebook as a pod, do a port forward using kubectl and access the notebook UI from my system at localhost:8888 . All seems to be working fine. I am able to run commands successfully from the notebook.</p>
<p>Now I am trying to access Azure Data Lake Gen2 from my notebook using <a href="https://hadoop.apache.org/docs/current/hadoop-azure/abfs.html#Default:_Shared_Key" rel="noreferrer">Hadoop ABFS connector</a>.
I am setting the Spark context as below.</p>
<pre><code>from pyspark import SparkContext, SparkConf
from pyspark.sql import SparkSession
# Create Spark config for our Kubernetes based cluster manager


sparkConf = SparkConf()
sparkConf.setMaster(&quot;k8s://https://kubernetes.default.svc.cluster.local:443&quot;)
sparkConf.setAppName(&quot;spark&quot;)
sparkConf.set(&quot;spark.kubernetes.container.image&quot;, &quot;&lt;&lt;my_repo&gt;&gt;/spark-py:latest&quot;)
sparkConf.set(&quot;spark.kubernetes.namespace&quot;, &quot;spark&quot;)
sparkConf.set(&quot;spark.executor.instances&quot;, &quot;3&quot;)
sparkConf.set(&quot;spark.executor.cores&quot;, &quot;2&quot;)
sparkConf.set(&quot;spark.driver.memory&quot;, &quot;512m&quot;)
sparkConf.set(&quot;spark.executor.memory&quot;, &quot;512m&quot;)
sparkConf.set(&quot;spark.kubernetes.pyspark.pythonVersion&quot;, &quot;3&quot;)
sparkConf.set(&quot;spark.kubernetes.authenticate.driver.serviceAccountName&quot;, &quot;spark&quot;)
sparkConf.set(&quot;spark.kubernetes.authenticate.serviceAccountName&quot;, &quot;spark&quot;)
sparkConf.set(&quot;spark.driver.port&quot;, &quot;29413&quot;)
sparkConf.set(&quot;spark.driver.host&quot;, &quot;my-notebook-deployment.spark.svc.cluster.local&quot;)

sparkConf.set(&quot;fs.azure.account.auth.type&quot;, &quot;SharedKey&quot;)
sparkConf.set(&quot;fs.azure.account.key.&lt;&lt;storage_account_name&gt;&gt;.dfs.core.windows.net&quot;,&quot;&lt;&lt;account_key&gt;&gt;&quot;)

spark = SparkSession.builder.config(conf=sparkConf).getOrCreate()
</code></pre>
<p>And then I am running the below command to read a csv file present in the ADLS location</p>
<pre><code>df = spark.read.csv(&quot;abfss://&lt;&lt;container&gt;&gt;@&lt;&lt;storage_account&gt;&gt;.dfs.core.windows.net/&quot;)
</code></pre>
<p>On runnining it I am getting the error
<em><strong>Py4JJavaError: An error occurred while calling o443.csv.
: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azurebfs.SecureAzureBlobFileSystem not found</strong></em></p>
<p>After some research, I found that I would have to explicitly include the hadoop-azure jar for the appropriate classes to be available. I downloaded the jar from <a href="https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-azure/3.2.2/hadoop-azure-3.2.2.jar" rel="noreferrer">here</a> , put it in the /spark-3.0.1-bin-hadoop3.2/jars folder and built the image again.</p>
<p><strong>Unfortunately I am still getting this error</strong>. I manually verified that the jar file is indeed present in the docker image and contains the class <code>org.apache.hadoop.fs.azurebfs.SecureAzureBlobFileSystem</code></p>
<p>I looked at the entrypoint.sh present at <code>spark-3.0.1-bin-hadoop3.2\kubernetes\dockerfiles\spark</code> folder which is the entry point of our spark docker image. It adds all the packages present in the <code>spark-3.0.1-bin-hadoop3.2\jar\</code> folder in the class path.</p>
<pre><code># If HADOOP_HOME is set and SPARK_DIST_CLASSPATH is not set, set it here so Hadoop jars are available to the executor.
# It does not set SPARK_DIST_CLASSPATH if already set, to avoid overriding customizations of this value from elsewhere e.g. Docker/K8s.
if [ -n &quot;${HADOOP_HOME}&quot;  ] &amp;&amp; [ -z &quot;${SPARK_DIST_CLASSPATH}&quot;  ]; then
  export SPARK_DIST_CLASSPATH=&quot;$($HADOOP_HOME/bin/hadoop classpath)&quot;
fi

if ! [ -z ${HADOOP_CONF_DIR+x} ]; then
  SPARK_CLASSPATH=&quot;$HADOOP_CONF_DIR:$SPARK_CLASSPATH&quot;;
fi
</code></pre>
<p>According to my understanding spark should be able to find the class in its classpath with any addition setJar configuration.</p>
<p>Can someone please guide me how to resolve this? I might be missing something quite basic here.</p>

<p>Applying the solution provided here...</p>
<p><a href="https://stackoverflow.com/questions/42978976/how-do-we-specify-maven-dependencies-in-pyspark">How do we specify maven dependencies in pyspark</a></p>
<p>We can start a Spark session and include the required Jar from Maven.</p>
<pre><code>from pyspark.sql import SparkSession


spark = SparkSession.builder.master(&quot;local[*]&quot;)\
        .config('spark.jars.packages', 'org.apache.hadoop:hadoop-azure:3.3.1')\
        .getOrCreate()
</code></pre>
