title: java.lang.NoSuchMethodError: org.apache.hadoop.security.ProviderUtils.excludeIncompatibleCredentialProviders while reading from Azure Blob Storage
body: <p>I am trying to read a CSV file stored in Azure Storage Account. For that, I have installed a spark on my Virtual Machine and trying to read a CSV file in a dataframe from pyspark.</p>
  <p>I read somewhere how to do that and I followed the steps and copied the latest hadoop-azure &amp; azure-storage JAR files on my /jar directories. Then, I came up with this error:-</p>
  <blockquote>
  <p>NoClassDefFoundError: org/apache/hadoop/fs/StreamCapabilities</p>
  </blockquote>
  <p>I searched for this error and found that I need to refer hadoop-azure-2.8.5.jar instead of latest hadoop-azure JAR. So, I replaced this JAR with the latest hadoop-azure jar and again executed my pyspark code.</p>
  <p>After executing my code, I encountered with another error: -</p>
  <blockquote>
  <p>: java.lang.NoSuchMethodError:
  org.apache.hadoop.security.ProviderUtils.excludeIncompatibleCredentialProviders(Lorg/apache/hadoop/conf/Configuration;Ljava/lang/Class;)Lorg/apache/hadoop/conf/Configuration;</p>
  </blockquote>
  <p>Also, below is my pyspark code: -</p>
  <pre><code>from pyspark import SparkContext
  from pyspark.sql import SparkSession
  from pyspark.sql import Window
  from pyspark.sql.types import *
  from pyspark.sql.functions import *
  
  spark = SparkSession.builder.getOrCreate()
  storage_account_name = &quot;&lt;storage_account_name&gt;&quot;
  storage_account_access_key = &quot;&lt;storage_account_access_key&gt;&quot;
  spark.conf.set(&quot;fs.azure.account.key.&quot; + storage_account_name + &quot;.blob.core.windows.net&quot;,storage_account_access_key)
  
  spark._jsc.hadoopConfiguration().set(&quot;fs.wasbs.impl&quot;,&quot;org.apache.hadoop.fs.azure.NativeAzureFileSystem&quot;)
  spark._jsc.hadoopConfiguration().set(&quot;fs.azure&quot;, &quot;org.apache.hadoop.fs.azure.NativeAzureFileSystem&quot;)
  spark._jsc.hadoopConfiguration().set(&quot;fs.azure.account.key.my_account.blob.core.windows.net&quot;, &quot;storage_account_access_key&quot;)
  
  
  df = spark.read.format(&quot;csv&quot;).option(&quot;inferSchema&quot;, &quot;true&quot;).load(&quot;wasbs://&lt;container_name&gt;@&lt;storage_account_name&gt;.blob.core.windows.net/&lt;path_to_csv&gt;/sample_file.csv&quot;)
  df.show()
  </code></pre>