title: Can Spark write to Azure Datalake Gen2?
body: <p>It seems impossible to write to Azure Datalake Gen2 using spark, unless you're using Databricks.</p>
  
  <p>I'm using <code>jupyter</code> with <code>almond</code> to run spark in a notebook locally.</p>
  
  <p>I have imported the hadoop dependencies:</p>
  
  <pre class="lang-scala prettyprint-override"><code>import $ivy.`org.apache.hadoop:hadoop-azure:2.7.7`
  import $ivy.`com.microsoft.azure:azure-storage:8.4.0` 
  </code></pre>
  
  <p>which allows me to use the <code>wasbs://</code> protocol when trying to write my dataframe to azure</p>
  
  <pre class="lang-scala prettyprint-override"><code>    spark.conf.set(
          "fs.azure.sas.[container].prodeumipsadatadump.blob.core.windows.net", 
          "?sv=2018-03-28&amp;ss=b&amp;srt=sco&amp;sp=rwdlac&amp;se=2019-09-09T23:33:45Z&amp;st=2019-09-09T15:33:45Z&amp;spr=https&amp;sig=[truncated]")
  </code></pre>
  
  <p>This is where the error comes:</p>
  
  <pre class="lang-scala prettyprint-override"><code>val data = spark.read.json(spark.createDataset(
    """{"name":"Yin", "age": 25.35,"address":{"city":"Columbus","state":"Ohio"}}""" :: Nil))
  
  data
    .write
    .orc("wasbs://[filesystem]@[datalakegen2storageaccount].blob.core.windows.net/lalalalala")
  </code></pre>
  
  <p>We are now greeted with "Blob API is not yet supported for hierarchical namespace accounts" error:</p>
  
  <pre><code>org.apache.hadoop.fs.azure.AzureException: com.microsoft.azure.storage.StorageException: Blob API is not yet supported for hierarchical namespace accounts.
  </code></pre>
  
  <p>So is this indeed impossible? Should I just abandon the Datalake gen2 and just use regular blob storage? Microsoft really dropped the ball in creating a "Data lake" product but creating no documentation for a connector with spark.</p>