title: PySpark/DataBricks: How to read parquet files using 'file:///' and not 'dbfs'
body: <p>I am trying to use <code>petastorm</code> in a different manner which requires that I tell it where my parquet files are stored through one of the following:</p>
  <p><code>hdfs://some_hdfs_cluster/user/yevgeni/parquet8</code>, or <code>file:///tmp/mydataset</code>, or <code>s3://bucket/mydataset</code>, or <code>gs://bucket/mydataset</code>. Since I am on DataBricks and given other constraints, my option is to use the <code>file:///</code> option.</p>
  <p>However, I am at a loss as to how specify the location of my parquet files. I continually get rejected saying that <code>Path does not exist:</code></p>
  <h3>Here is what I am doing:</h3>
  <pre><code># save spark df to parquet
  dbutils.fs.rm('dbfs:/mnt/team01/assembled_train.parquet', recurse=True)
  assembled_train.write.parquet('dbfs:/mnt/team01/assembled_train')
  </code></pre>
  <pre><code># look at files
  display(dbutils.fs.ls('mnt/team01/assembled_train/'))
  </code></pre>
  <pre><code># results
  path    name    size
  dbfs:/mnt/team01/assembled_train/_SUCCESS   _SUCCESS    0
  dbfs:/mnt/team01/assembled_train/_committed_2150262571233317067 _committed_2150262571233317067  856
  dbfs:/mnt/team01/assembled_train/_started_2150262571233317067   _started_2150262571233317067    0
  dbfs:/mnt/team01/assembled_train/part-00000-tid-2150262571233317067-79e6b077-3770-47a9-9fec-155a412768f1-1035357-1-c000.snappy.parquet  part-00000-tid-2150262571233317067-79e6b077-3770-47a9-9fec-155a412768f1-1035357-1-c000.snappy.parquet   578991
  dbfs:/mnt/team01/assembled_train/part-00001-tid-2150262571233317067-79e6b077-3770-47a9-9fec-155a412768f1-1035358-1-c000.snappy.parquet  part-00001-tid-2150262571233317067-79e6b077-3770-47a9-9fec-155a412768f1-1035358-1-c000.snappy.parquet   579640
  dbfs:/mnt/team01/assembled_train/part-00002-tid-2150262571233317067-79e6b077-3770-47a9-9fec-155a412768f1-1035359-1-c000.snappy.parquet  part-00002-tid-2150262571233317067-79e6b077-3770-47a9-9fec-155a412768f1-1035359-1-c000.snappy.parquet   580675
  dbfs:/mnt/team01/assembled_train/part-00003-tid-2150262571233317067-79e6b077-3770-47a9-9fec-155a412768f1-1035360-1-c000.snappy.parquet  part-00003-tid-2150262571233317067-79e6b077-3770-47a9-9fec-155a412768f1-1035360-1-c000.snappy.parquet   579483
  dbfs:/mnt/team01/assembled_train/part-00004-tid-2150262571233317067-79e6b077-3770-47a9-9fec-155a412768f1-1035361-1-c000.snappy.parquet  part-00004-tid-2150262571233317067-79e6b077-3770-47a9-9fec-155a412768f1-1035361-1-c000.snappy.parquet   578807
  dbfs:/mnt/team01/assembled_train/part-00005-tid-2150262571233317067-79e6b077-3770-47a9-9fec-155a412768f1-1035362-1-c000.snappy.parquet  part-00005-tid-2150262571233317067-79e6b077-3770-47a9-9fec-155a412768f1-1035362-1-c000.snappy.parquet   580942
  dbfs:/mnt/team01/assembled_train/part-00006-tid-2150262571233317067-79e6b077-3770-47a9-9fec-155a412768f1-1035363-1-c000.snappy.parquet  part-00006-tid-2150262571233317067-79e6b077-3770-47a9-9fec-155a412768f1-1035363-1-c000.snappy.parquet   579202
  dbfs:/mnt/team01/assembled_train/part-00007-tid-2150262571233317067-79e6b077-3770-47a9-9fec-155a412768f1-1035364-1-c000.snappy.parquet  part-00007-tid-2150262571233317067-79e6b077-3770-47a9-9fec-155a412768f1-1035364-1-c000.snappy.parquet   579810
  </code></pre>
  <p>While testing with a basic dataframe load from the file structure, like so:</p>
  <pre><code>df1 = spark.read.option(&quot;header&quot;, &quot;true&quot;).parquet('file:///mnt/team01/assembled_train/part-00000-tid-2150262571233317067-79e6b077-3770-47a9-9fec-155a412768f1-1035357-1-c000.snappy.parquet')```
  </code></pre>
  <p>I get file does not exist.</p>