title: Spark : Read file only if the path exists
body: <p>I am trying to read the files present at <code>Sequence</code> of Paths in scala. Below is the sample (pseudo) code:</p>
  
  <pre><code>val paths = Seq[String] //Seq of paths
  val dataframe = spark.read.parquet(paths: _*)
  </code></pre>
  
  <p>Now, in the above sequence, some paths exist whereas some don't. Is there any way to ignore the missing paths while reading <code>parquet</code> files (to avoid <code>org.apache.spark.sql.AnalysisException: Path does not exist</code>)?</p>
  
  <p>I have tried the below and it seems to be working, but then, I end up reading the same path twice which is something I would like to avoid doing:</p>
  
  <pre><code>val filteredPaths = paths.filter(p =&gt; Try(spark.read.parquet(p)).isSuccess)
  </code></pre>
  
  <p>I checked the <code>options</code> method for <code>DataFrameReader</code> but that does not seem to have any option that is similar to <code>ignore_if_missing</code>.</p>
  
  <p>Also, these paths can be <code>hdfs</code> or <code>s3</code> (this <code>Seq</code> is passed as a method argument) and while reading, I don't know whether a path is <code>s3</code> or <code>hdfs</code> so can't use <code>s3</code> or <code>hdfs</code> specific API to check the existence.</p>