title: Testing file matching in pyspark
body: <p>I have a <a href="https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.SparkContext.html" rel="nofollow noreferrer">SparkContext</a> and a bunch of files that I want to access using the <a href="https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.SparkContext.textFile.html?highlight=textfile#pyspark.SparkContext.textFile" rel="nofollow noreferrer">textFile</a> method. The logic to find which files I need to access is complex and requires careful testing.</p>
  <p>In my testing environment (using pytest), all the python files and files the <code>SparkContext</code> needs to access are on my local machine. The problem I encounter is that creating a <code>SparkContext</code> object in this testing environment and loading data using <code>textFile</code> often fails with a timeout exceeding 1 minute (cf <a href="https://stackoverflow.com/questions/30944931/mock-a-spark-rdd-in-the-unit-tests">Mock a Spark RDD in the unit tests</a>).</p>
  <p>In my testing environment, how to test that the file patterns I send to <code>textFile</code> are right, and therefore ensure that <code>textFile</code> returns an <code>rdd</code> from the right files?</p>