title: Spark: Reading CSV files from list of paths in a DataFrame Row
body: <p>I have a Spark DataFrame as follows:</p>
  
  <pre><code># ---------------------------------
  # - column 1 - ...  -   column 5  -
  # ---------------------------------
  # - ...             - Array of paths
  </code></pre>
  
  <p>Columns 1 to 4 contain strings and the fifth column contains list of strings, that are actually paths to CSV files I wish to read as Spark Dataframes. I cannot find anyway to read them. Here's a simplified version with just a single column and the column with the list of paths:</p>
  
  <pre><code>from pyspark.sql import SparkSession,Row
  
  spark = SparkSession \
          .builder \
          .appName('test') \
          .getOrCreate()
  
  simpleRDD = spark.sparkContext.parallelize(range(10))
  simpleRDD = simpleRDD.map(lambda x: Row(**{'a':x,'paths':['{}_{}.csv'.format(y**2,y+1) for y in range(x+1)]}))
  
  simpleDF = spark.createDataFrame(simpleRDD)
  print(simpleDF.head(5))
  </code></pre>
  
  <p>This gives:</p>
  
  <pre><code>[Row(a=0, paths=['0_1.csv']),  
   Row(a=1, paths=['0_1.csv', '1_2.csv']),  
   Row(a=2, paths=['0_1.csv', '1_2.csv', '4_3.csv']),  
   Row(a=3, paths=['0_1.csv', '1_2.csv', '4_3.csv', '9_4.csv']),  
   Row(a=4, paths=['0_1.csv', '1_2.csv', '4_3.csv', '9_4.csv', '16_5.csv'])]
  </code></pre>
  
  <p>I would like then to do something like this:</p>
  
  <pre><code>simpleDF = simpleDF.withColumn('data',spark.read.csv(simpleDF.paths))
  </code></pre>
  
  <p>...but this of course, does not work.</p>