title: How to read parquet files using pyspark when paths are listed in a dataframe
body: <p>I have a dataframe which contains list of 1M s3 paths something like below:</p>
  <p>input_df_s3_path</p>
  <pre><code>|path                    |
  --------------------------
  S3://abcbucket/fhg.parquet
  s3://abcbucket/gbj.parquet
  </code></pre>
  <p>I want to read all of these parquet files in an efficient way considering large data volume.</p>
  <p>This is how I am doing this:</p>
  <pre><code>input_df_s3_path = spark.read.csv(&quot;s3a://bucketname/s3_path.csv&quot;)
      s3_paths = input_df_s3_path.select('path').rdd.map(lambda row : row[0]).collect()
      input_df = sqlContext.read.parquet(*s3_paths)
  </code></pre>
  <p>Is there any other way so that I can read the files in more efficient and faster way?</p>