title: Processing CSV files one by one in pyspark
body: <p>CSV files are placed in HDFS folder. All the files are of same schema. I have to read the CSV files and create partitions based on file name. Is there a way to read all the CSV files in Dataframe and process it one by one?</p>
  
  <pre><code>File name : file1_20200417123.csv
  Parition : year_month=202004 day=17
  
  File name : file2_20190212208.csv
  Partition : year_month=201902 day=12
  </code></pre>
  
  <p>Need to read these 2 files and create partitions and save the data to respective partitions.</p>