title: PySpark - loop through each row of dataframe and run a hive query
body: <p>I have a dataframe with 100  rows [ name, age, date, hour] . I need to partition this dataframe with distinct values of date. Let's say there are 20 distinct date values in these 100 rows , then i need to spawn up 20 parallel hive queries where each hive QL will join each of these partitions with a hive table . Hive table  - [dept, couse , date] is partitioned by date field. </p>
  
  <p>Hive table is huge and hence I need to optimize these joins in to multiple smaller joins and then aggregate these results. Any recommendations on how can I achieve this ?</p>