title: Pyspark: join and union in for loop
body: <p>I have a really simple logic that I would like to understand how I can make it work in pyspark.</p>
  <pre><code>for data in df1:
      spark_data_row = spark.createDataFrame(data=[data])
      spark_data_row = spark_data_row.join(df2)
      df2= df2.union(spark_data_row)
  </code></pre>
  <p>Basically, I want to join each row of df_1 to df_2 and then append it to df2 which is initially empty. The resulting df2 is the dataframe of interest for me.</p>
  <p>However, spark runs infinitely on even a small dataframe df1 with 100 rows. I am no sure why this is happening. Can someone suggest another way of doing this?</p>
  <p>Side note: If I comment out either the join or the union statement in this loop, it works in seconds!</p>