title: Broadcast join in Pyspark got OnOutOfMemoryError
body: <p>I use broadcast join in pyspark on a not very large dataframe, but got oom error. Code is very easy as follow:</p>
  
  <pre><code>from pyspark.sql.functions import broadcast
  df1 = sc.read.parquet("xxx")
  df2 = sc.read.parquet("xxx")
  df = df1.join(broadcast(df2), on=["key"])
  </code></pre>
  
  <p>after some seconds i got OOM error</p>
  
  <pre><code># java.lang.OutOfMemoryError: Java heap space
  # -XX:OnOutOfMemoryError="kill -9 %p"
  #   Executing /bin/sh -c "kill -9 26024"...
  </code></pre>
  
  <p>df2 is not very larget, 1868895 lines. The hdfs file is about 142M. I can easily load all data in driver code and use it as a UDF there is no problem. But when using broadcast join, the OOM appear. </p>
  
  <p>My configuration is as follow:</p>
  
  <pre><code>spark.python.worker.memory=2048m
  spark.executor.memory=8g
  spark.driver.memory=7g
  </code></pre>