title: PySpark Inner join causing Cartesian join
body: <p>I am writing the movie recommender codes in Pyspark. The Recommendation output from ALS is an array inside the movie_id column and another array inside the rating column. But when I am trying to explode the columns individually into temporary dataframes and then join them using 'user_id' the 'inner' join is resulting in a cartesian product.</p>
  
  <pre><code>user_recs_one = user_recs.where(user_recs.user_id == 1)
  user_recs_one.show(truncate=False)
  
  +-------+-------------------------------------------------------+
  |user_id|recommendations                                        |
  +-------+-------------------------------------------------------+
  |1      |[[1085, 6.1223927], [1203, 6.0752907], [745, 5.954721]]|
  +-------+-------------------------------------------------------+
  
  user_recs_one
  DataFrame[user_id: int, recommendations: array&lt;struct&lt;movie_id:int,rating:float&gt;&gt;]
  
  user_recs_one = user_recs_one.select("user_id", "recommendations.movie_id", "recommendations.rating")
  user_recs_one.show(truncate=False)
  
  +-------+-----------------+--------------------------------+
  |user_id|movie_id         |rating                          |
  +-------+-----------------+--------------------------------+
  |1      |[1085, 1203, 745]|[6.1223927, 6.0752907, 5.954721]|
  +-------+-----------------+--------------------------------+
  
  user_recs_one
  DataFrame[user_id: int, movie_id: array&lt;int&gt;, rating: array&lt;float&gt;]
  
  
  x = user_recs_one.select("user_id", F.explode(col("movie_id")).alias("movie_id"))
  x.show()
  
  +-------+--------+
  |user_id|movie_id|
  +-------+--------+
  |      1|    1085|
  |      1|    1203|
  |      1|     745|
  +-------+--------+
  
  y = user_recs_one.select("user_id", 
  F.explode(col("rating")).alias("rating"))
  y.show()
  
  +-------+---------+
  |user_id|   rating|
  +-------+---------+
  |      1|6.1223927|
  |      1|6.0752907|
  |      1| 5.954721|
  +-------+---------+
  
  x.join(y, on='user_id', how='inner').show()
  
  +-------+--------+---------+
  |user_id|movie_id|   rating|
  +-------+--------+---------+
  |      1|    1085|6.1223927|
  |      1|    1085|6.0752907|
  |      1|    1085| 5.954721|
  |      1|    1203|6.1223927|
  |      1|    1203|6.0752907|
  |      1|    1203| 5.954721|
  |      1|     745|6.1223927|
  |      1|     745|6.0752907|
  |      1|     745| 5.954721|
  +-------+--------+---------+
  </code></pre>