title: Pyspark: Calculating sum of two correspoding columns, based on conditions of two columns in two RDDs
body: <p>I have two RDDs with same columns: <br>
  rdd1 :-</p>
  
  <pre>
  +-----------------+
  |mid|uid|frequency|
  +-----------------+
  | m1| u1|        1|
  | m1| u2|        1|
  | m2| u1|        2|
  +-----------------+
  </pre>
  
  <p>rdd2 :-</p>
  
  <pre>
  +-----------------+
  |mid|uid|frequency|
  +-----------------+
  | m1| u1|       10|
  | m2| u1|       98|
  | m3| u2|       21|
  +-----------------+
  </pre>
  
  <p>I want to calculate sum of <code>frequencies</code> based on <code>mid</code> and <code>uid</code>. Result should be something like:</p>
  
  <pre>
  +-----------------+
  |mid|uid|frequency|
  +-----------------+
  | m1| u1|       11|
  | m2| u1|      100|
  | m3| u2|       21|
  +-----------------+
  </pre>
  
  <p>Thanks in advance.</p>
  
  <p>EDIT:
  I achieved the solution in this way as well (Using map-reduce):</p>
  
  <pre><code>from pyspark.sql.functions import col
  
  data1 = [("m1","u1",1),("m1","u2",1),("m2","u1",2)]
  data2 = [("m1","u1",10),("m2","u1",98),("m3","u2",21)]
  df1 = sqlContext.createDataFrame(data1,['mid','uid','frequency'])
  df2 = sqlContext.createDataFrame(data2,['mid','uid','frequency'])
  
  df3 = df1.unionAll(df2)
  df4 = df3.map(lambda bbb: ((bbb['mid'], bbb['uid']), int(bbb['frequency'])))\
               .reduceByKey(lambda a, b: a+b)
  
  p = df4.map(lambda p: (p[0][0], p[0][1], p[1])).toDF()
  
  p = p.select(col("_1").alias("mid"), \
               col("_2").alias("uid"), \
               col("_3").alias("frequency"))
  
  p.show()
  </code></pre>
  
  <p>Output:</p>
  
  <pre>
  +---+---+---------+
  |mid|uid|frequency|
  +---+---+---------+
  | m2| u1|      100|
  | m1| u1|       11|
  | m1| u2|        1|
  | m3| u2|       21|
  +---+---+---------+
  </pre>