title: Pyspark filter dataframe by columns of another dataframe
body: <p>Not sure why I'm having a difficult time with this, it seems so simple considering it's fairly easy to do in R or pandas.  I wanted to avoid using pandas though since I'm dealing with a lot of data, and I believe <code>toPandas()</code> loads all the data into the driver’s memory in pyspark.</p>
  <p>I have 2 dataframes: <code>df1</code> and <code>df2</code>.  I want to filter <code>df1</code> (remove all rows) where <code>df1.userid = df2.userid</code> AND <code>df1.group = df2.group</code>.  I wasn't sure if I should use <code>filter()</code>, <code>join()</code>, or <code>sql</code>  For example:</p>
  <pre><code>df1:
  +------+----------+--------------------+
  |userid|   group  |      all_picks     |
  +------+----------+--------------------+
  |   348|         2|[225, 2235, 2225]   |
  |   567|         1|[1110, 1150]        |
  |   595|         1|[1150, 1150, 1150]  |
  |   580|         2|[2240, 2225]        |
  |   448|         1|[1130]              |
  +------+----------+--------------------+
  
  df2:
  +------+----------+---------+
  |userid|   group  |   pick  |
  +------+----------+---------+
  |   348|         2|     2270|
  |   595|         1|     2125|
  +------+----------+---------+
  
  Result I want:
  +------+----------+--------------------+
  |userid|   group  |      all_picks     |
  +------+----------+--------------------+
  |   567|         1|[1110, 1150]        |
  |   580|         2|[2240, 2225]        |
  |   448|         1|[1130]              |
  +------+----------+--------------------+
  </code></pre>
  <p>EDIT:
  I've tried many join() and filter() functions, I believe the closest I got was:</p>
  <pre><code>cond = [df1.userid == df2.userid, df2.group == df2.group]
  df1.join(df2, cond, 'left_outer').select(df1.userid, df1.group, df1.all_picks) # Result has 7 rows
  </code></pre>
  <p>I tried a bunch of different join types, and I also tried different</p>
  <pre><code>cond values:
      cond = ((df1.userid == df2.userid) &amp; (df2.group == df2.group)) # result has 7 rows
      cond = ((df1.userid != df2.userid) &amp; (df2.group != df2.group)) # result has 2 rows
  </code></pre>
  <p>However, it seems like the joins are adding additional rows, rather than deleting.</p>
  <p>I'm using <code>python 2.7</code> and <code>spark 2.1.0</code></p>