title: Filter based on another RDD in Spark
body: <p>I would like to keep only the employees which does have a departement ID referenced in the second table. </p>
  
  <pre><code>Employee table
  LastName    DepartmentID
  Rafferty    31
  Jones   33
  Heisenberg  33
  Robinson    34
  Smith   34
  
  Department table
  DepartmentID
  31  
  33  
  </code></pre>
  
  <p>I have tried the following code which does not work:</p>
  
  <pre><code>employee = [['Raffery',31], ['Jones',33], ['Heisenberg',33], ['Robinson',34], ['Smith',34]]
  department = [31,33]
  employee = sc.parallelize(employee)
  department = sc.parallelize(department)
  employee.filter(lambda e: e[1] in department).collect()
  
  Py4JError: An error occurred while calling o344.__getnewargs__. Trace:
  py4j.Py4JException: Method __getnewargs__([]) does not exist
  </code></pre>
  
  <p>Any ideas? I am using Spark 1.1.0 with Python. However, I would accept a Scala or Python answer.</p>