title: Split a RDD into Multiple RDD based on value without doing `collect()` and `filter()`
body: <p>I want to split an RDD into multiple RDD based on a value in a row. The values in rows are pre-known and are fixed in nature.</p>
  
  <p>for e.g.</p>
  
  <pre><code>source_rdd = sc.parallelize([('a',1),('a',2),('a',3),('b',4),('b',5),('b',6)])
  </code></pre>
  
  <p>should be split into two RDDs with one containing only <code>a</code> and another containing only <code>b</code> as keys</p>
  
  <ol>
  <li>I have tried <code>groupByKey</code> method and able to do it successfully after doing a <code>collect()</code> operation on grouped RDD, which I cannot do in production due to memory constraints</li>
  </ol>
  
  <pre><code>a_rdd, b_rdd = source_rdd.keyBy(lambda row: row[0]).groupByKey().collect()</code></pre>
  
  <ol start="2">
  <li>The current implementation is to apply multiple filter operation to get each RDD</li>
  </ol>
  
  <pre><code>a_rdd = source_rdd.filter(lambda row: row[0] == 'a')
  b_rdd = source_rdd.filter(lambda row: row[0] == 'b')
  </code></pre>
  
  <p>Can this be optimized further, what will be the best way to do it in production with data which cannot fit in memory?</p>
  
  <p><strong>Usage:</strong> These RDD will be converted into different Dataframes (one for each key), each with different schema and stored in S3 as output.</p>
  
  <p><strong>Note:</strong> I would prefer <code>pyspark</code> implementation. I have read a lot of stack overflow answers and blogs, and could not find anyway which is working for me yet.</p>
  
  <p>I have already seen question which is marked duplicate for, which I have already mentioned in my question. I have asked this question as the provided solution seems not the most optimised way and is 3 years old.</p>