title: Combining Two Spark Streams On Key
body: <p>I have two kafka streams that contain results for two parallel operations, I need a way to combine both streams so I can process the results in a single spark transform. Is this possible? (illustration below)</p>
  
  <pre><code>Stream 1 {id:1,result1:True}
  Stream 2 {id:1,result2:False}
         JOIN(Stream 1, Stream 2, On "id") -&gt; Output Stream {id:1,result1:True,result2:False}
  </code></pre>
  
  <p>Current code that isn't working:</p>
  
  <pre><code>    kvs1 = KafkaUtils.createStream(sparkstreamingcontext, ZOOKEEPER, NAME+"_stream", {"test_join_1": 1})
      kvs2 = KafkaUtils.createStream(sparkstreamingcontext, ZOOKEEPER, NAME+"_stream", {"test_join_2": 1})
  
      messages_RDDstream1 = kvs1.map(lambda x: x[1])
      messages_RDDstream2 = kvs2.map(lambda x: x[1])
  
      messages_RDDstream_Final = messages_RDDstream1.join(messages_RDDstream2)
  </code></pre>
  
  <p>When I pass two sample jsons to each Kafka queue with the same ID field, nothing is returned in my final RDD stream. I imaging I am missing the stage of converting my Kafka JSON string message into a Tuple?</p>
  
  <p>I have also tried the following:</p>
  
  <pre><code>kvs1.map(lambda (key, value): json.loads(value))
  </code></pre>
  
  <p>and</p>
  
  <pre><code>kvs1.map(lambda x: json.loads(x))
  </code></pre>
  
  <p>To no avail</p>
  
  <p>Cheers</p>
  
  <p>Adam</p>