title: spark streaming DirectKafkaInputDStream: kafka data source can easily stress the driver node
body: <p>I am building a prototype with spark streaming 1.5.0. <code>DirectKafkaInputDStream</code> is used. </p>
  
  <p>And a simple stage to read from kafka by <code>DirectKafkaInputDStream</code> can't handle massive amount of messages. The stage spends longer time then batch interval, once the message rate reach or exceed a certain value. And the rate is much lower than I expect. ( I have done another benchmark of my kafka cluster with multiple consumer instances in different servers)</p>
  
  <pre><code>JavaPairInputDStream&lt;String, String&gt; recipeDStream =
          KafkaUtils.createDirectStream(jssc, 
              String.class, 
              String.class, 
              StringKeyDecoder.class,
              StringDecoder.class,
              kafkaParams, kafkaTopicsSet);
  </code></pre>
  
  <p>After reading this <a href="http://www.spark.tc/custom-spark-streaming-dstreams-and-receivers/" rel="nofollow">article</a>, I realize that the <code>DirectKafkaInputDStream</code> is run  on the same node as the driver program. is it ture? If so, then <code>DirectKafkaInputDStream</code> can easily be stressed as it read all message in one node then dispatch to all executors. </p>
  
  <p>And it means <code>JavaPairReceiverInputDStream</code> has better performance in handling high volume data, since receivers runs on multiple executor instances. </p>
  
  <p>Am I right? Can someone explain this? Thank you.</p>