title: Kinesis Streaming Example Not Working in Cluster Mode on EMR (EMR 4.3, Spark 1.6)
body: <p>I’m trying unsuccessfully to run a version of the Kinesis Streaming word-count example on EMR 4.3 in cluster mode. Specifically, no messages are being read from Kinesis, even though I can access the stream's metadata.</p>
  
  <p>This same code does run in client mode on an identical EMR cluster (i.e. with "local[*]"), but when I try to do it in cluster mode the 1st job of the kinesis receiver is stuck:
  <a href="https://i.stack.imgur.com/b2C0v.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/b2C0v.png" alt="spark ui jobs"></a></p>
  
  <p>And I see nothing in the  streaming page of the Spark UI:
  <a href="https://i.stack.imgur.com/Buege.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Buege.png" alt="spark ui streaming"></a></p>
  
  <p>Initially I thought this was a matter of resources/ number of threads, but based on the configuration and what I see in the YARN and Spark UIs, this doesn't seem to be the case (see all relevant configurations below).</p>
  
  <p><strong>I'm looking for any pointers to why the application cannot read from Kinesis, or for suggested changes in configuration or setup to make this work in cluster mode.</strong></p>
  
  <hr>
  
  <p><em>Configuration and setup details</em></p>
  
  <p>The relevant Kinesis stream has a single Shard.</p>
  
  <p>I use the following configuration in the EMR cluster setting:</p>
  
  <pre><code>[{"classification":"capacity-scheduler",
  
  "properties":{"yarn.scheduler.capacity.resource-calculator":"org.apache.hadoop.yarn.util.resource.DominantResourceCalculator"}},
  
  {"classification":"spark","properties":{"maximizeResourceAllocation":"true"}},
  
  {"classification":"spark-defaults","properties":{
  
  "spark.executor.instances":"0",
  "spark.dynamicAllocation.enabled":"true"}}]
  </code></pre>
  
  <p>And this is how my environment setting look like in the spark ui:
   <a href="https://i.stack.imgur.com/ZnmsT.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/ZnmsT.png" alt="saprkui env"></a></p>
  
  <p>The code that I’m trying to run : </p>
  
  <pre><code> val appName =  "ks_"+DateTime.now().toString(formatter);
   val sparkConf = new SparkConf().setAppName(appName)
  
  
  val sc = new SparkContext(sparkConf);
  val batchIntervalInSec = 5
  val batchInterval = Seconds(batchIntervalInSec)     
  
  val ssc = new StreamingContext(sc, batchInterval)
  ssc.checkpoint("/checkpoint")
   val kinesisClient = new AmazonKinesisClient(credentials)
  kinesisClient.setEndpoint(endpointUrl)
  val numShards = kinesisClient.describeStream(streamName).getStreamDescription().getShards().size
  
  val numStreams = numShards
  
  val kinesisCheckpointInterval = Seconds(batchIntervalInSec-1) 
  val regionName = RegionUtils.getRegionByEndpoint(endpointUrl).getName()
  
  
   val kinesisStreams = (0 until numStreams).map { i =&gt;
    KinesisUtils.createStream(ssc, appName, streamName, endpointUrl, regionName,
      InitialPositionInStream.LATEST, kinesisCheckpointInterval, StorageLevel.MEMORY_ONLY)
  }
  
  val unionStreams = ssc.union(kinesisStreams)
  
  val words = unionStreams.flatMap(byteArray =&gt; new String(byteArray).split(" "))
  
  
  val wordCounts = words.map(word =&gt; (word, 1)).reduceByKey(_ + _)
  
  wordCounts.print()
  ssc.start()
  ssc.awaitTermination();
  </code></pre>
  
  <p>And this is the spark command that I run:</p>
  
  <pre><code>spark-submit --deploy-mode cluster 
  --class com.komoona.spark.kmn_spark_scala.KinesisStream 
  --master yarn --conf spark.executor.cores=4 
  --conf spark.executor.instances=2 
  --conf spark.streaming.blockInterval=1000ms 
   --jars /home/hadoop/lib/spark-streaming-kinesis-asl-assembly_2.10-1.6.0.jar,/home/hadoop/lib/amazon-kinesis-client-1.6.1.jar, 
  test_app_full.jar
  </code></pre>
  
  <p>EDIT: 
  I noticed that in spite having 2 executors configured (as specified in the command line), only one executor and the driver are shown running in the spark UI:
  <a href="https://i.stack.imgur.com/iM21S.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/iM21S.png" alt="executors"></a></p>
  
  <p>Could this be the source to the issue? Any ideas what can cause this?</p>