title: PySpark distributed processing on a YARN cluster
body: <p>I have Spark running on a Cloudera CDH5.3 cluster, using YARN as the resource manager. I am developing Spark apps in Python (PySpark).</p>
  
  <p>I can submit jobs and they run succesfully, however they never seem to run on more than one machine (the local machine I submit from).</p>
  
  <p>I have tried a variety of options, like setting --deploy-mode to cluster and --master to yarn-client and yarn-cluster, yet it never seems to run on more than one server.</p>
  
  <p>I can get it to run on more than one core by passing something like --master local[8], but that obviously doesn't distribute the processing over multiple nodes.</p>
  
  <p>I have a very simply Python script processing data from HDFS like so:</p>
  
  <pre><code>import simplejson as json
  from pyspark import SparkContext
  sc = SparkContext("", "Joe Counter")
  
  rrd = sc.textFile("hdfs:///tmp/twitter/json/data/")
  
  data = rrd.map(lambda line: json.loads(line))
  
  joes = data.filter(lambda tweet: "Joe" in tweet.get("text",""))
  
  print joes.count()
  </code></pre>
  
  <p>And I am running a submit command like:</p>
  
  <pre><code>spark-submit atest.py --deploy-mode client --master yarn-client
  </code></pre>
  
  <p>What can I do to ensure the job runs in parallel across the cluster?</p>