title: Spark output when submitting via Yarn cluster vs. client
body: <p>I am new to Spark and just got it running on my cluster (Spark 2.0.1 on a 9 node cluster running Community version of MapR). I submit the wordcount example via</p>
  
  <pre><code>./bin/spark-submit --master yarn --jars ~/hadoopPERMA/jars/hadoop-lzo-0.4.21-SNAPSHOT.jar examples/src/main/python/wordcount.py ./README.md
  </code></pre>
  
  <p>and get the following output</p>
  
  <pre><code>17/04/07 13:21:34 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
  : 68
  help: 1
  when: 1
  Hadoop: 3
  ...
  </code></pre>
  
  <p>Looks like everything is working properly. When I add <code>--deploy-mode cluster</code> I get the following output:</p>
  
  <pre><code>17/04/07 13:23:52 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
  </code></pre>
  
  <p>So no errors <strong>but I am not seeing the wordcount results</strong>. What am I missing? I see the job in my History Server and it says it completed successfully. Also I checked my user directory in DFS but no new files were written except for this empty directory: <code>/user/myuser/.sparkStaging</code></p>
  
  <p><strong>Code</strong> (wordcount.py example shipped with Spark):</p>
  
  <pre><code>from __future__ import print_function
  import sys
  from operator import add
  from pyspark.sql import SparkSession
  
  
  if __name__ == "__main__":
      if len(sys.argv) != 2:
          print("Usage: wordcount &lt;file&gt;", file=sys.stderr)
          exit(-1)
  
      spark = SparkSession\
          .builder\
          .appName("PythonWordCount")\
          .getOrCreate()
  
      lines = spark.read.text(sys.argv[1]).rdd.map(lambda r: r[0])
      counts = lines.flatMap(lambda x: x.split(' ')) \
                .map(lambda x: (x, 1)) \
                .reduceByKey(add)
      output = counts.collect()
      for (word, count) in output:
          print("%s: %i" % (word, count))
  
      spark.stop()
  </code></pre>