title: EMR step did not appear to run/create jobs
body: <p>I am trying to get EMR to run a simple hello world type app. </p>
  
  <pre><code>from pyspark import SparkContext
  from operator import add
  
  sc = SparkContext()
  data = sc.parallelize(list("Hello World"))
  data.show()
  counts = data.map(lambda x: (x, 1)).reduceByKey(add).sortBy(lambda x: x[1], ascending=False).coalesce(1).saveAsTextFile('s3://pinfare-glue/emr/output.txt')
  print("Hello EMR")
  sc.stop()
  </code></pre>
  
  <p>But I dont see any hello in logs. Neither is there any files in S3</p>
  
  <p>I have previously uploaded the script into S3 and ran this command</p>
  
  <pre><code>aws emr add-steps --cluster-id j-XXXXXX --steps "Type=spark,Name=Test,Args=[--deploy-mode,cluster,--master,yarn,--conf,spark.yarn.submit.waitAppCompletion=false,--num-executors,2,--executor-cores,2,--executor-memory,8g,s3://XXXXXX/emr-test.py,s3://XXXXXX/emr-in,s3://XXXXXX/emr-out],ActionOnFailure=CONTINUE"
  </code></pre>
  
  <p>My logs look like: </p>
  
  <pre><code>2019-05-24T04:24:12.718Z INFO Ensure step 4 jar file command-runner.jar
  2019-05-24T04:24:12.718Z INFO StepRunner: Created Runner for step 4
  INFO startExec 'hadoop jar /var/lib/aws/emr/step-runner/hadoop-jars/command-runner.jar spark-submit --deploy-mode cluster --master yarn --conf spark.yarn.submit.waitAppCompletion=false --num-executors 2 --executor-cores 2 --executor-memory 8g s3://XXXXXX/emr-test.py s3://XXXXXX/emr-in s3://XXXXXX/emr-out'
  INFO Environment:
    PATH=/sbin:/usr/sbin:/bin:/usr/bin:/usr/local/sbin:/opt/aws/bin
    LESS_TERMCAP_md=[01;38;5;208m
    LESS_TERMCAP_me=[0m
    HISTCONTROL=ignoredups
    LESS_TERMCAP_mb=[01;31m
    AWS_AUTO_SCALING_HOME=/opt/aws/apitools/as
    UPSTART_JOB=rc
    LESS_TERMCAP_se=[0m
    HISTSIZE=1000
    HADOOP_ROOT_LOGGER=INFO,DRFA
    JAVA_HOME=/etc/alternatives/jre
    AWS_DEFAULT_REGION=ap-southeast-1
    AWS_ELB_HOME=/opt/aws/apitools/elb
    LESS_TERMCAP_us=[04;38;5;111m
    EC2_HOME=/opt/aws/apitools/ec2
    TERM=linux
    runlevel=3
    LANG=en_US.UTF-8
    AWS_CLOUDWATCH_HOME=/opt/aws/apitools/mon
    MAIL=/var/spool/mail/hadoop
    LESS_TERMCAP_ue=[0m
    LOGNAME=hadoop
    PWD=/
    LANGSH_SOURCED=1
    HADOOP_CLIENT_OPTS=-Djava.io.tmpdir=/mnt/var/lib/hadoop/steps/s-31Z3K7X3KH3N6/tmp
    _=/etc/alternatives/jre/bin/java
    CONSOLETYPE=serial
    RUNLEVEL=3
    LESSOPEN=||/usr/bin/lesspipe.sh %s
    previous=N
    UPSTART_EVENTS=runlevel
    AWS_PATH=/opt/aws
    USER=hadoop
    UPSTART_INSTANCE=
    PREVLEVEL=N
    HADOOP_LOGFILE=syslog
    PYTHON_INSTALL_LAYOUT=amzn
    HOSTNAME=ip-172-31-128-120
    HADOOP_LOG_DIR=/mnt/var/log/hadoop/steps/s-31Z3K7X3KH3N6
    EC2_AMITOOL_HOME=/opt/aws/amitools/ec2
    SHLVL=5
    HOME=/home/hadoop
    HADOOP_IDENT_STRING=hadoop
  INFO redirectOutput to /mnt/var/log/hadoop/steps/s-31Z3K7X3KH3N6/stdout
  INFO redirectError to /mnt/var/log/hadoop/steps/s-31Z3K7X3KH3N6/stderr
  INFO Working dir /mnt/var/lib/hadoop/steps/s-31Z3K7X3KH3N6
  INFO ProcessRunner started child process 16829 :
  hadoop   16829  4392  0 04:24 ?        00:00:00 bash /usr/lib/hadoop/bin/hadoop jar /var/lib/aws/emr/step-runner/hadoop-jars/command-runner.jar spark-submit --deploy-mode cluster --master yarn --conf spark.yarn.submit.waitAppCompletion=false --num-executors 2 --executor-cores 2 --executor-memory 8g s3://XXXXXX/emr-test.py s3://XXXXXX/emr-in s3://XXXXXX/emr-out
  2019-05-24T04:24:16.724Z INFO HadoopJarStepRunner.Runner: startRun() called for s-31Z3K7X3KH3N6 Child Pid: 16829
  INFO Synchronously wait child process to complete : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
  INFO waitProcessCompletion ended with exit code 0 : hadoop jar /var/lib/aws/emr/step-runner/hadoop-...
  INFO total process run time: 14 seconds
  2019-05-24T04:24:28.813Z INFO Step created jobs: 
  2019-05-24T04:24:28.813Z INFO Step succeeded with exitCode 0 and took 14 seconds
  </code></pre>