title: Passing parameters into dataproc pyspark job
body: <p>How do you pass parameters into the python script being called in a dataproc pyspark job submit?  Here is a cmd I've been mucking with:</p>
  
  <pre><code> gcloud dataproc jobs submit pyspark --cluster my-dataproc \ 
   file:///usr/test-pyspark.py \
   --properties=^:^p1="7day":p2="2017-10-01"
  </code></pre>
  
  <p>This is the output returned:</p>
  
  <pre><code> Job [vvvvvvv-vvvv-vvvv-vvvv-0vvvvvv] submitted. Waiting for job output...
   Warning: Ignoring non-spark config property: p2=2017-10-01 
   Warning: Ignoring non-spark config property: p1=7day 
   Found script=/usr/test-pyspark.py 
   Traceback (most recent call last):   
     File "/usr/test-pyspark.py", line 52, in &lt;module&gt;
       print(sys.argv[1]) 
   IndexError: list index out of range`
  </code></pre>
  
  <p>Clearly doesn't recognize the 2 params I'm trying to pass in.  I also tried:</p>
  
  <pre><code>me@my-dataproc-m:~$  gcloud dataproc jobs submit pyspark --cluster=my-dataproc test-pyspark.py 7day 2017-11-01
  </code></pre>
  
  <p>But that returned with:</p>
  
  <pre><code>ERROR: (gcloud.dataproc.jobs.submit.pyspark) unrecognized arguments:
    7day
    2017-11-01
  </code></pre>
  
  <p>The pattern I use to pass params with the hive jobs doesn't work for pyspark.</p>
  
  <p>Any help appreciated!</p>
  
  <p>Thanks,
  Melissa</p>