title: Submit a Python project to Dataproc job
body: <p>I have a python project, whose folder has the structure</p>
  
  <pre><code>main_directory - lib - lib.py
                 - run - script.py
  </code></pre>
  
  <p><code>script.py</code> is </p>
  
  <pre><code>from lib.lib import add_two
  spark = SparkSession \
      .builder \
      .master('yarn') \
      .appName('script') \
      .getOrCreate()
  
  print(add_two(1,2))
  </code></pre>
  
  <p>and <code>lib.py</code> is</p>
  
  <pre><code>def add_two(x,y):
      return x+y
  </code></pre>
  
  <p>I want to launch as a Dataproc job in GCP. I have checked online, but I have not understood well how to do it. I am trying to launch the script with </p>
  
  <pre class="lang-sh prettyprint-override"><code>gcloud dataproc jobs submit pyspark --cluster=$CLUSTER_NAME --region=$REGION \
    run/script.py
  </code></pre>
  
  <p>But I receive the following error message:</p>
  
  <pre><code>from lib.lib import add_two
  ModuleNotFoundError: No module named 'lib.lib'
  </code></pre>
  
  <p>Could you help me on how I should do to launch the job on Dataproc? The only way I have found to do it is to remove the absolute path, making this change to <code>script.py</code>:</p>
  
  <pre><code> from lib import add_two
  </code></pre>
  
  <p>and the launch the job as </p>
  
  <pre class="lang-sh prettyprint-override"><code>gcloud dataproc jobs submit pyspark --cluster=$CLUSTER_NAME --region=$REGION \
    --files /lib/lib.py \
    /run/script.py
  </code></pre>
  
  <p>However, I would like to avoid the tedious process to list the files manually every time. </p>
  
  <p>Following the suggestion of @Igor, to pack in a zip file I have found that </p>
  
  <pre><code>zip -j --update -r libpack.zip /projectfolder/* &amp;&amp; spark-submit --py-files libpack.zip /projectfolder/run/script.py
  </code></pre>
  
  <p>works. However, this puts all files in the same root folder in libpack.zip, so if there were files with the same names in subfolders this would not work. </p>
  
  <p>Any suggestions?</p>