title: Problem importing modules from a .zip file (created in python using zipfile package) with --py-files on an EMR in Spark
body: <p>I am trying to archive my application in my test file to spark submit on an EMR cluster like this:</p>
  
  <p>Folder structure of modules:</p>
  
  <pre><code>app
  --- module1
  ------ test.py
  ------ test2.py
  --- module2
  ------ file1.py
  ------ file2.py
  </code></pre>
  
  <p>Zip function I'm calling from my tests</p>
  
  <pre><code>import zipfile
  import os
  
  def zip_deps():
      # make zip
  
      module1_path = '../module1'
      module2_path = '../module2'
      try:
          with zipfile.ZipFile('deps.zip', 'w', zipfile.ZIP_DEFLATED) as zipf:
              info = zipfile.ZipInfo(module1_path +'/')
              zipf.writestr(info, '')
              for root, dirs, files in os.walk(module1_path):
                  for d in dirs:
                      info = zipfile.ZipInfo(os.path.join(root, d)+'/')
                      zipf.writestr(info, '')
                  for file in files:
                      zipf.write(os.path.join(root, file),os.path.relpath(os.path.join(root, file)))
  
              info = zipfile.ZipInfo(module2_path +'/')
              zipf.writestr(info, '')
              for root, dirs, files in os.walk(module2_path):
                  for d in dirs:
                      info = zipfile.ZipInfo(os.path.join(root, d)+'/')
                      zipf.writestr(info, '')
                  for file in files:
                      zipf.write(os.path.join(root, file),os.path.relpath(os.path.join(root, file)))
      except:
          print('Unexpected error occurred while creating file deps.zip')
      zipf.close()
  
  </code></pre>
  
  <p>The deps.zip is created correctly, as far as I can see it zips all the files I want, and each module folder is at the base level of the zip. 
  In fact the exact zip created using:
  <code>zip -r deps.zip module1 module2</code>
  is the same structure and THIS works when I spark submit it with </p>
  
  <pre><code>spark-submit --py-files deps.zip driver.py 
  </code></pre>
  
  <p>Error from EMR:</p>
  
  <pre><code>Traceback (most recent call last):
    File "driver.py", line 6, in &lt;module&gt;
      from module1.test import test_function
  ModuleNotFoundError: No module named 'module1'
  </code></pre>
  
  <p>FWIW I also tried zipping using a subprocess with the following commands and I got the same error on my EMR in spark</p>
  
  <pre><code>os.system("zip -r9 deps.zip ../module1")
  os.system("zip -r9 deps.zip ../module2")
  </code></pre>
  
  <p>I don't know why a zip file created in python would be different than outside of python, but I've spent the last few days on this and hopefully someone can help!</p>
  
  <p>Thanks!!</p>