title: No such file or directory in spark cluster mode
body: <p>I am writing a spark-streaming application using pyspark which basically process the data.</p>
  
  <p>Inshort packaging overview:
  This application contains several modules and some config files which are non .py files <strong>(ex:.yaml or .json)</strong>.</p>
  
  <p>I am packaging this entire application in package.zip file and submitting this package.zip to spark.</p>
  
  <p>Now the problem is when i issue the spark-submit command in yarn cluster mode. I get IOError. Below is stacktrace</p>
  
  <pre><code> Traceback (most recent call last):
     File "main/main.py", line 10, in &lt;module&gt;
       import logger.logger
     File "package.zip/logger/logger.py", line 36, in get_logger
     IOError: [Errno 2] No such file or directory: 'logger/config.yaml'
  </code></pre>
  
  <p>Spark-Command :
  <strong>spark-submit --master yarn-cluster --py-files package.zip main/main.py</strong></p>
  
  <p>But when I am submitting job in yarn-client mode the application works as expected.</p>
  
  <p>My understanding: </p>
  
  <ol>
  <li>When I submit the job in client mode the <strong>spark driver</strong> runs in same machine where I have issued the command. And the package is distributed across all nodes.</li>
  <li>And when I issue the command in cluster mode the both <strong>spark driver</strong> and <strong>application master</strong> runs in single node(which is not client who submitted code.) and still package is distribute to all nodes in cluster.</li>
  </ol>
  
  <p>In both the cases package.zip is available to all nodes then why is that only py files are getting loaded and non py files are failed to load in cluster mode. </p>
  
  <p>Can any one please help me to understand the situation here and resolve the problem? </p>
  
  <p><strong>Updated--</strong></p>
  
  <p><strong>Observations</strong></p>
  
  <p>In <strong>Client Mode</strong> The zipped package is unzipped in the path where driver script is running.
  Where as in <strong>Cluster Mode</strong> the zip package shared across all node but not unzipped. 
  Here do I need to unzip package in all nodes ?
  <strong>Is there any way to tell spark to unzip package in worker node?</strong></p>