title: How to run py job spark-submit in yarn cluster mode including pip packages (e.g numpy) on hdfs
body: <p>Usually when running such spark job: </p>
  
  <pre><code>spark-submit --master yarn --deploy-mode cluster \
               --driver-memory &lt;xxG&gt; \
               --conf &lt;all relevant configs&gt; hdfs:///test-numpy.py
  </code></pre>
  
  <p>We need to have the numpy pip package installed on each node of the hadoop cluster.</p>
  
  <p>Since there are lots of pip packages used by our code, we prefer to have a zip file to include all of our dependencies and load it to the hdfs and use it from there, without the need to change something in the python code.</p>
  
  <p>How can it be done?
  We tried to create a <code>.zip</code> file with virtualenv and to run it as follow:</p>
  
  <pre><code>spark-submit --master yarn --deploy-mode cluster \
               --driver-memory &lt;xxG&gt; \
               --conf &lt;all relevant configs&gt; \
               --py-files hdfs:///dependencies.zip hdfs:///test-numpy.py
  </code></pre>
  
  <p>BUT the job failed with this import error: </p>
  
  <pre><code>ImportError: No module named numpy
  </code></pre>
  
  <p>Is it the right way to use pip packages? </p>