title: Pyspark program failed to run in OOZIE due to import error
body: <p>I am trying to run a simple PySpark program in Oozie and I am failing to run it successfully. I have a simple PySpark program which loads a file on HDFS in RDD. I convert that RDD to a DATAFRAME and then convert it PANDAS. I am getting an error for importing PANDAS.
  I have anaconda python distribution installed on my Cloudera VM version 5.4.2. I have the Anaconda dir (<code>/home/cloudera/anaconda/bin/</code>) added to my systems <code>CLASSPATH</code>. the following is the output of <code>echo $PATH</code> statement:</p>
  
  <blockquote>
    <p>/home/cloudera/anaconda/bin:/home/cloudera/anaconda/bin:/usr/local/firefox:/sbin:/usr/java/jdk1.7.0_67-cloudera/bin:/usr/local/apache-ant/apache-ant-1.9.2/bin:/usr/local/apache-maven/apache-maven-3.0.4/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cloudera/bin</p>
  </blockquote>
  
  <p>The program runs fine via CLI but it is failing when I try to run it via OOZIE job. </p>
  
  <p>The following is the error log I get:</p>
  
  <blockquote>
    <p>Stdoutput /usr/lib/spark/python/pyspark/sql/context.py:156: UserWarning:   Using RDD of dict to inferSchema is deprecated,please use pyspark.sql.Row instead<br>
    Stdoutput   warnings.warn("Using RDD of dict to inferSchema is deprecated,"
    Stdoutput Traceback (most recent call last):<br>
    Stdoutput   File "/home/cloudera/Dataframe/apps/shell/lib/test.py", line 48, in <br>
    Stdoutput     cleand = inputRDD.toPandas()<br>
    Stdoutput   File "/usr/lib/spark/python/pyspark/sql/dataframe.py", line 717, in toPandas<br>
    Stdoutput     import pandas as pd<br>
    Stdoutput ImportError: No module named pandas<br>
    Exit code of the Shell command 1<br>
    &lt;&lt;&lt; Invocation of Shell command completed &lt;&lt;&lt;<br>
    &lt;&lt;&lt; Invocation of Main class completed &lt;&lt;&lt;<br>
    Failing Oozie Launcher, Main class   [org.apache.oozie.action.hadoop.ShellMain], exit code [1]<br>
    Oozie Launcher failed, finishing Hadoop job gracefully<br>
    Oozie Launcher, uploading action data to HDFS sequence file:   hdfs://quickstart.cloudera:8020/user/cloudera/oozie-oozi/0000000-150724190306010-oozie-oozi-W/shell-node--shell/action-data.seq<br>
    Oozie Launcher ends  </p>
  </blockquote>
  
  <p>Can anyone please help me how I can run the program successfully?</p>
  
  <p>Thank you.</p>