title: Cannot load main class from JAR file in Spark Submit
body: <p>I am trying to run a Spark job. This is my shell script, which is located at /home/full/path/to/file/shell/my_shell_script.sh:</p>
  
  <pre><code>confLocation=../conf/my_config_file.conf &amp;&amp;
  executors=8 &amp;&amp;
  memory=2G &amp;&amp;
  entry_function=my_function_in_python &amp;&amp;
  dos2unix $confLocation &amp;&amp;
  spark-submit \
          --master yarn-client \
          --num-executors $executors \
          --executor-memory $memory \
          --py-files /home/full/path/to/file/python/my_python_file.py $entry_function $confLocation
  </code></pre>
  
  <p>When I run this, I get an error that says:</p>
  
  <blockquote>
    <p>Error: Cannot load main class from JAR file: /home/full/path/to/file/shell/my_function_in_python</p>
  </blockquote>
  
  <p>My impression here is that it is looking in the wrong place (the python file is located in the python directory, not the shell directory).</p>