title: spark-submit with python entry points
body: <p>I have a script <code>wordcount.py</code><br>
  I used setuptools to create an entry point, named <code>wordcount</code>, so now I can call the command from anywhere in the system.<br>
  I am trying to execute it via spark-submit (command: <code>spark-submit wordcount</code>) but it is failing with the following error:</p>
  
  <p><code>Error: Cannot load main class from JAR file:/usr/local/bin/wordcount
  Run with --help for usage help or --verbose for debug output
  </code></p>
  
  <p>However the exact same command works fine when I provide the path to the python script (command: <code>spark-submit /home/ubuntu/wordcount.py</code>)</p>
  
  <p>Content of wordcount.py</p>
  
  <pre><code>import sys
  from operator import add
  
  from pyspark.sql import SparkSession
  
  def main(args=None):
      if len(sys.argv) != 2:
          print("Usage: wordcount &lt;file&gt;", file=sys.stderr)
          exit(-1)
  
      spark = SparkSession\
          .builder\
          .appName("PythonWordCount")\
          .getOrCreate()
  
      lines = spark.read.text(sys.argv[1]).rdd.map(lambda r: r[0])
      counts = lines.flatMap(lambda x: x.split(' ')) \
                    .map(lambda x: (x, 1)) \
                    .reduceByKey(add)
      output = counts.collect()
      for (word, count) in output:
          print("%s: %i" % (word, count))
  
      spark.stop()
  
  if __name__ == "__main__":
      main()
  </code></pre>
  
  <p>Do you know if there is a way to bypass this?<br>
  Thanks a lot in advance.</p>