title: PySpark : importing jars on runtime?
body: <p>I am trying to connect pyspark to mongodb.
  The cluster is already set up, so I need a dynamic import of mongo drivers.</p>
  
  <p>I have followed the docs (here and here <a href="https://spark.apache.org/docs/latest/configuration.html" rel="nofollow noreferrer">spark.jars</a>) and tried : </p>
  
  <pre><code>spark = SparkSession.builder\
              .config("spark.mongodb.input.uri" , "mongodb://127.0.0.1/test.coll")\
              .config("spark.mongodb.output.uri", "mongodb://127.0.0.1/test.coll")\
              .config("spark.jars.packages"     , "org.mongodb.spark:mongo-spark-connector_2.11:2.2.1")\
              .getOrCreate()
  </code></pre>
  
  <p>But this : </p>
  
  <pre><code>df = spark.read.format("com.mongodb.spark.sql.DefaultSource").option("uri","mongodb://127.0.0.1/people.contacts").load()
  </code></pre>
  
  <p>gives me an error : </p>
  
  <pre><code>Py4JJavaError: An error occurred while calling o326.load.
  : java.lang.NoClassDefFoundError: com/mongodb/ConnectionString
      at com.mongodb.spark.config.MongoCompanionConfig$$anonfun$4.apply(MongoCompanionConfig.scala:278)
      at com.mongodb.spark.config.MongoCompanionConfig$$anonfun$4.apply(MongoCompanionConfig.scala:278)
  </code></pre>
  
  <p>From all the tutorials on the net it is unclear if we can upload packages during runtime, even if <a href="https://spark.apache.org/docs/latest/submitting-applications.html#advanced-dependency-management" rel="nofollow noreferrer">spark doc</a>  seems to say it is possible.</p>
  
  <p>The examples heavily relie on psark-submit". But I am myself using jupyter notebook, without spark included => I load it through the "findspark" function, and then declare my context.</p>