title: External packages (jars) in pyspark shell - How To
body: <p>What is the proper way to include external packages (jars) in a pyspark shell?</p>
  <p>I am using pyspark from a jupyter notebook.</p>
  <p>I would like to read from kafka using spark, via the <code>spark-sql-kafka</code> library, as explained here: <a href="https://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html#deploying" rel="nofollow noreferrer">https://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html#deploying</a>.</p>
  <p>I am trying to import the library via the <code>--packages</code> option, set in the environment variable <code>PYSPARK_SUBMIT_ARGS</code>.</p>
  <p>But</p>
  <ul>
  <li>I am not sure about the exact version and name of the package to use,</li>
  <li>I don't know whether I also need to include spark-streaming or not, whether I have to specify some repository with <code>--repositories</code> or not,</li>
  <li>I don't know whether it's better to download the jar and specify local paths (do they have to be on the machine where jupyter is running, or on the machine where yarn is running? I'm using <code>--master yarn</code> and <code>--deploy-mode client</code>) or to rely on <code>--packages</code></li>
  <li>I don't know whether options specified after <code>pyspark-shell</code> in <code>PYSPARK_SUBMIT_ARGS</code> are left out or not (If I try to specify <code>--packages</code> options before <code>pyspark-shell</code> I can't instantiate the spark context at all)</li>
  <li>How can I check whether some package was correctly downloaded and is available to be used</li>
  <li>I don't know what is the route that such downloaded jars (or jars in general) take. How many times are they replicated? Do they pass through the driver? Do these things change if I'm using a cluster manager as YARN? Do they change if I'm using everything from a spark-shell in a jupyter notebook?</li>
  </ul>
  <p>Resources I read so far:</p>
  <ul>
  <li><p>Docs and guides:</p>
  <ul>
  <li><p><a href="https://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html#deploying" rel="nofollow noreferrer">Structured streaming kafka integration</a></p>
  </li>
  <li><p><a href="https://spark.apache.org/docs/latest/configuration.html" rel="nofollow noreferrer">Spark configuration</a></p>
  </li>
  <li><p><a href="https://spark.apache.org/docs/latest/submitting-applications.html" rel="nofollow noreferrer">Spark submitting applications</a></p>
  </li>
  <li><p><a href="https://zeppelin.apache.org/docs/0.8.2/setup/deployment/spark_cluster_mode.html#spark-on-yarn-mode" rel="nofollow noreferrer">Zeppelin notebook on cluster mode documentation</a></p>
  </li>
  <li><p><a href="https://jaceklaskowski.gitbooks.io/mastering-apache-spark/spark-submit.html" rel="nofollow noreferrer">jaceklakowski on spark-submit, config options, notebooks, spark context, etc.</a></p>
  </li>
  <li><p><a href="https://spark.apache.org/docs/latest/api/python/pyspark.html#pyspark.SparkContext" rel="nofollow noreferrer">pyspark docs</a></p>
  </li>
  <li><p><a href="https://docs.cloudera.com/documentation/enterprise/5-7-x/topics/spark_building.html" rel="nofollow noreferrer">cloudera building spark applications</a></p>
  </li>
  <li><p><a href="https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html" rel="nofollow noreferrer">Spark structured streaming programming guide</a></p>
  </li>
  </ul>
  </li>
  <li><p>Examples:</p>
  <ul>
  <li><p><a href="https://mtpatter.github.io/bilao/notebooks/html/01-spark-struct-stream-kafka.html" rel="nofollow noreferrer">notebook with spark structured streaming and spark sql kafka</a></p>
  </li>
  <li><p><a href="https://dzone.com/articles/basic-example-for-spark-structured-streaming-amp-k" rel="nofollow noreferrer">dzone basic example for spark structured streaming</a></p>
  </li>
  </ul>
  </li>
  <li><p>Issues and questions:</p>
  <ul>
  <li><p><a href="https://stackoverflow.com/questions/42906068/spark-2-1-structured-streaming-using-kakfa-as-source-with-python-pyspark">structured streaming using kafka as source with python</a></p>
  </li>
  <li><p><a href="https://github.com/jupyter/notebook/issues/743" rel="nofollow noreferrer">Java gateway process exited before sending the driver its port number</a></p>
  </li>
  <li><p><a href="https://stackoverflow.com/questions/45522153/does-pyspark-support-spark-streaming-kafka-0-10-lib">does pyspark support spark streaming kafka 0-10-lib</a></p>
  </li>
  <li><p><a href="https://stackoverflow.com/questions/35946868/adding-custom-jars-to-pyspark-in-jupyter-notebook">adding custom jars to pyspark in jupyter notebook</a></p>
  </li>
  </ul>
  </li>
  <li><p>Repositories:</p>
  <ul>
  <li><p><a href="https://search.maven.org/artifact/org.apache.spark/spark-sql-kafka-0-10_2.10/2.2.3/jar" rel="nofollow noreferrer">spark-sql-kafka on search.maven.org</a></p>
  </li>
  <li><p><a href="https://mvnrepository.com/artifact/org.apache.spark/spark-sql-kafka-0-10_2.11/2.3.2.3.1.0.6-1" rel="nofollow noreferrer">spark-sql-kafka on MVNrepository</a></p>
  </li>
  </ul>
  </li>
  </ul>