title: Using Postgresql JDBC source with Apache Spark on EMR
body: <p>I have existing EMR cluster running and wish to create DF from Postgresql DB source.</p>
  
  <p>To do this, it seems you need to modify the spark-defaults.conf with the updated <code>spark.driver.extraClassPath</code> and point to the relevant PostgreSQL JAR that has been already downloaded on master &amp; slave nodes, <strong>or</strong> you can add these as arguments to a spark-submit job.</p>
  
  <p>Since I want to use existing Jupyter notebook to wrangle the data, and not really looking to relaunch cluster, what is the most efficient way to resolve this?</p>
  
  <p>I tried the following:</p>
  
  <ol>
  <li><p>Create new directory (/usr/lib/postgresql/ on master and slaves and copied PostgreSQL jar to it. (postgresql-9.41207.jre6.jar)</p></li>
  <li><p>Edited spark-default.conf to include wildcard location</p>
  
  <pre><code>spark.driver.extraClassPath  :/usr/lib/postgresql/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/$
  </code></pre></li>
  <li><p>Tried to create dataframe in Jupyter cell using the following code:</p>
  
  <pre class="lang-py prettyprint-override"><code>SQL_CONN = "jdbc:postgresql://some_postgresql_db:5432/dbname?user=user&amp;password=password"
  spark.read.jdbc(SQL_CONN, table="someTable", properties={"driver":'com.postgresql.jdbc.Driver'})
  </code></pre></li>
  </ol>
  
  <p>I get a Java error as per below:</p>
  
  <pre><code>Py4JJavaError: An error occurred while calling o396.jdbc.
  : java.lang.ClassNotFoundException: com.postgresql.jdbc.Driver
  </code></pre>
  
  <p>Help appreciated.</p>