title: Python worker exited unexpectedly exception - when importing psycopyg within mapPartitions
body: <p>I am new to spark. So bear with me.</p>
  
  <p>Here is what I am trying to do:</p>
  
  <p>I read entries from CSV file and check if the entry is present in the database and insert if not present. I don't want to use <code>rdd.write.jdbc</code> option because i think it will write entire dataframe. </p>
  
  <p>I am using <code>mapPartition</code> and trying to initialise the postgres connection with <code>psycopg2</code> library as follows:</p>
  
  <pre><code>def save_to_db(records):
  import psycopg2
  from psycopg2.extensions import AsIs
  url = 'postgres://postgres:@127.0.0.1:5432/spark_learn'
  conn = psycopg2.connect(url)
  conn.autocommit = True
  cursor = conn.cursor()
  for record in records:
      columns = record.keys()
      values = [record[key] for key in columns]
      cursor.execute("INSERT INTO heroes (%s) VALUES %s", (AsIs(','.join(columns)), tuple(values)))
  return records
  </code></pre>
  
  <p>But I am getting the error as follows:</p>
  
  <pre><code>Caused by: org.apache.spark.SparkException: Python worker exited unexpectedly (crashed)
  at org.apache.spark.api.python.BasePythonRunner$ReaderIterator$$anonfun$3.applyOrElse(PythonRunner.scala:486)
  at org.apache.spark.api.python.BasePythonRunner$ReaderIterator$$anonfun$3.applyOrElse(PythonRunner.scala:475)
  at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
  at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRunner.scala:593)
  at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRunner.scala:571)
  at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:406)
  at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
  at scala.collection.Iterator$class.foreach(Iterator.scala:891)
  at org.apache.spark.InterruptibleIterator.foreach(InterruptibleIterator.scala:28)
  at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59)
  at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:104)
  at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:48)
  at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:310)
  at org.apache.spark.InterruptibleIterator.to(InterruptibleIterator.scala:28)
  at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:302)
  at org.apache.spark.InterruptibleIterator.toBuffer(InterruptibleIterator.scala:28)
  at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:289)
  at org.apache.spark.InterruptibleIterator.toArray(InterruptibleIterator.scala:28)
  at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$13.apply(RDD.scala:945)
  at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$13.apply(RDD.scala:945)
  at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2101)
  at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2101)
  at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
  at org.apache.spark.scheduler.Task.run(Task.scala:121)
  at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:402)
  at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
  at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:408)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  ... 1 more
  Caused by: java.io.EOFException
  at java.io.DataInputStream.readInt(DataInputStream.java:392)
  at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRunner.scala:578)
  ... 26 more
  </code></pre>
  
  <p>Pls help me in this.</p>