title: What is the fastest way to read database using PySpark?
body: <p>I am trying to read a table of a database using PySpark and SQLAlchamy as follows:</p>
  
  <pre class="lang-python prettyprint-override"><code>SUBMIT_ARGS = "--jars mysql-connector-java-5.1.45-bin.jar pyspark-shell"
  os.environ["PYSPARK_SUBMIT_ARGS"] = SUBMIT_ARGS
  sc = SparkContext('local[*]', 'testSparkContext')
  sqlContext = SQLContext(sc)
  
  t0 = time.time()
  database_uri =  'jdbc:mysql://{}:3306/{}'.format("127.0.0.1",&lt;db_name&gt;)
  dataframe_mysql = sqlContext.read.format("jdbc").options(url=database_uri, driver = "com.mysql.jdbc.Driver", dbtable = &lt;tablename&gt;, user= &lt;user&gt;, password=&lt;password&gt;).load()
  print(dataframe_mysql.rdd.map(lambda row :list(row)).collect())
  
  t1 = time.time()
  database_uri2 =  'mysql://{}:{}@{}/{}'.format(&lt;user&gt;,&lt;password&gt;,"127.0.0.1",&lt;db_name&gt;)
  engine = create_engine(database_uri2)
  connection = engine.connect()
  s = text("select * from {}.{}".format(&lt;db_name&gt;,&lt;table_name&gt;))
  result = connection.execute(s)
  for each in result:
       print(each)
  t2= time.time()
  
  print("Time taken by PySpark:", (t1-t0))
  print("Time taken by SQLAlchamy", (t2-t1))
  </code></pre>
  
  <p>This is the time taken to fetch some 3100 rows:</p>
  
  <pre><code>Time taken by PySpark: 12.326745986938477
  Time taken by SQLAlchamy: 0.21664714813232422
  </code></pre>
  
  <p>Why SQLAlchamy is outperforming PySpark? Is there any way to make this faster? Is there any error in my approach?</p>