title: How to fetch data from S3 to Dataframe in Pyspark (Jupyter) quicker?
body: <p>Currently with default pyspark ( jupyter ) configurations, the s3 data( about 20 gb Parquet files) take 2.5 hours to read into a dataframe.</p>
  
  <p>However, When I changed the driver memory and executor memory to 7g strangely the task of read keeps failing with exception printed below. Did not modify anything in the code.</p>
  
  <ol>
  <li>What is causing the timeout?</li>
  <li>How else to make the DF read faster? It is not feasible to wait 2.5 hours to load data into a dataframe.</li>
  </ol>
  
  <p>Code : </p>
  
  <pre><code>    from pyspark import SparkContext
      from pyspark.sql import SQLContext
      from pyspark import SparkConf
      from pyspark.sql import SparkSession
      import pandas
  
      # SparkConf().set("spark.jars.packages","org.apache.hadoop:hadoop-aws:3.0.0-alpha3")
      sc = SparkContext.getOrCreate()
  
      sc._jsc.hadoopConfiguration().set("fs.s3n.awsAccessKeyId", 'A')
      sc._jsc.hadoopConfiguration().set("fs.s3n.awsSecretAccessKey", 's')
  
      sqlContext = SQLContext(sc)
      print(sc._conf.getAll())
      df = sqlContext.read.parquet("s3n://sm/data/scor/alpha/2016/parquet/part-*.parquet")
      df.count()
  </code></pre>
  
  <p>Error :</p>
  
  <pre><code>&lt;pre&gt; Py4JJavaError: An error occurred while calling o51.count.
  : org.apache.spark.SparkException: Job aborted due to stage failure: Task 3 in stage 3.0 failed 1 times, most recent failure: Lost task 3.0 in stage 3.0 (TID 362, localhost, executor driver): java.net.SocketTimeoutException: Read timed out
      at java.net.SocketInputStream.socketRead0(Native Method)
      at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
      at java.net.SocketInputStream.read(SocketInputStream.java:170)
      at java.net.SocketInputStream.read(SocketInputStream.java:141)
      at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
      at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:593)
      at sun.security.ssl.InputRecord.read(InputRecord.java:532)
  </code></pre>
  
  <p></p>
  
  <p>Pyspark instance started using : </p>
  
  <pre><code>     pyspark --jars "/opt/apps/spark-2.1.1-bin-hadoop2.7/jars/hadoop-aws-3.0.0-alpha3.jar,/opt/apps/spark-2.1.1-bin-hadoop2.7/jars/aws-java-sdk-sts-1.11.150.jar,/opt/apps/spark-2.1.1-bin-hadoop2.7/jars/aws-java-sdk-s3-1.11.150.jar,/opt/apps/spark-2.1.1-bin-hadoop2.7/jars/aws-java-sdk-core-1.11.150.jar,/opt/apps/spark-2.1.1-bin-hadoop2.7/jars/httpcore-4.4.6.jar,/opt/apps/spark-2.1.1-bin-hadoop2.7/jars/httpclient-4.5.3.jar" --driver-memory 7g --executor-memory 7g
  </code></pre>