title: Too slow reading CSV from HDFS into Pyspark dataframe
body: <p>I have developed a Spark-HDFS cluster with (5 workers and 5 datanodes) apart from the corresponding Spark master and Namenode.</p>
  
  <p>I'm programming in pyspark using the Jupyter Notebook which is initialized with <code>--executor-memory 30G --driver-memory 20G</code>. 
  The problem comes when i try to load a 348MB csv file from my HDFS into a pyspark DataFrame that it takes aprox. 15s</p>
  
  <p>I think that it's quite a lot of time for this small file. The code I'm using is:</p>
  
  <pre><code>sqlContext = SQLContext(sc)
  df1 = sqlContext.read.load("hdfs://namenode:8020/user/***/*****.csv", format='com.databricks.spark.csv',
  header='true',
  sep=';',
  inferSchema='true')
  </code></pre>
  
  <p>I've considered to load it into a RDD but since DataFrames are developed for this I think it fits better. Moreover, after loading this CSV I have to join another CSV which is just a column (20MB). </p>