title: Apache Spark - Speed malfuncition read.csv()
body: <p>I'm trying to perform machine learning on a 20 GB dataset which is in <code>.csv</code> format.
  One of the things that is advertised about spark is the speed. I'm a bit new to spark.
  In a pyspark environment if I simple perform a <code>spark.read.csv(file.csv)</code> it takes about 5 seconds on NFS and 1.5 seconds over HDFS. The problem with this the headers are labeled _C1, _C2, _C3, instead of using the actual headers of the dataset.
  So I thought I'd try the following to get the dataset to read:</p>
  <pre><code>spark.read.csv(headers= &quot;true&quot;, inferSchema=&quot;true&quot;, path= file.csv)
  </code></pre>
  <p>the dataframe provides the appropriate schemas, however this takes 10 minutes over NFS and longer on HDFS which is slower than Pandas. Is there a configuration setting I need to set to make it go faster.
  I've tried this datasets on a dask environment which takes only 4 seconds and the dask dataframe provides all the appropriate labels and headers, but my memory card doesn't have enough memory to load all the data which is why I can't use this option.</p>