title: Spark: sc.WholeTextFiles takes a long time to execute
body: <p>I have a cluster and I execute <code>wholeTextFiles</code> which should pull about a million text files who sum up to approximately <code>10GB</code> total
  I have one NameNode and two DataNode with <code>30GB</code> of RAM each, 4 cores each. The data is stored in <code>HDFS</code>.</p>
  
  <p>I don't run any special parameters and the job takes 5 hours to just read the data. Is that expected? are there any parameters that should speed up the read (spark configuration or partition, number of executors?)</p>
  
  <p>I'm just starting and I've never had the need to optimize a job before</p>
  
  <p><strong>EDIT:</strong> Additionally, can someone explain exactly how the wholeTextFiles function works? (not how to use it, but how it was programmed). I'm very interested in understand the partition parameter, etc. </p>
  
  <p><strong>EDIT 2:</strong> benchmark assessment</p>
  
  <p>So I tried repartition after the wholeTextFile, the problem is the same because the first read is still using the pre-defined number of partitions, so there are no performance improvements. Once the data is loaded the cluster performs really well... I have the following warning message when dealing with the data (for 200k files), on the wholeTextFile:</p>
  
  <pre><code>15/01/19 03:52:48 WARN scheduler.TaskSetManager: Stage 0 contains a task of very large size (15795 KB). The maximum recommended task size is 100 KB.
  </code></pre>
  
  <p>Would that be a reason of the bad performance? How do I hedge that?</p>
  
  <p>Additionally, when doing a saveAsTextFile, my speed according to Ambari console is 19MB/s. When doing a read with wholeTextFiles, I am at 300kb/s.....</p>
  
  <p>It seems that by increase the number of partitions in <code>wholeTextFile(path,partitions)</code>, I am getting better performance. But still only 8 tasks are running at the same time (my number of CPUs). I'm benchmarking to observe the limit...</p>