title: What to do with "WARN TaskSetManager: Stage contains a task of very large size"?
body: <p>I use spark 1.6.1. </p>
  
  <p>My spark application reads more than 10000 parquet files stored in s3. </p>
  
  <pre><code>val df = sqlContext.read.option("mergeSchema", "true").parquet(myPaths: _*)
  </code></pre>
  
  <p><code>myPaths</code> is an <code>Array[String]</code> that contains the paths of the 10000 parquet files. Each path is like this <code>s3n://bucketname/blahblah.parquet</code></p>
  
  <p>Spark warns message like below. </p>
  
  <blockquote>
    <p>WARN TaskSetManager: Stage  4 contains a task of very large size
    (108KB). The maximum recommended task size is 100KB.</p>
  </blockquote>
  
  <p>Spark has managed to run and finish the job anyway but I guess this can slow down spark processing job.</p>
  
  <p>Does anybody has a good suggestion about this problem?</p>