title: Spark using python: How to resolve Stage x contains a task of very large size (xxx KB). The maximum recommended task size is 100 KB
body: <p>I've just created python list of <code>range(1,100000)</code>.</p>
  
  <p>Using SparkContext done the following steps:</p>
  
  <pre><code>a = sc.parallelize([i for i in range(1, 100000)])
  b = sc.parallelize([i for i in range(1, 100000)])
  
  c = a.zip(b)
  
  &gt;&gt;&gt; [(1, 1), (2, 2), -----]
  
  sum  = sc.accumulator(0)
  
  c.foreach(lambda (x, y): life.add((y-x)))
  </code></pre>
  
  <p>Which gives warning as follows:</p>
  
  <p><strong>ARN TaskSetManager: Stage 3 contains a task of very large size (4644 KB). The maximum recommended task size is 100 KB.</strong></p>
  
  <p>How to resolve this warning? Is there any way to handle size? And also, will it affect the time complexity on big data?</p>