title: Spark OutOfMemoryError: Java heap space on Simple S3 List Code
body: <p>I have an S3 bucket of 15million tiny files. The files have the same prefix and a UUID suffix. In an attempt to parallelize the list needed to read the files into an RDD, I'm creating a set of the ~1260 possible first two letters and sending a separate list request for each. However, in trying to generate the RDD of a list of keys, I run into the classic OutofMemory Error. Here's the code, which seems relatively simple - why am I running into these errors?</p>
  
  <pre><code>#Create prefixes:
  import string, itertools
  prefix_chars=range(0,10)+list(string.ascii_lowercase)
  prefixes=[]
  for i in itertools.permutations(prefix_chars, 2): 
      prefix=''
      for x in i: prefix=prefix+str(x)
      prefixes.append(prefix)
  
  def listBucket(bucket_name, file_prefix):
      import boto
      s3 = boto.connect_s3()
      bucket = s3.get_bucket(bucket_name)
      keylist=[]
      for i in bucket.list(file_prefix): keylist.append(i)
      return keylist
  
  getKeys=sc.parallelize(prefixes).repartition(len(prefixes)).flatMap(lambda x: listBucket(bucket,file_prefix)).collect()
  </code></pre>
  
  <p>I'm starting spark with the following conf:</p>
  
  <pre><code>conf = (SparkConf()
           .set("spark.executor.instances",50)
           .set("spark.executor.cores",10)
           .set("spark.executor.memory", "22g")
           .set("spark.akka.frameSize","100")
           .set("spark.driver.maxResultSize","24g")
           .set("spark.driver.memory","40g"))
  
  sc = SparkContext(conf = conf)
  </code></pre>
  
  <p>I'm running on EMR, Spark version 1.3 (for bootstrap dependencies, I had to use an older version of EMR and Spark).</p>