title: Gzip files with Spark
body: <p>I have a Spark job which takes several thousand files as input and downloads them from Amazon S3 and process them in the map phase, where each map step returns a string. I'd like to compress outputs to <code>.tar.gz</code> file and upload it to S3 afterwards. One way to do it is</p>
  
  <pre><code>outputs = sc.map(filenames).collect()
  for output in outputs:
      with tempfile.NamedTemporaryFile() as tar_temp:
          tar = tarfile.open(tar_temp.name, "w:gz")
          for output in outputs:
              with tempfile.NamedTemporaryFile() as output_temp:
                  output_temp.write(output)
                  tar.add(output_temp.name)
          tar.close()
  </code></pre>
  
  <p>the problem is that <code>outputs</code> don't fit into memory (but they fit on disk). Is there a way to save the outputs to master filesystem in map phase? Or perhaps use loop <code>for output in outputs</code> as a generator so that I don't have to load everything into memory?</p>