title: How can I make smaller temporary files?
body: <p>So I am writing this program which creates a picture of the mandelbrot set, and I have been incrementally making it better. right now, each process that is spawned writes some data to a temporary file, which is used later on to put the picture together. Right now, however, the temporary files are ALOT bigger than the actual picture is itself, and I don't have have any ideas on how to make them smaller. How do I efficiently write integer data to a file, and get it back? I intend to eventual make this very scalable, so I would need to be able to write arbitrarily long integers for the pixel indices, but the color data is always going to be three integers that have a max value of 255. Here is my code:</p>
  
  <pre><code>import multiprocessing
  
  def pixproc(y0, yn, xsteps, ysteps, fname):
      XMIN, YMIN = -2., -1.
      XLEN, YLEN = 3, 2
      with open(fname, 'w') as f:
          for y in xrange(y0, yn):
              print y
              for x in xrange(xsteps):
                  c=complex(XMIN + XLEN*(1.*x/xsteps),
                            YMIN + YLEN*(1.*y/ysteps))
                  k=c
                  for i in xrange(256):
                      k = k*k + c
                      if abs(k)&gt;2: break
                  if 0&lt;i&lt;32:
                      #print 'Success!', i
                      print &gt;&gt;f, x, y, 8*i, 0, 0 #This is that part of
                  if 32&lt;=i&lt;255:                  #my code that I am trying
                      #print 'Success!', i       #to improve. The rest of 
                      print &gt;&gt;f, x, y, 255, i, i #the code is given for context
      return                                     #and isn't relevant to my question
  
  
  def main(xsteps, ysteps):
      pool = multiprocessing.Pool()
      n = multiprocessing.cpu_count()
      step = height / n
      fnames = ["temp" + str(i) for i in xrange(n)]
      for i in xrange(n):
          pool.apply_async(pixproc, 
                           (step*i, 
                            step*(i+1), 
                            xsteps, 
                            ysteps, 
                            fnames[i]))
      pool.close()
      pool.join()
      return fnames
  
  
  if __name__=="__main__":
      from PIL import Image
      import sys
      width, height = map(int, sys.argv[1:])
      picname = "mandelbrot1.png"
      fnames = main(width, height)
      im = Image.new("RGB", (width, height))
      pp = im.load()
      for name in fnames:
          with open(name) as f:
              for line in f:
                  line = map(int, line.rstrip('\n').split(' '))
                  pp[line[0], line[1]] = line[2], line[3], line[4]
      im.save(picname)
  </code></pre>
  
  <p>When I try to make a picture that is 3000x2000, the actual picture is 672 KB, but the temporary files are both close to 30 MB! Can someone suggest a better way to store the data in files? (The important part is in the function pixproc)</p>