title: How to reduce memory usage of threaded python code?
body: <p>I wrote about 50 classes that I use to connect and work with websites using mechanize and threading. They all work concurrently, but they don't depend on each other. So that means 1 class - 1 website - 1 thread. It's not particularly elegant solution, especially for managing the code, since lot of the code repeats in each class (but not nearly enough to make it into one class to pass arguments, as some sites may require additional processing of retrieved data in middle of methods - like 'login' - that others might not need). As I said, it's not elegant -- But it works. Needless to say I welcome all recommendations how to write this better without using 1 class for each website approach. Adding additional functionality or overall code management of each class is a daunting task.</p>
  
  <p>However, I found out, that each thread takes about 8MB memory, so with 50 running threads we are looking at about 400MB usage. If it was running on my system I wouldn't have problem with that, but since it's running on a VPS with only 1GB memory, it's starting to be an issue. Can you tell me how to reduce the memory usage, or are there any other way to to work with multiple sites concurrently?</p>
  
  <p>I used this quick test python program to test if it's the data stored in variables of my application that is using the memory, or something else. As you can see in following code, it's only processing sleep() function, yet each thread is using 8MB of memory.</p>
  
  <pre><code>from thread import start_new_thread
  from time import sleep
  
  def sleeper():
      try:
          while 1:
              sleep(10000)
      except:
          if running: raise
  
  def test():
      global running
      n = 0
      running = True
      try:
          while 1:
              start_new_thread(sleeper, ())
              n += 1
              if not (n % 50):
                  print n
      except Exception, e:
          running = False
          print 'Exception raised:', e
      print 'Biggest number of threads:', n
  
  if __name__ == '__main__':
      test()
  </code></pre>
  
  <p>When I run this, the output is:</p>
  
  <pre><code>50
  100
  150
  Exception raised: can't start new thread
  Biggest number of threads: 188
  </code></pre>
  
  <p>And by removing <code>running = False</code> line, I can then measure free memory using <code>free -m</code> command in shell:</p>
  
  <pre><code>             total       used       free     shared    buffers     cached
  Mem:          1536       1533          2          0          0          0
  -/+ buffers/cache:       1533          2
  Swap:            0          0          0
  </code></pre>
  
  <p>The actual calculation why I know it's taking about 8MB per thread is then simple by dividing dividing the difference of memory used before and during the the above test application is running, divided by maximum threads it managed to start.</p>
  
  <p>It's probably only allocated memory, because by looking at <code>top</code>, the python process uses only about 0.6% of memory.</p>