title: Making massive numbers of HTTP requests in Python
body: <p>I'm trying to test a web application, part of this involves making ~10K requests, taking the few &lt;1K that return <code>200 OK</code> and going through their data. The webapp is buggy and there are false positives, so each 200 OK needs to be at least triple-checked.</p>
  
  <p>Working in Python I was trying to do this with threading and urllib, but on linux I get thread errors after ~920 threads. (My theory is it's <code>/proc/sys/kernel/threads-max</code> divided by thirty which is eerily accurate, but it's perturbing that each thread would register as 30 threads with the os). In any case, I'm looking for a good solution for this task. I've looked into Twisted, but it seems like I will still be bound by threading.</p>
  
  <p>Any ideas?</p>