Take the 2-minute tour ×

I would love to have this programm improve a lot in speed. It reads +- 12000 pages in 10 minutes. I was wondering if there is something what would help a lot to the speed? I hope you guys know some tips. I am supposed to read +- millions of pages... so that would take way too long :( Here is my code:

from eventlet.green import urllib2                          
import httplib                                              
import time                                                 
import eventlet   

# Create the URLS in groups of 400 (+- max for eventlet)                                    
def web_CreateURLS():
    print str(str(time.asctime( time.localtime(time.time()) )).split(" ")[3])
    for var_indexURLS in xrange(0, 2000000, 400):
        var_URLS = []
        for var_indexCRAWL in xrange(var_indexURLS, var_indexURLS+400):

# Return the HTML Source per URL
def web_ReturnHTML(url):
        return [urllib2.urlopen(url[0]).read(), url[1]]
    except urllib2.URLError:
        print "UrlError"

# Analyse the HTML Source
def web_ScanURLS(var_URLS):
    pool = eventlet.GreenPool()
        for var_HTML in pool.imap(web_ReturnHTML, var_URLS):
               # do something etc..
    except TypeError: pass

share|improve this question
You appear to be fetching 20 urls per second, each request is taking on average 50ms. That page appears to be 81K. So your current network throughput is 1.58mbps. To achieve a 100 fold increase (1.2 million in 10 minutes) you would have to make 100 times as many requests. That would require a 158mbps connection and you would be making 2000 requests per second. This doesn't seem possible to me. –  Nathan Villaescusa Oct 6 '12 at 20:58
Possible...just not on an ordinary residential internet connection. Plus, it might be construed as a DoS attack on the webserver. –  nneonneo Oct 6 '12 at 22:36

1 Answer 1

I like using greenlets.. but I often benefit from using multiple processes spread over lots of systems.. or just one single system letting the OS take care of all the checks and balances of running multiple processes.

Check out ZeroMQ at http://zeromq.org/ for some good examples on how to make a dispatcher with a TON of listeners that do whatever the dispatcher says. Alternatively check out execnet for a method of quickly getting started with executing remote or local tasks in parallel.

I also use http://spread.org/ a lot and have LOTS of systems listening to a common spread daemon.. it's a very useful message bus where results can be pooled back to and dispatched from a single thread pretty easily.

And then of course there is always redis pub/sub or sync. :)

"Share the load"

share|improve this answer

Your Answer


