Take the 2-minute tour ×

I recently tried to speed up a little tool (which uses urllib2 to send a request to the (unofficial)twitter-button-count-url (> 2000 urls) and parses it´s results) with the multiprocessing module (and it´s worker pools). I read several discussion here about multithreading (which slowed the whole thing down compared to a standard, non-threaded version) and multiprocessing, but i could´t find an answer to a (probably very simple) question:

Can you speed up url-calls with multiprocessing or ain´t the bottleneck something like the network-adapter? I don´t see which part of, for example, the urllib2-open-method could be parallelized and how that should work...

EDIT: THis is the request i want to speed up and the current multiprocessing-setup:

 urls=["www.foo.bar", "www.bar.foo",...]

 def getTweets(self,urls):
    for i in urls:
            self.tw_que=urllib2.urlopen(tw_url %(i))
        except ValueError:
            print ....
    return self.tweets 

 if __name__ == '__main__':
    pool = multiprocessing.Pool(processes=4)            
    result = [pool.apply_async(getTweets(i,)) for i in urls]
    [i.get() for i in result]
share|improve this question
EDIT: Using 30 Processes, there is a significant speed-up (if my measures are not confounded/biased due tue connection limits, cache-realted stuff etc.) –  dorvak Aug 2 '11 at 20:59
how many processors for the 30 processes? @dorvak –  mgoldwasser Jun 19 at 21:26

4 Answers 4

up vote 3 down vote accepted

Take a look at a look at gevent and specifically at this example: concurrent_download.py. It will be reasonably faster than multiprocessing and multithreading + it can handle thousands of connections easily.

share|improve this answer

Ah here comes yet another discussion about the GIL. Well here's the thing. Fetching content with urllib2 is going to be mostly IO-bound. Native threading AND multiprocessing will both have the same performance when the task is IO-bound (threading only becomes a problem when it's CPU-bound). Yes you can speed it up, I've done it myself using python threads and something like 10 downloader threads.

Basically you use a producer-consumer model with one thread (or process) producing urls to download, and N threads (or processes) consuming from that queue and making requests to the server.

Here's some pseudo-code:

# Make sure that the queue is thread-safe!!

def producer(self):
    # Only need one producer, although you could have multiple
    with fh = open('urllist.txt', 'r'):
        for line in fh:

def consumer(self):
    # Fire up N of these babies for some speed
    while True:
        url = self.queue.dequeue()
        dh = urllib2.urlopen(url)
        with fh = open('/dev/null', 'w'): # gotta put it somewhere

Now if you're downloading very large chunks of data (hundreds of MB) and a single request completely saturates the bandwidth, then yes running multiple downloads is pointless. The reason you run multiple downloads (generally) is because requests are small and have a relatively high latency / overhead.

share|improve this answer
Thx, i added some code from the tool and my "implementation" with multiprocessing, which doesn´t really make things faster (the request are pretty small, btw) –  dorvak Aug 2 '11 at 0:13
Have you considered that perhaps you're getting throttled? This API may only allow a certain number of simultaneous connections per IP (or possibly a max number of requests per minute). –  Chris Aug 2 '11 at 0:15
Yep, that´s a good point..this was just an example (maybe code.google.com/p/urllib3) would be a good solution with the reusing connection) –  dorvak Aug 2 '11 at 0:19

What you do when you split web requests over several processes is to parallelize the network latencies (i.e. the waiting for responses). So you should normally get a good speedup, since most of the processes should sleep most of the time, waiting for an event.

Or use Twisted. ;)

share|improve this answer

It depends! Are you contacting different servers, are the transferred files small or big, do you loose much of the time waiting for the server to reply or by transferring data,...

Generally, multiprocessing involves some overhead and as such you want to be sure that the speedup gained by parallelizing the work is larger than the overhead itself.

Another point: network and thus I/O bound applications work – and scale – better with asynchronous I/O and an event driven architecture instead of threading or multiprocessing, as in such applications much of the time is spent waiting on I/O and not doing any computation.

For your specific problem, I would try to implement a solution by using Twisted, gevent, Tornado or any other networking framework which does not use threads to parallelize connections.

share|improve this answer

Your Answer


