Take the 2-minute tour ×

Trying to find a simple example that clearly shows a single task being divided for multi-threading.

Quite frankly, many of the examples are overly sophisticated thus making the flow tougher to play with. Anyone care to share their breakthrough sample or point to an example?

As well, what is the best docs? many Google look-ups are too specific (for me at this stage).

Thanks in advance.

share|improve this question
A good general discussion around this topic can be found in Python's Hardest Problem by Jeff Knupp. In summary, it seems threading is not for beginners. –  Matthew Walker Sep 4 '13 at 7:12
haha, I tend to think that threading is for everyone, but beginners are not for threading :))))) –  Bohdan Sep 17 '13 at 3:37
add comment

7 Answers

Here's a simple example: you need to try a few alternative URLs and return the contents of the first one to respond.

import Queue
import threading
import urllib2

# called by each thread
def get_url(q, url):

theurls = '''http://google.com http://yahoo.com'''.split()

q = Queue.Queue()

for u in theurls:
    t = threading.Thread(target=get_url, args = (q,u))
    t.daemon = True

s = q.get()
print s

This is a case where threading is used as a simple optimization: each subthread is waiting for a URL to resolve and respond, in order to put its contents on the queue; each thread is a daemon (won't keep the process up if main thread ends -- that's more common than not); the main thread starts all subthreads, does a get on the queue to wait until one of them has done a put, then emits the results and terminates (which takes down any subthreads that might still be running, since they're daemon threads).

Proper use of threads in Python is invariably connected to I/O operations (since CPython doesn't use multiple cores to run CPU-bound tasks anyway, the only reason for threading is not blocking the process while there's a wait for some I/O). Queues are almost invariably the best way to farm out work to threads and/or collect the work's results, by the way, and they're intrinsically threadsafe so they save you from worrying about locks, conditions, events, semaphores, and other inter-thread coordination/communication concepts.

share|improve this answer
Still relevant. Great answer. –  stefgosselin Oct 9 '12 at 3:26
Thanks again, MartelliBot. –  JRM Jan 7 '13 at 1:51
Thanks again, MartelliBot. I've updated the example to wait for all to urls to respond: import Queue, threading, urllib2 q = Queue.Queue() urls = '''a.com b.com c.com'''.split() urls_received = 0 def get_url(q, url): req = urllib2.Request(url) resp = urllib2.urlopen(req) q.put(resp.read()) global urls_received urls_received +=1 print urls_received for u in urls: t = threading.Thread(target=get_url, args = (q,u)) t.daemon = True t.start() while q.empty() and urls_received < len(urls): s = q.get() print s –  JRM Jan 7 '13 at 2:23
@JRM: if you look at the next answer below, I think that a better way to wait until the threads are finished would be to use the join() method, since that would make the main thread wait until they're done without consuming processor by constantly checking the value. @Alex: thanks, this is exactly what I needed to understand how to use threads. –  krs013 May 31 '13 at 5:33
I note that solution will only print out one of the pages. To print both pages from the queue simply run the command again: s = q.get() print s @krs013 You don't need the join because Queue.get() is blocking. –  Tom Anderson Jan 7 at 5:50
show 7 more comments

NOTE: For actual parallelization in Python, you should use the multiprocessing module to fork multiple processes that execute in parallel (due to the global interpreter lock, Python threads provide interleaving but are in fact executed serially, not in parallel, and are only useful when interleaving I/O operations).

However, if you are merely looking for interleaving (or are doing I/O operations that can be parallelized despite the global interpreter lock), then the threading module is the place to start. As a really simple example, let's consider the problem of summing a large range by summing subranges in parallel:

import threading

class SummingThread(threading.Thread):
     def __init__(self,low,high):
         super(SummingThread, self).__init__()

     def run(self):
         for i in range(self.low,self.high):

thread1 = SummingThread(0,500000)
thread2 = SummingThread(500000,1000000)
thread1.start() # This actually causes the thread to run
thread1.join()  # This waits until the thread has completed
# At this point, both threads have completed
result = thread1.total + thread2.total
print result

Note that the above is a very stupid example, as it does absolutely no I/O and will be executed serially albeit interleaved (with the added overhead of context switching) in CPython due to the global interpreter lock.

share|improve this answer
@Alex, I didn't say it was practical, but it does demonstrate how to define and spawn threads, which I think is what the OP wants. –  Michael Aaron Safyan May 17 '10 at 4:39
While this does show how to define and spawn threads, it actually does not sum the subranges in parallel. thread1 runs until it's completed while the main thread blocks, then the same thing happens with thread2, then the main thread resumes and prints out the values they accumulated. –  martineau Feb 17 at 19:32
Shouldn't that be super(SummingThread, self).__init__()? As in stackoverflow.com/a/2197625/806988 –  James Andres Mar 6 at 9:37
@JamesAndres, assuming that no one inherits from "SummingThread", then either one works fine; in such a case super(SummingThread, self) is just a fancy way to look up the next class in the method resolution order (MRO), which is threading.Thread (and then subsequently calling init on that in both cases). You are right, though, in that using super() is better style for current Python. Super was relatively recent at the time that I provided this answer, hence calling directly to the super class rather than using super(). I'll update this to use super, though. –  Michael Aaron Safyan Mar 6 at 11:16
WARNING: Don't use multithreading in tasks like this! As was shown by Dave Beazley: dabeaz.com/python/NewGIL.pdf, 2 python threads on 2 CPUs carry out a CPU-heavy task 2 times SLOWER than 1 thread on 1 CPU and 1.5 times SLOWER than 2 threads on 1 CPU. This bizarre behavior is due to mis-coordination of efforts between OS and Python. A real-life use case for threads is an I/O heavy task. E.g. when you perform read/writes over network, it makes sense to put a thread, waiting for data to be read/written, to background and switch CPU to another thread, which needs to process data. –  Bob May 15 at 23:13
show 1 more comment

Like others mentioned, CPython can use threads only for I\O waits due to GIL. If you want to benefit from multiple cores for CPU-bound tasks, use multiprocessing:

from multiprocessing import Process

def f(name):
    print 'hello', name

if __name__ == '__main__':
share|improve this answer
could you explain a little what this does? –  pandita Sep 14 '13 at 16:01
@pandita: the code creates a process, then starts it. So now there's two things happening at once: the main line of the program, and the process that's starting with the target, f function. In parallel, the main program now just waits for the process to exit, joining up with it. If the main part just exited, the subprocess might or might not run to completion, so doing a join is always recommended. –  shavenwarthog Jul 2 at 5:56
add comment

Just a note, Queue is not required for threading.

This is the simplest example I could imagine that shows 10 processes running concurrently.

import threading
from random import randint
from time import sleep

def printNumber(number):
   # Sleeps a random 1 to 10 seconds
   print str(number)

thread_list = []

for i in range(1,10):
   # Instatiates the thread
   # (i) does not make a sequence, so (i,)
   t = threading.Thread(target=printNumber, args=(i,))
   # Sticks the thread in a list so that it remains accessible 

# Starts threads
for thread in thread_list:

# From http://docs.python.org/2/library/threading.html#thread-objects
for thread in thread_list:

# Demonstrates that the main process waited for threads to complete
print "Done"
share|improve this answer
Add the last quote to "Done to make it print "Done" –  nwaomachux Feb 11 at 9:53
add comment

The answer from Alex Martelli helped me, however here is modified version that I thought was more useful (at least to me).

import Queue
import threading
import urllib2

worker_data = ['http://google.com', 'http://yahoo.com', 'http://bing.com']

#load up a queue with your data, this will handle locking
q = Queue.Queue()
for url in worker_data:

#define a worker function
def worker(queue):
    queue_full = True
    while queue_full:
            #get your data off the queue, and do some work
            url= queue.get(False)
            data = urllib2.urlopen(url).read()
            print len(data)

        except Queue.Empty:
            queue_full = False

#create as many threads as you want
thread_count = 5
for i in range(thread_count):
    t = threading.Thread(target=worker, args = (q,))
share|improve this answer
Why not just break on the exception? –  Stavros Korokithakis Feb 9 at 19:16
you could, just personal preference –  JimJty Feb 10 at 21:10
add comment

For me, the perfect example for Threading is monitoring Asynchronous events. Look at this code.

# thread_test.py
import threading
import time 

class Monitor(threading.Thread):
    def __init__(self, mon):
        self.mon = mon

    def run(self):
        while True:
            if self.mon[0] == 2:
                print "Mon = 2"
                self.mon[0] = 3;

You can play with this code by opening an IPython session and doing something like:

>>>from thread_test import Monitor
>>>a = [0]
>>>mon = Monitor(a)
>>>a[0] = 2
Mon = 2
>>>a[0] = 2
Mon = 2

Wait a few minutes

>>>a[0] = 2
Mon = 2
share|improve this answer
simple and self-explanatory :) –  deeshank Apr 22 '13 at 9:28
How do we stop the thread manually? @dvreed77 –  deeshank Apr 22 '13 at 9:32
AttributeError: 'Monitor' object has no attribute 'stop' ? –  pandita Sep 14 '13 at 15:29
Aren't you blasting away CPU cycles while waiting for your event to happen? Not always a very practical thing to do. –  mogul Sep 16 '13 at 16:58
add comment

I found this very useful: create as many threads as cores and let them execute a (large) number of tasks (in this case, calling a shell program):

import Queue
import threading
import multiprocessing
import subprocess

q = Queue.Queue()
for i in range(30): #put 30 tasks in the queue

def worker():
    while True:
        item = q.get()
        #execute a task: call a shell program and wait until it completes
        subprocess.call("echo "+str(item), shell=True) 

cpus=multiprocessing.cpu_count() #detect number of cores
print("Creating %d threads" % cpus)
for i in range(cpus):
     t = threading.Thread(target=worker)
     t.daemon = True

q.join() #block until all tasks are done
share|improve this answer
Note that threads are started on the same CPU as the parent process. Thus, starting more threads on a box with more cores might not do what you want. +1 for executable answer! –  shavenwarthog Jun 27 at 16:32
@shavenwarthog sure one can adjust the "cpus" variable depending on one's needs. Anyway, the subprocess call will spawn subprocesses and these will be allocated cpus by the OS (python's "parent process" does not mean "same CPU" for the subprocesses). –  dolphin Jul 2 at 2:28
you're correct, my comment about "threads are started on the same CPU as the parent process" is wrong. Thanks for the reply! –  shavenwarthog Jul 2 at 5:51
add comment

Your Answer


