 <p>I finally settled on using a file-based lock to ensure that the task doesn't run twice:</p> <pre><code>def get_lock(name): fd = open('/tmp/' + name, 'w') try: flock(fd, LOCK_EX | LOCK_NB) # open for exclusive locking return fd except IOError as e: logger.warn('Could not get the lock for ' + str(name)) fd.close() return None def release_lock(fd): sleep(2) # extend the time a bit longer in the hopes that it blocks the other proc flock(fd, LOCK_UN) fd.close() </code></pre> <p>It's a bit of a hack, but seems to be working...</p>