title: GAE - What is the fastest way to add tasks to queue? Why does this appear to be so slow?
body: <p>I am using Google App Engine (Python) to process some event messages in real time. In short I have 100+ tasks that I need to run fast when a message comes in. I have tried a few approaches (deferred library, threads) and I think the best solution involves using the task queue and asynchronously adding these tasks to the queue I want. Here's an example of what I am doing.</p>
  
  <pre><code>tasks = []
  task = Task(url=url_for('main.endpoints_worker'),params={'id': id})
  tasks.append(task.add_async(queue_name='event-message'))
  
  for task in tasks:
      task.get_result()
  </code></pre>
  
  <p>When I do this most of my time is spent adding these tasks to the queue. Is there a way to speed this up? Is there a better approach?</p>
  
  <p><a href="https://i.stack.imgur.com/nYEfB.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/nYEfB.png" alt="Example of Task Queue Bulk Add Time"></a></p>
  
  <p>Now to be honest I get vastly different times each time I run this. Sometimes I am around 100ms (which would be fine) but other times I am up around 1s. </p>
  
  <p>I would have thought spreading out the work would have been faster but bulk adding to the task queue out performs. With suggested approach below here is what I am seeing:</p>
  
  <pre><code>tasks = [Task(url=url_for('main.endpoints_worker'),params={'id': id}) for id in id_list]
  rpc = Queue('event-message').add_async(tasks)
  rpc.get_result()
  </code></pre>
  
  <p><a href="https://i.stack.imgur.com/UPnZI.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/UPnZI.png" alt="enter image description here"></a></p>
  
  <p><strong>UPDATE: I need to examine this issue again due to the 100 task limit when adding to a queue.</strong>
  I have greatly improved the throughput of my code by batching the creation of my tasks (groups of 100) but I still don't understand why adding multiple groups of tasks to a queue slows down so quickly. One task queue.add_async runs &lt; 40ms no issue. When I do 2 or more queue.add_async that time slows down. I would love to know why? Also how do I get around this?</p>
  
  <p><a href="https://i.stack.imgur.com/jrAHT.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/jrAHT.png" alt="enter image description here"></a></p>
  
  <p>When I add batches of tasks without async they each take &lt; 40ms. Why do they take so much longer when async is used? </p>
  
  <p><a href="https://i.stack.imgur.com/0gqQS.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/0gqQS.png" alt="enter image description here"></a></p>
  
  <p><strong>Another Update</strong> I thought the issue may be contention related but even when I add each of these tasks to a different queue I get the same results.</p>