title: Queue based processing
body: <p>So I've been looking into building an application that relies on something similar to a messaging bus. The idea is to be extremely fault tolerant. I have a queue of tasks that need to be performed, and here are the steps that I believe would be the end goal in a queue based system.</p>
  
  <ul>
  <li>A/B server initially adds an item to the queue to be worked on.</li>
  <li>C server listening on queue sees a new item on the queue, and begins to work on it. I believe it should lock the item with a timeout, (If the server crashes, etc..., I need other workers to be able to work on it.)</li>
  </ul>
  
  <p>One of two things happens now:</p>
  
  <ul>
  <li>C server fails to respond to said task, or it has taken long past the timeout, and the queue unlocks it and passes it to server D for processing</li>
  </ul>
  
  <p><strong>- OR -</strong></p>
  
  <ul>
  <li>C server completes the task and ultimately removes the entry from the queue.</li>
  </ul>
  
  <p>I was looking into different solutions, and I see a lot of people using REDIS as a backend for performing this operation, but it's queue is fairly simplistic. For example <a href="http://redis.io/commands/rpoplpush" rel="nofollow"><code>RPOPLPUSH</code></a> will remove the key from the queue. What happens if the server crashes? The queue now thinks it processed that item, and we have a lost task.</p>
  
  <p>What steps are recommended for ensuring task completions, and noting task failures to be able to be reproccessed by another server? I intend to write the tasks in go and I'm open to using cloud services such as AWS.</p>