title: Is it possible to ensure unique messages are in a rabbitmq queue?
body: <p>Basically my consumers are producers as well.  We get an initial dataset and it gets sent to the queue.  A consumer takes an item and processes it, from that point there's 3 possibilities:</p>
  
  <ol>
  <li>Data is good and gets putting a 'good' queue for storage</li>
  <li>Data is bad and discarded </li>
  <li>Data is not good(yet) or bad(yet) so data is broken down into smaller parts and sent back to the queue for further processing.</li>
  </ol>
  
  <p>My problem is with step 3, because the queue grows very quickly at first its possible that a piece of data is broken down into a part thats duplicated in the queue and the consumers continue to process it and end up in a infinite loop.  </p>
  
  <p>I think the way to prevent against this is to prevent duplicates from going into the queue.  I can't do this on the client side because over the course of an hour I may have many cores dealing with billions of data points(to have each client scan it before submitting would slow me down too much).  I think this needs to be done on the server side but, like I mentioned, the data is quite large and I don't know how to efficiently ensure no duplicates.</p>
  
  <p>I might be asking the impossible but thought I'd give it a shot. Any ideas would be greatly appreciated.  </p>