title: Request-response with kafka and key-value store
body: <p><strong>TLDR: Have a request/response pattern. Currently requests done via activemq queue and response is done via memcached key-value store (which is polled by front end). Want to move to kafka for a variety of reasons, wondering if we can re-architect the response path to not use memcached.</strong></p>
  
  <p>I am trying to understand what would be the best practice system design for the following problem.</p>
  
  <p>We have a frontend that generates requests that require heavy processing. The app needs a response in order to advance. Occasionally we need to undo/step back (which gets you to the previous state(s)). There is a cluster of backends that can perform the heavy processing step.</p>
  
  <p>In our current setup the frontend pushes requests into a queue (currently activemq) and the backends processes items from the queue as they can and stores results in a key-value store (memcached), with the key being the UUID of the message from the queue (which is itself a unique session id + non-unique step id). The frontend is polling the store for the UUID of the message. This has the advantage that a front-end might lose connection/etc but as long as session id is preserved we can ping the key-value store for the result we need. We also occasionally need to move back/undo actions and we can walk back the results in the key-value store (since each step has its own UUID and all UUIDs are known).</p>
  
  <p>However, in the future we would like to be able to do the response at least partially through the queue, this way we can have some analytics tools as consumers for both request and response. The "minimal-change" would be to have the response producers push into a queue and have memcached being one of the consumers. But maybe there is a better way. We are also looking at switching from activemq to Kafka as this would give us replayability (but we don't have experience with kafka). </p>
  
  <p>Looking at Kafka, it looks like to get a particular message you would need to scan the entire partition, is there an easier way to retrieve a particular message? Do we generate a topic for every sequence of interactions? If we want to replay but don't know the offsets what is our recourse (other than looking through very many messages)? Our load is pretty small (~ 1 mil messages/day) so I guess anything works but what would be the best practices (the infamous, what if we scale)?</p>