title: Store streaming data - fast, cheap, reliable and good for batch consumption
body: <p>I have a (spring-boot) web service that generates a json response for each request. This response, while returned to the querying user, also needs to be archived somewhere (so that we know what we responded with to the user).</p>
  
  <p>The service needs to support 4,000 requests/second. As such, we need the archival method to be fast. The archived data would later be consumed by a map-reduce (batch) job.</p>
  
  <p>I want to know which solution to use - Kafka, S3, or any other solution. The service has been deployed to AWS. So solutions within AWS are ideal.</p>
  
  <p>The requirements are as follows:</p>
  
  <ul>
  <li>Writes should be fast 94K req/s at least).</li>
  <li>Writes should be non-blocking (so that the service response time is not affected).</li>
  <li>Reads need not be fast but should be suitable for consumption by map-reduce jobs.</li>
  <li>Data should be resilient to server crashes etc.</li>
  <li>Should not be too expensive to write/store and read.</li>
  </ul>
  
  <p>There is no <code>data retirement plan</code>, i.e. the data needs to persist until the end of time.</p>
  
  <p>Which solutions do you recommend?</p>