title: How to design a REST API to fetch a large (ephemeral) data stream?
body: <p>Imagine a request that starts a long running process whose output is a large set of records.</p>
  
  <p>We could start the process with a POST request:</p>
  
  <blockquote>
    <p>POST /api/v1/long-computation</p>
  </blockquote>
  
  <p>The output consists of a large sequence of numbered records, that must be sent to the client. Since the output is large, the server does not store everything, and so maintains a window of records with a upper limit on the size of the window. Let's say that it stores upto 1000 records (and pauses computation whenever this many records are available). When the client fetches records, the server may subsequently delete those records and so continue with generating more records (as more slots in the 1000-length window are free).</p>
  
  <p>Let's say we fetch records with:</p>
  
  <blockquote>
    <p>GET /api/v1/long-computation?ack=213</p>
  </blockquote>
  
  <p>We can take this to mean that the server should return records starting from index 214. When the server receives this request, it can assume that the (well-behaved) client is acknowledging that records up to number 213 are received by the client and so it deletes them, and then returns records starting from number 214 to whatever is available at that time.</p>
  
  <p>Next if the client requests:</p>
  
  <blockquote>
    <p>GET /api/v1/long-computation?ack=214</p>
  </blockquote>
  
  <p>the server would delete record 214 and return records starting from 215.</p>
  
  <p>This seems like a reasonable design until it is noticed that GET requests need to be safe and idempotent (see section 9.1 in the <a href="https://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html" rel="nofollow noreferrer">HTTP RFC</a>).</p>
  
  <p>Questions:</p>
  
  <ol>
  <li>Is there a better way to design this API?</li>
  <li>Is it OK to keep it as GET even though it appears to violate the standard?</li>
  <li>Would it be reasonable to make it a POST request such as: 
  
  <blockquote>
    <p>POST /api/v1/long-computation/truncate-and-fetch?ack=213</p>
  </blockquote></li>
  </ol>