RESTful API to safeguard server and client from large datasets <p>I am working on designing a RESTful API and need second opinion on the design. I will be abstracting away the problem statement for better understanding.</p> <p>Consider a URI <code>/search?key1=value1&amp;key2=value2</code>, which can potentially return a huge result set for a given search criteria for key1 and key2. </p> <p>My mandate is to make sure that the server and client are bounded by limits to prevent performance degradation. If that limit is reached and the intended data is not found in result set, user will be asked to refine the search query to narrow down. (I am not thinking of pagination, that is for a different problem set)</p> <p>Approach is to allow client specify a limit to server that it(client) can comfortably handle, and to help server set a limit for itself to prevent from generating huge result sets affecting performance.</p> <p>Client can do <code>/search?key1=value1&amp;key2=value2&amp;maxresults=xxxx</code> to specify it's limit.</p> <p>Server can set it's own limit as a configuration param for search URI. While serving a request, server will take a min of (client's limit, server's limit) and generate result set satisfying the effective limit.</p> <p>The JSON generated will have a meta data part which will mention if the result was truncated or not, and the effective limit set. The client can inspect this part and ask the user to refine search if "truncated" is "true". The problem domain actually allows the user to refine to a single item.</p> <pre><code>{ "result": { "truncated": "true", "limit": "2000", "data": [ { "id": "1" }, { "id": "2" } ... { "id": "2000" } ] } } </code></pre> <p>The questions I am trying to answer are:</p> <ul> <li>Is this violating any REST principles? </li> <li>Is there a standard convention to do the same that I might follow? </li> <li>Are there good examples on public APIs that you can quote? (Jira RESTful API has a couple of examples) </li> <li>Is there any gotcha in this design which may affect us in the future?</li> </ul> <p>Any view on this will be appreciated ...</p> <p>Thanks!</p>