title: How to use ES query with offset+limit >1000
body: <p>I have an API exposed for my client where I am using ES to fetch data for specific time range. The number is records is well over 1 million. Now, I had to provide another feature where I give them offset and limit where the client can fetch number of records(limit) from the offset.</p>
  <p>My ES query is formed like</p>
  <p><code>{&quot;from&quot;:10000,&quot;size&quot;:2001,&quot;timeout&quot;:&quot;60s&quot;,&quot;query&quot;:{&quot;bool&quot;:{&quot;must&quot;:[{&quot;terms&quot;:{&quot;tollId&quot;:[&quot;59850&quot;],&quot;boost&quot;:1.0}},{&quot;range&quot;:{&quot;updatedAt&quot;:{&quot;from&quot;:&quot;2020-08-15T00:00:00.000Z&quot;,&quot;to&quot;:null,&quot;include_lower&quot;:true,&quot;include_upper&quot;:true,&quot;boost&quot;:1.0}}},{&quot;range&quot;:{&quot;updatedAt&quot;:{&quot;from&quot;:null,&quot;to&quot;:&quot;2020-12-15T22:08:21.000Z&quot;,&quot;include_lower&quot;:true,&quot;include_upper&quot;:true,&quot;boost&quot;:1.0}}}],&quot;adjust_pure_negative&quot;:true,&quot;boost&quot;:1.0}},&quot;sort&quot;:[{&quot;updatedAt&quot;:{&quot;order&quot;:&quot;desc&quot;}}]}</code></p>
  <p>When I execute this on Elastic Search, I get</p>
  <pre><code>&quot;failed_shards&quot;: [
  {
    &quot;shard&quot;: 0,
    &quot;index&quot;: &quot;companydatabase&quot;,
    &quot;node&quot;: &quot;vQU6NjSVRK6dKNLsWkfqEw&quot;,
    &quot;reason&quot;: {
    &quot;type&quot;: &quot;query_phase_execution_exception&quot;,
    &quot;reason&quot;: &quot;Result window is too large, from + size must be less than or equal to: [10000] but was [12001]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting.&quot;
  }
  </code></pre>
  <p>The solution is to use Scroll API to fetch the records but I cant use scroll Api when I have to fetch records from some offset to some limit.</p>
  <p>Am I missing something? Is there any way to tackle this or I will have to get all the records(documents) everytime and filter the result?</p>