title: Breaking MongoDB $in query input array to smaller chunks improves performance?
body: <p>I'm running in to a performance issue on MongoDB where I'm doing an <code>$in</code> query. When the input array length of <code>$in</code> query is large (around 70) my API gateway endpoint times out since AWS API gateway endpoints times out after <strong>30s</strong>.</p>
  <p>Even for smaller arrays the <code>$in</code> query takes around <strong>21 seconds</strong> even though it is successful which is bad</p>
  <pre class="lang-js prettyprint-override"><code>    let departmentIds = [..........................  can be large]
  
      let query = {
        userId: someid,
        'department.department_id': {
          $in: departmentIds
        }
      };
  
  //db query
     let result =  db
      .collection(TABLE_NAME)
      .find(query)
      .project({
        anotherfield: 1,
        department: 1
      })
      .toArray();
  </code></pre>
  <p>My plan was to chunk the <code>departmentIds</code> array into smaller arrays and then in a for loop do the query and concat the <code>result</code> array. My issue is will this do any improvement in performance since I'm reducing the processing from the DB side a little.</p>
  <p>Any other strategies to improve this are welcome.</p>