title: MogoDB - Query has become extremely slow on a PSA replicaset
body: <p>I have a MongoDB replica set with one primary, one arbiter, one vote bearing secondary, and two no vote bearing secondary servers.</p>
  <p>The two non-vote bearing secondaries are bare metal mongo installs to help with heavy aggregation queries. (each 16cores/128GBRam/SSD)</p>
  <p>The replica set member configuration is at the bottom of the question.</p>
  <p>The following query is run multiple times a minute (around 100) against different collections (~10).</p>
  <p>The query used to take no longer than 2 seconds at worst.</p>
  <p>Since about a week ago I have noticed that applications that depend on that query were erroring on timeout. I have run the query against the replica set connection (mongo+srv), or individual members - with the same result - &quot;<em><strong>timeout</strong></em>&quot;</p>
  <p>I cannot recall any events to have caused the issue and I am wondering if the query might be constructed somehow fundamentally wrong to be causing the issue.</p>
  <p>The query is <em>selecting the least extracted item that has not been deleted and has not been archived.</em></p>
  <p>I have indexed the fields &quot;<em>attributes</em>&quot;, &quot;<em>attributes.deleted</em>&quot;, &quot;<em>attributes.archieved</em>&quot;, &quot;<em>last_extracted</em>&quot; but this has not helped.</p>
  <p>The size of the collections is between 2gb/50K records to 10gb/150k records.</p>
  <p><strong>Is there anything wrong with this query?</strong></p>
  <pre><code>db.getCollection(&quot;threads&quot;).aggregate(
      [{'$match': 
              {'$and': 
                  [{'$or': 
                      [{'attributes': 
                          {'$exists': false}}, 
                       {'attributes.deleted': 
                              {'$exists': false}}, 
                       {'attributes.deleted': {'$eq': false}}]}, 
                   {'$or': 
                       [{'attributes.archived': 
                           {'$exists': false}}, 
                           {'attributes.archived': 
                               {'$eq': false}}]}
                   ]}}, 
        {'$project': {'_id': 1, 'url': 1, 'last_extraction': 1}}, 
        {'$sort': {'last_extraction': 1}}, 
        {'$limit': 1}
        ])
  </code></pre>
  <p><strong>Replica Set Members Configuration</strong></p>
  <pre><code>rs0:SECONDARY&gt; cfg.members
  [
      {
          &quot;_id&quot; : 1,
          &quot;host&quot; : &quot;XXXXXX1:27777&quot;,
          &quot;arbiterOnly&quot; : false,
          &quot;buildIndexes&quot; : true,
          &quot;hidden&quot; : false,
          &quot;priority&quot; : 0.8,
          &quot;tags&quot; : {
  
          },
          &quot;slaveDelay&quot; : NumberLong(0),
          &quot;votes&quot; : 1
      },
      {
          &quot;_id&quot; : 2,
          &quot;host&quot; : &quot;XXXXX2:27777&quot;,
          &quot;arbiterOnly&quot; : false,
          &quot;buildIndexes&quot; : true,
          &quot;hidden&quot; : false,
          &quot;priority&quot; : 0.2,
          &quot;tags&quot; : {
  
          },
          &quot;slaveDelay&quot; : NumberLong(0),
          &quot;votes&quot; : 1
      },
      {
          &quot;_id&quot; : 3,
          &quot;host&quot; : &quot;XXXXX3:27777&quot;,
          &quot;arbiterOnly&quot; : true,
          &quot;buildIndexes&quot; : true,
          &quot;hidden&quot; : false,
          &quot;priority&quot; : 0,
          &quot;tags&quot; : {
  
          },
          &quot;slaveDelay&quot; : NumberLong(0),
          &quot;votes&quot; : 1
      },
      {
          &quot;_id&quot; : 4,
          &quot;host&quot; : &quot;XXXXXX4:27777&quot;,
          &quot;arbiterOnly&quot; : false,
          &quot;buildIndexes&quot; : true,
          &quot;hidden&quot; : false,
          &quot;priority&quot; : 0,
          &quot;tags&quot; : {
  
          },
          &quot;slaveDelay&quot; : NumberLong(0),
          &quot;votes&quot; : 0
      },
      {
          &quot;_id&quot; : 5,
          &quot;host&quot; : &quot;XXXXXXX5:27776&quot;,
          &quot;arbiterOnly&quot; : false,
          &quot;buildIndexes&quot; : true,
          &quot;hidden&quot; : false,
          &quot;priority&quot; : 0,
          &quot;tags&quot; : {
  
          },
          &quot;slaveDelay&quot; : NumberLong(0),
          &quot;votes&quot; : 0
      }
  ]
  
  </code></pre>