title: MongoDB concurrent queries on different collections are slow
body: <p>I have multiple mongodb collections with 100000 documents in each collection and 10000 columns in each document.
  There is a python script which executes aggregate queries in a multi-threaded fashion. Each thread invokes an aggregate on a separate collection.
  Here's the corresponding code:</p>
  
  <pre><code>import pymongo
  import time
  import threading
  import sys
  
  mongocli = pymongo.MongoClient(host="192.168.99.100", username="admin", password="admin123", \
      authSource="analyse_db")
  db = mongocli['analyse_db']
  collection = sys.argv[1]
  column = sys.argv[2]
  threaded = int(sys.argv[3])
  # Start times of time column of every collection.
  # There are 7 collections for each day in a week.
  start_times = [1587288313, 1587374713, 1587461113, 1587547513, 1587633514, 1587719914, 1587806314]
  interval = 86000
  time_column = "timestamp_EP"
  
  col = db[collection]
  group_query = {"$group": {"_id": "", "result": {"$avg": "$" + column}}}
  
  def aggregate_thread(local_col, start_time, end_time, thread_id):
      local_mongocli = pymongo.MongoClient(host="192.168.99.100", username="admin", password="admin123", \
      authSource="analyse_db")
      local_db = local_mongocli['analyse_db']
      local_col_obj = local_db[local_col]
      #cursor = local_col.aggregate([{"$match": {time_column: \
      #    {"$gte": start_time, "$lt": end_time}}}, group_query])
      thr_start_time = time.time()
      cursor = local_col_obj.aggregate([group_query])
      try:
          print(cursor.next())
      except Exception:
          print("no data")
      print("thread " + str(thread_id) + " completion time: ", time.time() - thr_start_time)
  
  def aggregate_threaded():
      thr_list = []
      nthreads = len(start_times)
      for i in range(0, nthreads):
          ftime = start_times[i]
          ttime = ftime + interval
          local_col = collection + "_day" + str(i)
          thr = threading.Thread(target=aggregate_thread, args=[local_col, ftime, ttime, i], daemon=True)
          thr_list.append(thr)
          thr.start()
      for thr in thr_list:
          _res = thr.join()
  
  proc_start = time.time()
  aggregate_threaded()
  print(time.time() - proc_start)
  </code></pre>
  
  <p>Now, when the script is executed, the amount of time it takes to complete the aggregation is proportional to the number of threads. i.e. the latency linearly increases with number of queries concurrently executed.
  Here's the result of script (7 threads executing aggregate queries on different collections):</p>
  
  <pre><code>C:\Users\AJINKYA\itanta\live_data&gt;python perf_test3.py Archive-LiveDataLog5 tag1 1
  {'_id': '', 'result': 50.054371460796965}
  thread 3 completion time:  140.73034977912903
  {'_id': '', 'result': 50.20921849745933}
  thread 5 completion time:  146.46782159805298
  {'_id': '', 'result': 50.064871338705366}
  thread 4 completion time:  147.66157269477844
  {'_id': '', 'result': 50.17267241078592}
  thread 1 completion time:  151.08023619651794
  {'_id': '', 'result': 49.85328077580493}
  thread 2 completion time:  151.3344430923462
  {'_id': '', 'result': 49.993023336937945}
  thread 0 completion time:  151.4148395061493
  {'_id': '', 'result': 49.89189660585342}
  thread 6 completion time:  151.54917550086975
  151.57819509506226
  </code></pre>
  
  <p>So it took 151 seconds for 7 threads to finish aggregations.
  On the other hand, if there is only single thread executing aggregation on a single collection, it takes much less time. Here's the result if <code>start_times</code> in above script has only one element:</p>
  
  <pre><code>C:\Users\AJINKYA\itanta\live_data&gt;python perf_test3.py Archive-LiveDataLog5 tag1 1
  {'_id': '', 'result': 49.993023336937945}
  thread 0 completion time:  43.36866021156311
  43.37059950828552
  </code></pre>
  
  <p>It took just 43 seconds if there is only one aggregate query.
  My expectation was that the multi-threaded queries would take roughly the same amount of time as the single-threaded one.
  Now it's apparent that multiple queries are not executed concurrently in mongodb.
  Is this the known limitation of mongodb?
  Is there any configuration parameter in mongodb to control concurrency?</p>