title: Ridiculously slow mongoDB query on small collection in simple but big database
body: <p>So I have a super simple database in mongoDB with a few collections:</p>
  
  <pre><code>&gt; show collections
  Aggregates &lt;-- count: 92
  Users &lt;-- count: 68222
  Pages &lt;-- count: 1728288847, about 1.1TB
  system.indexes
  </code></pre>
  
  <p>The <code>Aggregates</code> collection is an aggregate of the <code>Pages</code> collection, and each document looks like this:</p>
  
  <pre><code>&gt; db.Aggregates.findOne()
  {
          "_id" : ObjectId("50f237126ba71610eab3aaa5"),
          "daily_total_pages" : 16929799,
          "day" : 21,
          "month" : 9,
          "year" : 2011
  }
  </code></pre>
  
  <p>Very simple. However, let's try and get the total page loads by adding all 92 days <code>daily page loads</code> together:</p>
  
  <pre><code>&gt;&gt;&gt; def get_total():
  ...     start = datetime.now()
  ...     print sum([x['daily_total_pages'] for x in c.Aggregates.find()])
  ...     end = datetime.now()
  ...     print (end-start).seconds
  ...
  &gt;&gt;&gt; get_total()
  1728288847
  43
  </code></pre>
  
  <p><strong>43 seconds?!??!??!?!</strong></p>
  
  <p>Those 92 aggregate results are tiny! I might as well just store them in a text file, this is crazy.</p>
  
  <p>Or are they tiny? How big are they on disk according to mongo?</p>
  
  <pre><code>&gt; db.Aggregates.stats()
  {
          "ns" : "c.AggregateResults",
          "count" : 92,
          "size" : 460250104,
          "avgObjSize" : 5002718.521739131,
          "storageSize" : 729464832,
          "numExtents" : 7,
          "nindexes" : 2,
          "lastExtentSize" : 355647488,
          "paddingFactor" : 1.0690000000000066,
          "systemFlags" : 1,
          "userFlags" : 0,
          "totalIndexSize" : 16352,
          "indexSizes" : {
                  "_id_" : 8176,
                  "date_1" : 8176
          },
          "ok" : 1
  }
  </code></pre>
  
  <p><strong>438 megabytes total for those tiny daily numbers?</strong> Each one is approximately 280 bytes, so they should be a maximum of 25~30kb total. So the storage is huge and the query is super slow. Is it possible it could be fragmented on disk? I created the aggregates after inserting the documents into the the full <code>Pages</code> collection.</p>
  
  <p>Anyone have any insights into this madness? :O</p>
  
  <hr>
  
  <p>Edit: Solved with more specific find() query by Jared. The video below that Sammaye provided also gives some very interesting storage insights. </p>
  
  <hr>
  
  <p>Edit 2: So I found out that using sys.getsizeof() is a <em>really</em> unreliable method of finding out your document's sizes since it doesn't recurse down any trees. So actually my docs were quite large, and the best thing to do was to use find({}, {'daily_page_loads'}) as a more specific query!</p>