title: Why is mongoDB taking up so much RAM space?
body: <p>I've managed to write a simple indexer script for mongoDB using pymongo. But I've no idea why would indexing, adding documents and querying would take up 96GB of the RAM on my server.</p>
  
  <p><strong>Is it because my query isn't optimized? How could i optimize my query instead of</strong> <code>database.find_one({"eng":src})</code></p>
  
  <p><strong>How else could i optimize my indexer script?</strong></p>
  
  <p>So my inputs are as such (the actual data inputs have <strong>2 million+ lines</strong> of varying length of sentence):</p>
  
  <p><strong>#srcfile</strong></p>
  
  <pre><code>You will be aware from the press and television that there have been a number of bomb explosions and killings in Sri Lanka.
  One of the people assassinated very recently in Sri Lanka was Mr Kumar Ponnambalam, who had visited the European Parliament just a few months ago.
  Would it be appropriate for you, Madam President, to write a letter to the Sri Lankan President expressing Parliament's regret at his and the other violent deaths in Sri Lanka and urging her to do everything she possibly can to seek a peaceful reconciliation to a very difficult situation?
  Yes, Mr Evans, I feel an initiative of the type you have just suggested would be entirely appropriate.
  If the House agrees, I shall do as Mr Evans has suggested.
  </code></pre>
  
  <p><strong>#trgfile</strong></p>
  
  <pre><code>Wie Sie sicher aus der Presse und dem Fernsehen wissen, gab es in Sri Lanka mehrere Bombenexplosionen mit zahlreichen Toten.
  Zu den Attentatsopfern, die es in jüngster Zeit in Sri Lanka zu beklagen gab, zählt auch Herr Kumar Ponnambalam, der dem Europäischen Parlament erst vor wenigen Monaten einen Besuch abgestattet hatte.
  Wäre es angemessen, wenn Sie, Frau Präsidentin, der Präsidentin von Sri Lanka in einem Schreiben das Bedauern des Parlaments zum gewaltsamen Tod von Herrn Ponnambalam und anderen Bürgern von Sri Lanka übermitteln und sie auffordern würden, alles in ihrem Kräften stehende zu tun, um nach einer friedlichen Lösung dieser sehr schwierigen Situation zu suchen?
  Ja, Herr Evans, ich denke, daß eine derartige Initiative durchaus angebracht ist.
  Wenn das Haus damit einverstanden ist, werde ich dem Vorschlag von Herrn Evans folgen.
  </code></pre>
  
  <p>An example doc looks like this</p>
  
  <pre><code>{ 
      "_id" : ObjectId("50f5fe8916174763f6217994"), 
      "deu" : "Wie Sie sicher aus der Presse und dem Fernsehen wissen, gab es in Sri 
               Lanka mehrere Bombenexplosionen mit zahlreichen Toten.\n", 
      "uid" : 13, 
      "eng" : "You will be aware from the press and television that there have been a 
               number of bomb explosions and killings in Sri Lanka." 
  }
  </code></pre>
  
  <p><strong>My code</strong>:</p>
  
  <pre class="lang-py prettyprint-override"><code># -*- coding: utf8 -*-
  import codecs, glob, os
  from pymongo import MongoClient
  from itertools import izip
  from bson.code import Code
  
  import sys
  reload(sys)
  sys.setdefaultencoding("utf-8")
  
  # Gets first instance of matching key given a value and a dictionary.    
  def getKey(dic, value):
    return [k for k,v in dic.items() if v == value]
  
  def langiso (lang, isochar=3):
    languages = {"en":"eng",
                 "da":"dan","de":"deu",
                 "es":"spa",
                 "fi":"fin","fr":"fre",
                 "it":"ita",
                 "nl":"nld",
                 "zh":"mcn"}
    if len(lang) == 2 or isochar==3:
      return languages[lang]
    if len(lang) == 3 or isochar==2:
      return getKey(lang)
  
  def txtPairs (bitextDir):
    txtpairs = {}
    for infile in glob.glob(os.path.join(bitextDir, '*')):
      #print infile
      k = infile[-8:-3]; lang = infile[-2:]
      try:
        txtpairs[k] = (txtpairs[k],infile) if lang == "en" else (infile,txtpairs[k]) 
      except:
        txtpairs[k] = infile
    for i in txtpairs:
      if len(txtpairs[i]) != 2:
        del txtpairs[i]
    return txtpairs
  
  def indexEuroparl(sfile, tfile, database):   
    trglang = langiso(tfile[-2:]) #; srclang = langiso(sfile[-2:]) 
  
    maxdoc = database.find().sort("uid",-1).limit(1)
    uid = 1 if maxdoc.count() == 0 else maxdoc[0]
  
    counter = 0
    for src, trg in izip(codecs.open(sfile,"r","utf8"), \
                         codecs.open(tfile,"r","utf8")):
      quid = database.find_one({"eng":src})
      # If sentence already exist in db
      if quid != None:
        if database.find({trglang: {"$exists": True}}):
          print "Sentence uniqID",quid["uid"],"already exist."
          continue
        else:
          print "Reindexing uniqID",quid["uid"],"..."
          database.update({"uid":quid["uid"]}, {"$push":{trglang:trg}})
      else:
        print "Indexing uniqID",uid,"..."
        doc = {"uid":uid,"eng":src,trglang:trg}
        database.insert(doc)
        uid+=1
      if counter == 1000:
        for i in database.find():
          print i
        counter = 0
      counter+=1
  
  connection = MongoClient()
  db = connection["europarl"]
  v7 = db["v7"]
  
  srcfile = "eng-deu.en"; trgfile = "eng-deu.de"
  indexEuroparl(srcfile,trgfile,v7)
  
  # After indexing the english-german pair, i'll perform the same indexing on other language pairs
  srcfile = "eng-spa.en"; trgfile = "eng-spa.es"
  indexEuroparl(srcfile,trgfile,v7)
  </code></pre>