title: How to populate mongoose with a large data set
body: <p>I'm attempting to load a store catalog into MongoDb (2.2.2) using Node.js (0.8.18) and Mongoose (3.5.4) -- all on Windows 7 64bit. The data set contains roughly 12,500 records. Each data record is a JSON string. </p>
  
  <p>My latest attempt looks like this:</p>
  
  <pre><code>var fs = require('fs');
  var odir = process.cwd() + '/file_data/output_data/';
  var mongoose = require('mongoose');
  var Catalog = require('./models').Catalog;
  
  var conn = mongoose.connect('mongodb://127.0.0.1:27017/sc_store');
  
  exports.main = function(callback){
      var catalogArray = fs.readFileSync(odir + 'pc-out.json','utf8').split('\n');
      var i = 0;
  
      Catalog.remove({}, function(err){
          while(i &lt; catalogArray.length){
              new Catalog(JSON.parse(catalogArray[i])).save(function(err, doc){
                  if(err){
                      console.log(err);
                  } else {
                      i++;                    
                  }
              });
              if(i === catalogArray.length -1) return callback('database populated');
          }
      });
  };
  </code></pre>
  
  <p>I have had a lot of problems trying to populate the database. Under previous scenarios (and this one), node pegs the processor and eventually runs out of memory. Note that in this scenario, I'm trying to allow Mongoose to save a record, and then iterate to the next record once the record saves. </p>
  
  <p>But the iterator inside of the Mongoose save function never gets incremented. In addition, it never throws any errors. But if I put the iterator (i) outside of the asynchronous call to Mongoose, it will work, provided the number of records that I try to load are not too big (I have successfully loaded 2,000 this way). </p>
  
  <p>So my questions are: Why isn't the iterator inside of the Mongoose save call ever incremented? And, more importantly, what is the best way to load a large data set into MongoDb using Mongoose?</p>
  
  <p>Rob</p>