title: Efficient retrieval of large data sets in Mongo mapper?
body: <p>I am storing a large amount of Twitter data, and would like to retrieve about 500k records for data processing at a time. I have a TwitterTweet mongo document that contains basic tweet data, and try to retrieve it as follows:</p>
  
  <p><code>weekly_tweets = TwitterTweet.all(:created_at.gt =&gt; 1.week.ago, :fields =&gt; [:created_at, :text, :from_user])</code></p>
  
  <p>Trouble is, this take up a LOT of time and memory - is there any way to make this more scalable and efficient. I have thought of using map reduce, but it looks very complicated for what I want to do - text processing and regexp stuff on the tweets.</p>