title: Reading 40 GB csv file into R using bigmemory
body: <p>The title is pretty self explanatory here but I will elaborate as follows. Some of my current techniques in attacking this problem are based on the solutions presented in <a href="https://stackoverflow.com/questions/3094866/trimming-a-huge-3-5-gb-csv-file-to-read-into-r">this</a> question. However, I am facing several challenges and constraints so I was wondering if someone might attempt to take a stab at this problem. I am trying to figure out the problem using the <a href="http://www.bigmemory.org/" rel="nofollow noreferrer">bigmemory</a> package but I have been running into difficulties. </p>
  
  <p><strong>Present Constraints:</strong></p>
  
  <ul>
  <li>Using a linux server with 16 GB of RAM</li>
  <li>Size of 40 GB CSV</li>
  <li>No of rows: 67,194,126,114</li>
  </ul>
  
  <p><strong>Challenges</strong></p>
  
  <ul>
  <li>Need to be able to randomly sample smaller datasets (5-10 Million rows) from a big.matrix or equivalent data structure.</li>
  <li>Need to be able to remove any row with a single instance of NULL while parsing into a big.matrix or equivalent data structure. </li>
  </ul>
  
  <p>So far, results are not good. Evidently, I am failing at something or maybe, I just don't understand the <a href="http://cran.r-project.org/web/packages/bigmemory/bigmemory.pdf" rel="nofollow noreferrer">bigmemory documentation</a> well enough. So, I thought I would ask here to see if anyone has used </p>
  
  <p>Any tips, advice on this line of attack etc.? Or should I change to something else? I apologize if this question is very similar to the previous but I thought by scale of data was about 20 times bigger than the previous questions. Thanks !</p>