title: What causes R to crash while working with large data sets?
body: <p><strong>Problem:</strong></p>
  
  <p>I have multiple datasets of transactional data that I use to predict an event (binary classification as outcome). One of them has 10,587,989 rows with 23 columns. I am attempting to run <code>gradient boosting</code> with 10 fold cv and ctree (<code>package:party</code>) but every time I run these models my system crashes. </p>
  
  <p><strong>Hardware:</strong></p>
  
  <p>16 cores, 48 gig of RAM, 48 gig of SWAP</p>
  
  <p><strong>Question:</strong> </p>
  
  <p>What causes R to crash while working with large data sets even after utilizing parallel processing, adding more memory, bouncing the system? </p>
  
  <p><strong>Things I have tried:</strong></p>
  
  <ul>
  <li><p>Enabled parallel processing through <code>doParallel</code>, execute xgBoost
  through <code>caret</code>, I see every core lighting up and RAM and swap being
  fully utilized through <code>top</code> function in linux but it eventually
  crashes everytime.</p></li>
  <li><p>Bounced the RStudio server, rebooted the system as initial
  maneuvering but problem persists.</p></li>
  </ul>
  
  <p>I did find people commenting about H2O. I also reached out to a vendor and asked him for a solution, he suggested Sparkly but you need Hadoop layer in your server to run Sparkly. </p>