title: Best way to work with large amounts of CSV data quickly
body: <p>I have large CSV datasets (10M+ lines) that need to be processed. I have two other files that need to be referenced for the output—they contain data that amplifies what we know about the millions of lines in the CSV file. The goal is to output a new CSV file that has each record merged with the additional information from the other files.</p>
  
  <p>Imagine that the large CSV file has transactions but the customer information and billing information is recorded in two other files and we want to output a new CSV that has each transaction linked to the customer ID and account ID, etc.</p>
  
  <p>A colleague has a functional program written in Java to do this but it is very slow. The reason is that the CSV file with the millions of lines has to be walked through many, many, many times apparently.</p>
  
  <p>My question is—yes, I am getting to it—how should I approach this in Ruby? The goal is for it to be faster (18+ hours right now with very little CPU activity)</p>
  
  <p>Can I load this many records into memory? If so, how should I do it?</p>
  
  <p>I know this is a little vague. Just looking for ideas as this is a little new to me.</p>