title: InnoDB Table Bulk Insert
body: <p>I have a MySQL data table with about half-a-billion rows in it. We need to run calculations on this data by reading it, and the calculated data (which is a standardized form of the original data) needs to be written into another InnoDB table.</p>
  
  <p>The setup we currently have is a virtual cloud with a machine in it as well as the database, therefore the machine-DB connection is very fast.</p>
  
  <p>The calculations that occur on the data (as well as reading it) are very fast, and the bottleneck of this entire process is the insertion of the standardized data into the InnoDB tables (the standardized data contains a few indicies, though not long, which slows down the insertion).</p>
  
  <p>Unfortunately, we cannot modify certain system variables like innodb_log_file_size (we are using Amazon AWS) which would help increase insert performance.</p>
  
  <p>What would be our best best to push all this data onto MySQL? Since the calculation process is straightforward, I can pretty much write a Python script that takes the standardized data and outputs it in any format. Inserting this data on the fly as the calculations occur is painfully slow, and gets slower with time. </p>
  
  <p>I guess the question would be then, what is the best process (in terms of input format, and actual import) for inserting bulk data into InnoDB tables?</p>