title: Finding maximum CSV field sizes for large file using Ruby
body: <p>I'm trying to determine maximum field sizes for a large CSV file (~5GB) with over 300 fields that I want to dump into a MySQL table. The CSV file schema I have for the file gives incorrect maximum field lengths, so I run into errors on the table import. I'm running Ruby 2.0 on Windows. </p>
  
  <p>I'm using an array to store the maximum field lengths according to the field's index (or column) location, i.e. ignoring the fields actual name in the header. I've tried fancier things like using hashes, inject, and zip, etc, but it seems that a simple array works fastest here.</p>
  
  <pre><code> field_lengths[0] = Max length of first field   
   field_lengths[1] = Max length of second field
   etc.
  </code></pre>
  
  <p>The file is too large to slurp at once or parse column-wise using CSV. So, I open the CSV file and use CSV#foreach to parse each line (ignoring the header using the :headers => true option). For each row, I loop through the parsed array of field values and compare the field's length with the current maximum length stored in the field_length array. I realize there are much easier ways to do this with smaller files. This method works OK for larger files, but I still haven't been able to make it to the end of my particular file using this method. </p>
  
  <p>To get around not being able to finish the file, I currently define a number of lines to read including the header (=n), and break once I've reached the n-th line. In the example below, I read 101 lines from the CSV file. (1 header row + 100 actual data rows). I'm not sure how many total rows are in the file since the process hasn't finished.</p>
  
  <pre><code> require 'csv'
   require 'pp'
  
   data_file = 'big_file.csv'
  
   # We're only reading the first 101 lines in this example
   n = 101
   field_lengths = []
  
   File.open(data_file) do |f|
     CSV.foreach(f, :headers =&gt; true, :header_converters =&gt; :symbol) do |csv_row|
         break if $. &gt; n
         csv_row.fields.each_with_index do |a,i|
             field_lengths[i] ||= a.to_s.length
             field_lengths[i] = a.to_s.length if field_lengths[i] &lt; a.to_s.length
         end
     end
   end
  
   pp field_lengths
  </code></pre>
  
  <p>IO#read can read a certain number of bytes, but if I parse the file by bytes the records may get split. Does anyone have alternate suggestions for parsing the CSV file, by splitting it up into smaller files? O'Reilly's Ruby Cookbook (Lucas Carlson &amp; Leonard Richardson, 2006, 1st ed), suggests breaking a large file into chunks (as below), but I'm not sure how to extend it to this example, particularly dealing with the line breaks etc. </p>
  
  <pre><code> class File 
      def each_chunk(chunk_size = 1024)
          yield read(chunk_size) until eof?
      end
   end
  
   open("bigfile.txt") do |f|
      f.each_chunk(15) {|chunk| puts chunk}
   end
  </code></pre>