title: How can I access the data in many large CSV files quickly from Perl?
body: <p>I have a number of scripts that currently read in a lot of data from some .CSV files. For efficiency, I use the <a href="http://search.cpan.org/perldoc/Text::CSV_XS" rel="nofollow noreferrer">Text::CSV_XS</a> module to read them in and then create a hash using one of the columns as an index. However, I have a <strong>lot</strong> of files and they are quite large. And each of the scripts needs to read in the data all over again. </p>
  
  <p>The question is: How can I have persistent storage of these Perl hashes so that all them can be read back in with a minimum of CPU?</p>
  
  <p>Combining the scripts is not an option. I wish...</p>
  
  <p>I applied the 2nd rule of optimization and used profiling to find that the vast majority of the CPU (about 90%) was in:</p>
  
  <pre><code>Text::CSV_XS::fields
  Text::CSV_XS::Parse
  Text::CSV_XS::parse
  </code></pre>
  
  <p>So, I made a test script that read in all the .CSV files (<strong>Text::CSV_XS</strong>), dumped them using the <strong>Storable</strong> module, and then went back and read them back in using the <strong>Storable</strong> module. I profiled this so I could see the CPU times:</p>
  
  <pre><code>$ c:/perl/bin/dprofpp.bat
  Total Elapsed Time = 1809.397 Seconds
    User+System Time = 950.5560 Seconds
  Exclusive Times
  %Time ExclSec CumulS #Calls sec/call Csec/c  Name
   25.6   243.6 243.66    126   1.9338 1.9338  Storable::pretrieve
   20.5   194.9 194.92 893448   0.0002 0.0002  Text::CSV_XS::fields
   9.49   90.19 90.198 893448   0.0001 0.0001  Text::CSV_XS::Parse
   7.48   71.07 71.072    126   0.5641 0.5641  Storable::pstore
   4.45   42.32 132.52 893448   0.0000 0.0001  Text::CSV_XS::parse
   (the rest was in terms of 0.07% or less and can be ignored)
  </code></pre>
  
  <p>So, using <strong>Storable</strong> costs about 25.6% to load back in as compared to <strong>Text::CSV_XS</strong> at about 35%. Not a lot of savings...</p>
  
  <p>Has anybody got a suggestion on how I can read in these data more efficiently?</p>
  
  <p>Thanks for your help.</p>