title: Python: Analysis on CSV files 100,000 lines x 40 columns
body: <p>I have about a 100 csv files each 100,000 x 40 <strike>rows</strike> columns. I'd like to do some statistical analysis on it, pull out some sample data, plot general trends, do variance and R-square analysis, and plot some spectra diagrams. For now, I'm considering numpy for the analysis.</p>
  
  <p>I was wondering what issues should I expect with such large files? I've already checked for erroneous data. What are your recommendations on doing statistical analysis? would it be better if I just split the files and do the whole thing in Excel? </p>