title: Efficient way to process CSV file into a numpy array
body: <p>CSV file may not be clean (lines with inconsistent number of elements), unclean lines would need to be disregarded.
  String manipulation is required during processing.</p>
  
  <p>Example input:</p>
  
  <pre><code>20150701 20:00:15.173,0.5019,0.91665
  </code></pre>
  
  <p>Desired output: float32 (pseudo-date, seconds in the day, f3, f4)</p>
  
  <pre><code>0.150701 72015.173 0.5019 0.91665 (+ the trailing trash floats usually get)
  </code></pre>
  
  <p>The CSV file is also very big, the numpy array in memory would be expected to take 5-10 GB, CSV file is over 30GB.</p>
  
  <p>Looking for an efficient way to process the CSV file and end up with a numpy array.</p>
  
  <p>Current solution: use csv module, process line by line and use a list() as a buffer that later gets turned to numpy array with asarray(). Problem is, during the turning process memory consumption is doubled and the copying process adds execution overhead.</p>
  
  <p>Numpy's genfromtxt and loadtxt don't appear to be able to process the data as desired.</p>