title: Importing a CSV with random number of columns
body: <p>I have a script that reads some measurement data in CSV form, and then does all kinds of plotting and stuff with it.</p>
  
  <p>Now I have a new dataset, where some idiot deemed it helpful to add some random comments at the end of the line, like so:</p>
  
  <pre><code>01.02.1988 00:00:00   ;   204.94     
  01.03.1988 00:00:00   ;   204.87 ; something
  01.04.1988 00:00:00   ;   205.41     
  01.05.1988 00:00:00   ;   205.64 ; something ; something else    
  01.06.1988 00:00:00   ;   205.59 ; also something    
  01.07.1988 00:00:00   ;   205.24
  </code></pre>
  
  <p>which gives me a nice</p>
  
  <pre><code>ValueError: Expected 2 fields in line 36, saw 3
  </code></pre>
  
  <p>and so on.</p>
  
  <p>According to <a href="https://stackoverflow.com/questions/20154303/pandas-read-csv-expects-wrong-number-of-columns-with-ragged-csv-file">this</a> and <a href="https://stackoverflow.com/questions/27020216/import-csv-with-different-number-of-columns-per-row-using-pandas">this</a> I have to use the <code>names=['whatever','else']</code> argument when reading it.</p>
  
  <p>But somehow this goes all kinds of wrong. So here's some examples:</p>
  
  <p><strong>CSV file</strong></p>
  
  <pre><code>Stuff
  more stuff I dont need
  Date;level;crap1;crap2;crap3;crap4;crap5;crap6
  01.01.1988 00:00:00   ;   204.87     
  01.02.1988 00:00:00   ;   204.94     
  01.03.1988 00:00:00   ;   204.87 
  </code></pre>
  
  <p>The "nice" header is obviously "handmade", but I should just be able to skip it!?</p>
  
  <p><strong>CSV reader</strong></p>
  
  <pre><code>ValReader = pd.read_csv(csv_list[counter],sep=r'\s*;',skiprows=DateStart,names=['Date','level','crap1','crap2','crap3','crap4','crap5','crap6'],usecols=['Date','level'],index_col='Date',dayfirst=True,parse_dates=True)
  </code></pre>
  
  <p><strong>What I get</strong></p>
  
  <pre><code>print 'ValReader'
                           level
  Date                          
  Date                     level
  01.04.2003 00:00:00     200.76
  01.05.2003 00:00:00     200.64
  01.06.2003 00:00:00     200.53
  </code></pre>
  
  <p>Which following that, causes level to get handled as string.</p>
  
  <p>OK, easy, that manual header line in the CSV (which worked well in a previous version, that only had to handle good data) is the culprit, so I just set <code>skiprows</code> to <code>skiprows=DateStart+1</code>, but that results in </p>
  
  <pre><code>ValueError: Number of passed names did not match number of header fields in the file
  </code></pre>
  
  <p>So obviously I got utterly lost in how pandas handles the names and positions of columns.</p>