title: Importing badly formed CSV into SQLite
body: <p>I am trying to parse and analyse log files by importing some large CSV files with millions of lines in to SQLite.<br />
  I was using my .NET program to achieve the same thing on smaller log files through LINQ with out any database.<br />
  But I was facing performance and scalability issues.<br />
  Then I found SQLite might be the best option to offload performance issues from my .NET program.
  I imported the log files using the following commands from SQLite command line.<br />
  (Names are replaced to just avoid providing unnecessary information):</p>
  <pre><code>sqlite&gt; Create Table MLogTable(DateTimeC text, Field1 text, Field2 text, Field3 text, Field4 text, Field5 text, Field6 text, Field7 text, Field8 text, Field9 text, Field10 text);      
  </code></pre>
  <p><code>sqlite&gt; .separator &quot;,&quot;</code></p>
  <pre><code>sqlite&gt; .import &quot;D:\\logs\\xyz\\abc.log&quot; MLogTable
  </code></pre>
  <p>The logging mechanism seems to have a problem and i get some incomplete lines in the Log file.
  For example:</p>
  <blockquote>
  <p>expected format is (f1, f2, f3 f4, f5, f6, f7, f8, f9 ,f10, f11)</p>
  <p>but actual format of few lines is (f1, f2)</p>
  </blockquote>
  <p>So the log file has few lines(out of millions of lines) not as expected and on import it results the following kind of error:</p>
  <blockquote>
  <p>Error: D:\logs\xyz\abc.log line 4499214: expected 11 columns of data but found 2</p>
  </blockquote>
  <p>The number of erroneous lines is very less and i have no control over how it can be corrected while logging.<br />
  So those lines need to be ignored while SQLite import is done.<br />
  Is it possible with SQLite?</p>