title: Fastest, Efficient, Elegant way of Parsing Strings to Dynamic types?
body: <p>I'm looking for the fastest (generic approach) to converting strings into various data types on the go. </p>
  
  <p>I am parsing large text data files generated by a something (files are several megabytes in size). This particulare function reads lines in the text file, parses each line into columns based on delimitters and places the parsed values into a .NET DataTable. This is later inserted into a database. My bottleneck by FAR is the string conversions (Convert and TypeConverter).</p>
  
  <p>I have to go with a dynamic way (i.e. staying away form "Convert.ToInt32" etc...) because I never know what types are going to be in the files. The type is determined by earlier configuration during runtime. </p>
  
  <p>So far I have tried the following and both take several minutes to parse a file. Note that 
  if I comment out this one line it runs in only a few hundred milliseconds.</p>
  
  <pre><code>row[i] = Convert.ChangeType(columnString, dataType);
  </code></pre>
  
  <p>AND </p>
  
  <pre><code>TypeConverter typeConverter = TypeDescriptor.GetConverter(type);
  row[i] = typeConverter.ConvertFromString(null, cultureInfo, columnString);
  </code></pre>
  
  <p>If anyone knows of a faster way that is generic like this I would like to know about it. Or if my whole approach just sucks for some reason I'm open to suggestions. But please don't point me to non-generic approaches using hard coded types; that is simply not an option here.</p>
  
  <p><strong>UPDATE - Multi-threading to Improve Performance Test</strong></p>
  
  <p>In order to improve performance I have looked into splitting up parsing tasks to multiple threads. I found that the speed increased somewhat but still not as much as I had hoped. However, here are my results for those who are interested.</p>
  
  <p><em><strong>System:</em></strong></p>
  
  <p>Intel Xenon 3.3GHz Quad Core E3-1245</p>
  
  <p>Memory: 12.0 GB</p>
  
  <p>Windows 7 Enterprise x64</p>
  
  <p><em><strong>Test:</em></strong></p>
  
  <p>The test function is this:</p>
  
  <p>(1) Receive an array of strings. (2) Split the string by delimitters. (3) Parse strings into data types and store them in a row. (4) Add row to data table. (5) Repeat (2)-(4) until finished.</p>
  
  <p>The test included 1000 strings, each string being parsed into 16 columns, so that is 16000 string conversions total. I tested single thread, 4 threads (because of quad core), and 8 threads (because of hyper-threading). Since I'm only crunching data here I doubt adding more threads than this would do any good. So for the single thread it parses 1000 strings, 4 threads parse 250 strings each, and 8 threads parse 125 strings each. Also I tested a few different ways of using threads: thread creation, thread pool, tasks, and function objects.</p>
  
  <p><em><strong>Results:</em></strong>
  Result times are in Milliseconds.</p>
  
  <p>Single Thread:</p>
  
  <ul>
  <li>Method Call: 17720</li>
  </ul>
  
  <p>4 Threads</p>
  
  <ul>
  <li>Parameterized Thread Start: 13836</li>
  <li>ThreadPool.QueueUserWorkItem: 14075</li>
  <li>Task.Factory.StartNew: 16798</li>
  <li>Func BeginInvoke EndInvoke: 16733</li>
  </ul>
  
  <p>8 Threads</p>
  
  <ul>
  <li>Parameterized Thread Start: 12591</li>
  <li>ThreadPool.QueueUserWorkItem: 13832</li>
  <li>Task.Factory.StartNew: 15877</li>
  <li>Func BeginInvoke EndInvoke: 16395</li>
  </ul>
  
  <p>As you can see the fastest is using Parameterized Thread Start with 8 threads (the number of my logical cores). However it does not beat using 4 threads by much and is only about 29% faster than using a single core. Of course results will vary by machine. Also I stuck with a </p>
  
  <pre><code>    Dictionary&lt;Type, TypeConverter&gt;
  </code></pre>
  
  <p>cache for string parsing as using arrays of type converters did not offer a noticeable performance increase and having one shared cached type converter is more maintainable rather than creating arrays all over the place when I need them.</p>
  
  <p><strong>ANOTHER UPDATE:</strong></p>
  
  <p>Ok so I ran some more tests to see if I could squeeze some more performance out and I found some interesting things. I decided to stick with 8 threads, all started from the Parameterized Thread Start method (which was the fastest of my previous tests). The same test as above was run, just with different parsing algorithms.
  I noticed that</p>
  
  <pre><code>    Convert.ChangeType and TypeConverter
  </code></pre>
  
  <p>take about the same amount of time. Type specific converters like </p>
  
  <pre><code>    int.TryParse
  </code></pre>
  
  <p>are slightly faster but not an option for me since my types are dynamic. ricovox had some good advice about exception handling. My data does indeed have invalid data, some integer columns will put a dash '-' for empty numbers, so type converters blow up at that: meaning every row I parse I have at least one exception, thats 1000 exceptions! Very time consuming.</p>
  
  <p>Btw this is how I do my conversions with TypeConverter. Extensions is just a static class and GetTypeConverter just returns a cahced TypeConverter. If an exceptions is thrown during the conversion, a default value is used.</p>
  
  <pre><code>public static Object ConvertTo(this String arg, CultureInfo cultureInfo, Type type, Object defaultValue)
  {
    Object value;
    TypeConverter typeConverter = Extensions.GetTypeConverter(type);
  
    try
    {
      // Try converting the string.
      value = typeConverter.ConvertFromString(null, cultureInfo, arg);
    }
    catch
    {
      // If the conversion fails then use the default value.
      value = defaultValue;
    }
  
    return value;
  }
  </code></pre>
  
  <p><em><strong>Results:</em></strong></p>
  
  <p>Same test on 8 threads - parse 1000 lines, 16 columns each, 250 lines per thread.</p>
  
  <p>So I did 3 new things. </p>
  
  <p>1 - Run the test: check for known invalid types before parsing to minimize exceptions.
  i.e. if(!Char.IsDigit(c)) value = 0; OR columnString.Contains('-') etc...</p>
  
  <p><em>Runtime: 29ms</em></p>
  
  <p>2 - Run the test: use custom parsing algorithms that have try catch blocks.</p>
  
  <p><em>Runtime: 12424ms</em></p>
  
  <p>3 - Run the test: use custom parsing algorithms checking for invalid types before parsing to minimize exceptions.</p>
  
  <p><em>Runtime 15ms</em></p>
  
  <p>Wow! As you can see eliminating the exceptions made a world of difference. I never realized how expensive exceptions really were! So If I minimize my exceptions to TRULY unknown cases, then the parsing algorithm runs three orders of magnitude faster. I'm considering this absolutely solved. I believe I will keep the dynamic type conversion with TypeConverter, it is only a few milliseconds slower. Checking for known invalid types before converting avoids exceptions and that speeds things up incredibly! Thanks to ricovox  for pointing that out which made me test this further.</p>