title: Design advice. Using DataTable or List<MyObject> for a generic rule checker
body: <p>I have about 100,000 lines of generic data. Columns/Properties of this data are user definable and are of the usual data types (string, int, double, date). There will be about 50 columns/properties.</p>
  
  <p><strong>I have 2 needs:</strong><br>
  <li><strong>To be able to calculate new columns/properties using an expression</strong><br>
  e.g. Column3 = Column1 * Column2.<br>
  Ultimately I would like to be able to use external data using a callback, <br>e.g. Column3 = Column1 * <em>GetTemperature</em><br>
  The expression is relatively simple, maths operations, sum, count &amp; IF are the only necessary functions.<br>
  <li><strong>To be able to filter/group the data and perform aggregations</strong><br>
  e.g. Sum(Data.Column1) Where(Data.Column2 == "blah")
  <br><br>
  <strong>As far as I can see I have two options:</strong><br>
  1. Using a <strong>DataTable</strong>. <br>
    => Point 1 above is achieved by using DataColumn.Expression<br>
    => Point 2 above is acheived by using DataTable.DefaultView.RowFilter or DataTable.Select() &amp; C# code<br><br>
  2. Using a List of generic Objects each with a <strong>Dictionary&lt; string, object ></strong> to store the values.<br>
    => Point 1 could be achieved by something like NCalc<br>
    => Point 2 is achieved using LINQ</p>
  
  <pre>
  DataTable:
  Pros: DataColumn.Expression is inbuilt
  Cons: RowFilter & coding c# is not as "nice" as LINQ, 
        DataColumn.Expression does not support callbacks(?) 
        => workaround could be to get & replace external value when creating 
           the calculated column
  
  GenericList:
  Pros: LINQ syntax, NCalc supports callbacks
  Cons: Implementing NCalc/generic calc engine
  </pre>
  
  <p>Based on the above I would think a GenericList approach would win, but something I have not factored in is the performance which for some reason I think would be better with a datatable.<br>
  Does anyone have a gut feeling / experience with LINQ vs. DataTable performance? <br>
  How about NCalc?<br>
  As I said there are about 100,000 rows of data, with 50 columns, of which maybe 20 are calculated.
  In total about 50 rules will be run against the data, so in total there will be 5 million row/object scans.</p>
  
  <p>Would really appreciate any insights. Thx.<br>
  ps. Of course using a database + SQL &amp; Views etc. would be the easiest solution, but for various reasons can't be implemented.</p>