title: How to pull large SQL server tables into C# for analysis
body: <p>I'm in need of a bit advice on how to best approach this problem. I inherited a project to build a reporting utility from an existing SQL server database. The database contains a "Raw Data" table where every production data point is dumped. The report needs to provide the average and Cpk of several of the data columns.</p>
  
  <p>The "Raw Data" table contains 25 columns, including an <code>ID</code> column, a <code>LotID</code> column, and a <code>TimeStamp</code> column, as well as 5 columns containing measurement data. Overall the table appears to have 20+ million records, and is essentially one huge flat file.</p>
  
  <p>What I need to do is provide two methods to search; by date range, and by lot ID. One lot ID can hold as many as 2M records.</p>
  
  <p>I started with developing a C# application that uses a simple SELECT query.</p>
  
  <pre><code>SELECT * 
  FROM tblRawData 
  WHERE [LotID] = "XXXX"
  </code></pre>
  
  <p>Then uses the <code>SqlDataAdapter.Fill</code> function to populate a <code>DataTable</code>. I also tried <code>SqlDataReader</code> and looped to populate the results into the <code>DataTable</code>.  </p>
  
  <p>The biggest problem I'm seeing is 90%+ memory usage (a bulk of it on the SQL Server process) with the occasional low memory warning and each query takes takes several minutes to run.</p>
  
  <p>Not being a SQL Server guru, I'm looking for advice as to whether this approach seems reasonable, or should I be looking to do something different?</p>