title: Strategies for working with large amounts of image data
body: <p>Technology stack: C# / .NET 4 / WinForms</p>
  
  <p>Background:</p>
  
  <p>The project on which I am working is a visualization application for a series of image stacks. Specifically, each image stack is aligned to a grid, show the same image at any one time, and processing functions are applied to the images currently in view. The image stacks themselves are 150-300 MB and each image is 512KB-1MB. A typical data set will consist of ~100 image stacks.</p>
  
  <p>Question:</p>
  
  <p>To try and work with this amount of data, I am using several technqiues:</p>
  
  <ul>
  <li>memory mapped files: Image stacks are loaded from disk at application launch</li>
  <li>compilation under x64 with unsafe code allowed: Obviously I need 64-bit address space for files of this size. I am moving the currently displayed image from the memory mapped file to to a method that generates a bitmap via Marshal.Copy with unsafe pointers. </li>
  <li>System.Threading.Tasks: I'm using parallel loops for processing where possible</li>
  <li>System.Drawing.BufferedGraphicsContext: Each image stack has one active image which is composited onto a BufferedGraphicsContext before being passed to a PictureBox for display to the user.</li>
  <li>High-end system requirements: quad-core CPU or better, SSD, 12GB memory, etc</li>
  </ul>
  
  <p>Yet even using all of the above, the reponsiveness leaves much to be desired. Using the SysInternals Process Explorer, CPU utilization is low (&lt; 25%) while memory usage climbs to the limit before garbage collection occurs.</p>
  
  <p>Profiling shows that most of the execution time is spent getting data out of the memory mapped files. I assume it's waiting as the OS pages the requested memory back into active memory?</p>
  
  <p>What else could I do to improve performance?</p>
  
  <p>Note:</p>
  
  <ul>
  <li>Most, if not all, image stacks will be viewable at the same time so clipping to the current viewport may not yield much speed.</li>
  <li>Resizing for display is an option, but the complete original data must still be available at all times for processing so it seems that would just be an extra step.</li>
  </ul>
  
  <p><strong>Update 1:</strong></p>
  
  <ul>
  <li>For memory, my development box only has 6 GB (and I'm attempting to load fewer files as a result), but the deployment system will have 24 GB.</li>
  <li>I am looking into using SSE optimizations through the Intel Performance Primitives and GPU accelleration via CUDA.</li>
  <li>The reason I am attempting to load all of the data into memory is because an important visualization step is cycling through the image stacks at 15-60 Hz and I was afraid of thrashing.</li>
  </ul>