title: Pass large amounts of data between app domains quickly
body: <p>I have an application used to import a large dataset (millions of records) from one database to another, doing a diff in the process (IE removing things that were deleted, updating things, etc).  Due to many foreign key constraints and such and to try to speed up the processing of the application, it loads up the entire destination database into memory and then tries to load up parts of the source database and does an in-memory compare, updating the destination in memory as it goes.  In the end it writes these changes back to the destination.  The databases do not match one to one, so a single table in one may be multiple tables in the other, etc.</p>
  
  <p>So to my question: it currently takes hours to run this process (sometimes close to a day depending on the amount of data added/changed) and this makes it very difficult to debug.  Historically, when we encounter a bug, we have made a change, and then rerun the app which has to load all of the data into memory again (taking quite some time) and then run the import process until we get to the part we were at and then we cross our fingers and hope our change worked.  This isn't fun :(</p>
  
  <p>To speed up the debugging process I am making an architectural change by moving the import code into a separate dll that is loaded into a separate appdomain so that we can unload it, make changes, and reload it and try to run a section of the import again, picking up where we left off, and seeing if we get better results.  I thought that I was a genius when I came up with this plan :)  But it has a problem.  I either have to load up all the data from the destination database into the second appdomain and then, before unloading, copy it all to the first using the [Serializable] deal (this is really really slow when unloading and reloading the dll) or load the data in the host appdomain and reference it in the second using MarshalByRefObject (which has turned out to make the whole process slow it seems)</p>
  
  <p><strong>So my question is: How can I do this quickly?  Like, a minute max!  I would love to just copy the data as if it was just passed by reference and not have to actually do a full copy.</strong></p>
  
  <p>I was wondering if there was a better way to implement this so that the data could better be shared between the two or at least quickly passed between them.  I have searched and found things recommending the use of a database (we are loading the data in memory to AVOID the database) or things just saying to use MarshalByRefObject.  I'd love to do something that easy but it hasn't really worked yet.</p>
  
  <p>I read somewhere that loading a C++ dll or unmanaged dll will cause it to ignore app domains and could introduce some problems.  Is there anyway I could use this to my advantage, IE, load an unmanaged dll that holds my list for me or something, and use it to trick my application into using the same memory are for both appdomains so that the lists just stick around when I unload the other dll by unloading the app domain?</p>
  
  <p>I hope this makes sense.  It's my first question on here so if I've done a terrible job do help me out.  This has frustrated me for a few days now.</p>