title: Handling large amount of data
body: <p>We are looking at integrating our platform with Salesforce via Salesforce Outbound Messaging (SOM). Every time client updates an object(s) in Salesforce, SOM will call our Webservice endpoint with the updated object(s) (up to 100 objects in a single call). Our Webservice needs to update corresponding records in our database. </p>
  
  <p>SOM works quite well for our purposes except for 1 issue.</p>
  
  <p>Some clients do mass nightly updates. It is not unusual for 200,000-500,000 objects to be updated. This means we will get 2,000-5,000 calls with 100 objects in very short time period. Our Webservice will be overwhelmed with so much data specially if multiple clients do mass updates close to each other.</p>
  
  <p>To deal with this large volume/spike Webserver will create message on Application server for each object in the SOM call. Another process will take messages from Message Queue and update the database. </p>
  
  <p><a href="http://blogs.msdn.com/b/johnbreakwell/archive/2008/02/29/what-are-msmq-s-limits-if-i-had-a-farthing-for-every-time.aspx" rel="nofollow">MSMQ is only limited by hardware</a> so should be able handle millions of messages whilst we clear backlog.</p>
  
  <p>The main question is this good design for dealing large amount of data/webservice calls? Is there better method?</p>