title: How efficient is Meteor's DDP at syncing very large collections?
body: <p>Meteor's DDP protocol works very well for syncing a small collection of data from a server to a browser-based client, which inherently limits the amount of data that is processed.</p>
  
  <p>However, consider a situation where Meteor is being used to sync a large collection from one server to another, or just the DDP protocol itself is used to sync one MongoDB with another. </p>
  
  <p><strong>How efficient is DDP in this case (computationally)? How well does it scale to several clients? Is the limit to performance only bandwidth or will DDP hit some CPU bound as well? What is the largest amount of data that can be reasonably synced over DDP right now?</strong> Is DDP just the wrong approach for doing this (see references below)?</p>
  
  <p>Some additional thoughts:</p>
  
  <ul>
  <li>As far as I know, the current version of DDP keeps track of each client's entire collection, so it can't be asymptotically very efficient.</li>
  <li><a href="http://meteorhacks.com/introducing-smart-collections.html" rel="nofollow noreferrer">Smart Collections</a> were created to improve the performance of server-to-client collection of syncing. But it's unclear to me if this is improving DDP or something else.</li>
  </ul>
  
  <p>See also: </p>
  
  <ul>
  <li><a href="https://stackoverflow.com/q/18734253/586086">How to implement real-time replication of MongoDB (or CouchDB) to many remote clients</a></li>
  <li><a href="https://stackoverflow.com/q/15931368/586086">DDP vs Straight MongoDB access for synching large amounts of data</a></li>
  </ul>
  
  <p><strong>EDIT</strong>:</p>
  
  <p>After some empirical experience with this, I have to conclude that the answer is "not very efficient". See <strong><a href="https://stackoverflow.com/a/21835534/586086">https://stackoverflow.com/a/21835534/586086</a></strong> for an explanation.</p>
  
  <p>Discussions with Meteor devs indicated that this problem will be addressed in the future with a revision of DDP and the publish-subscribe API, whereby the merge box will be removed and clients will handle merging. This will save CPU/memory on the server and allow for much larger datasets to be sent over the wire.</p>