title: rsync, directory containing 3 million files, can it keep up?
body: <p>I am working on a mac computer using bash commands via terminal.</p>
  
  <p>I am running a DNA sequencer that generates ~3-5 million files over the course of 48 hours. For speed reasons these files are saved to the computer's SSD. I would like to use fswatch and rsync commands to monitor the directory and transfer these files to a server as they are being generated to reduce the long transfer times post sequencing.</p>
  
  <p>Here is the command I have come up with.</p>
  
  <pre><code>fswatch -o ./ | (while read; do rsync -r -t /Source/Directory /Destination/Directory; done)
  </code></pre>
  
  <p>But I am worried that due to the large number of files >3 million and large total size > 100gb these tools might struggle to keep up. Is there a better strategy?</p>
  
  <p>Thanks for your help!</p>