title: Rsync folder with a million files, but very small incremental daily updates
body: <p>we run an <code>rsync</code> on a large folder. This has close to a million files inside it including html, jsp, gif/jpg, etc. We only need to of course incrementally update files. Just a few JSP and HTML files are updated in this folder, and we need this server to <code>rsync</code> to a different server, same folder. </p>
  
  <p>Sometimes rsync is running quite slow these days, so one of our IT team members created this command: </p>
  
  <pre><code>find /usr/home/foldername \
  -type f -name *.jsp -exec \
  grep -l &lt;ssi:include src=[^$]*${ {} ;`
  </code></pre>
  
  <p>This looks for only specific files which have <code>JSP</code> extension and which contain certain kinds of text, because these are the files which we need to <code>rsync</code>. But this command is consuming a lot of memory. I think it's a stupid way to rsync, but I'm being told this is how things will work. </p>
  
  <p>Some googling suggests that this should work on this folder too: </p>
  
  <pre><code>rsync -a --update --progress --rsh --partial /usr/home/foldername /destination/server 
  </code></pre>
  
  <p>I'm worried that this will be too slow on a daily basis, but I can't imagine why this will be slower than that silly find option that our IT folks are recommending. Any ideas about large directory rsyncs in the real world? </p>