title: Reliable and fast method for identification of file duplicates
body: <p>I have a continuously growing set of files and have to ensure that there are no duplicates. By duplicate I mean identical at byte level.</p>
  
  <p>The files are getting collected from various system, some of them also providing hash codes of the files (but some don't). Some files may exist at multiple systems but should be imported only once.</p>
  
  <p>I want do avoid unnecessary file transfers and I thought that I just compare hash codes before actually copying. However, as I said some of these systems don't provide a hash code and some use MD5 which I read isn't secure anymore.</p>
  
  <p>My questions:</p>
  
  <ul>
  <li>Is comparing hash codes enought to determine identical files?</li>
  <li>What should I do when systems use different hash codes?</li>
  <li>What should I do when systems don't provide a hash code?</li>
  </ul>