title: How to calculate a hash for a string (url) in bash for wget caching
body: <p>I'm building a little tool that will download files using wget, reading the urls from different files. The same url may be present in different files; the url may even be present in one file several times. It would be inefficient to download a page several times (every time its url found in the list(s)).</p>
  
  <p>Thus, the simple approach is to save the downloaded file and to instruct wget not to download it again if it is already there.</p>
  
  <p>That would be very straightforward; however the urls are very long (many many GET parameters) and therefore cannot be used as such for filenames (wget gives the error 'Cannot write to... [] file name too long').</p>
  
  <p>So, I need to rename the downloaded files. But for the caching mechanism to work, the renaming scheme needs to implement "one url &lt;=> one name": if a given url can have multiple names, the caching does not work (ie, if I simply number the files in the order they are found, I won't let wget identify which urls have already been downloaded).</p>
  
  <p>The simplest renaming scheme would be to calculate an md5 hash <em>of the filename</em> (and <strong>not</strong> of the file itself, which is what md5sum does); that would ensure the filename is unique and that a given url results in always the same name.</p>
  
  <p>It's possible to do this in Perl, etc., but can it be done directly in bash or using a system utility (RedHat)?</p>