title: Storing lots of duplicate strings. Looking for good hash function to save storage.
body: <p>I am looking for a good hash function to save on storage space while storing a lot of duplicate strings in Database. </p>
  
  <p>I have a DB where I have to store this
  DATE RATE DESCRIPTION
  xxx  yyyy zzzz</p>
  
  <p>100s of Millions of rows; The description is max say 1K string. The description string has high repeat; I will get many duplicates. To avoid that much (wasted) storage we are thinking of doing this</p>
  
  <p>Table 1
  DATE RATE DESC_ID</p>
  
  <p>Table 2
  DESC_ID DESCRIPTION</p>
  
  <p>Approach 1: DESC_ID == MD5 hash --> DB has primary key on DESCID; app generates the hash; DB writes are fast (that is what I am thinking)</p>
  
  <p>Approach 2: DESC_ID == DB generated ID; the unique key has to be DESCRIPTION; DB writes might be slower than the above approach. </p>
  
  <p>Question1: Should I stick with MD5 or are there better algos out there? Is it worthwhile to go to SHA-x functions for theoretically better collision avoidance, while taking a hit on extra storage an d compute time?</p>
  
  <p>Question2. Should I consider going to approach 2?</p>