title: How to hash a large object (dataset) in Python?
body: <p>I would like to calculate a hash of a Python class containing a dataset for Machine Learning. The hash is meant to be used for caching, so I was thinking of <code>md5</code> or <code>sha1</code>.
  The problem is that most of the data is stored in NumPy arrays; these do not provide a <code>__hash__()</code> member. Currently I do a <code>pickle.dumps()</code> for each member and calculate a hash based on these strings. However, I found the following links indicating that the same object could lead to different serialization strings:</p>
  
  <ul>
  <li><a href="http://www.gossamer-threads.com/lists/python/python/735907" rel="noreferrer">Hash of None varies per machine</a></li>
  <li><a href="http://www.aminus.org/blogs/index.php/2007/11/03/pickle_dumps_not_suitable_for_hashing?blog=2" rel="noreferrer">Pickle.dumps not suitable for hashing</a></li>
  </ul>
  
  <p>What would be the best method to calculate a hash for a Python class containing Numpy arrays?</p>