title: Fast String Array (Individual) Hashing in Python
body: <p>As the title suggests, I would love some suggestions on how to make the following code faster. I have tried multiple methods including utilizing Numba, a combination of the <code>np.fromiter()</code> and <code>map()</code> functions, as well as Numpy vectorizing the hash function so as to get it all to work faster.</p>
  <p>Take a look at the code below! Below the code are the results of the code from running it on my machine.</p>
  <p>Some notes for the future:</p>
  <ul>
  <li>the <code>CODES</code> variable holds the dimensions of the arrays that need to be hashed (generated random strings for StackOverflow purposes)</li>
  <li>the first compilation time for Numba is quite high, but I am willing to take on this cost if it means speeding up the code in the long run when running hundreds of thousands of times</li>
  </ul>
  <pre><code>import numpy as np
  import numba as nb
  import time
  
  # GENERATE RANDOM &quot;&lt;U64&quot; CHARACTER ARRAYS
  A, Z = np.array([&quot;A&quot;,&quot;Z&quot;]).view(&quot;int32&quot;) 
  
  CODES = (25058, 64272, 61425)
  LENGTH = 64
  
  tmp1 = np.random.randint(low = A, high = Z, size = CODES[0] * LENGTH, dtype = &quot;int32&quot;).view(f&quot;U{LENGTH}&quot;)
  tmp2 = np.random.randint(low = A, high = Z, size = CODES[1] * LENGTH, dtype = &quot;int32&quot;).view(f&quot;U{LENGTH}&quot;)
  tmp3 = np.random.randint(low = A, high = Z, size = CODES[2] * LENGTH, dtype = &quot;int32&quot;).view(f&quot;U{LENGTH}&quot;)
  
  # NUMBA ITERATION AND HASHING ONE BY ONE
  @nb.jit(nopython=True, fastmath=True, parallel=True, cache=True)
  def form_hashed_array(to_hash: np.ndarray) -&gt; np.ndarray:
      hashed = np.empty_like(to_hash, dtype=np.int64)
  
      for i in nb.prange(len(to_hash)):
          hashed[i] = hash(to_hash[i])
  
      return hashed
  
  print(&quot;--------&quot;)
  t2 = time.monotonic()
  
  t = time.monotonic()
  tmp4 = form_hashed_array(tmp1)
  print(time.monotonic() - t) # this first one will be larger due to compilation time
  
  t = time.monotonic()
  tmp5 = form_hashed_array(tmp2)
  print(time.monotonic() - t)
  
  t = time.monotonic()
  tmp6 = form_hashed_array(tmp3)
  print(time.monotonic() - t)
  
  print(&quot;NUMBA ITERATION TOOK: &quot; + str(time.monotonic() - t2))
  
  # MAP + FROMITER COMBINATION
  print(&quot;--------&quot;)
  t2 = time.monotonic()
  
  t = time.monotonic()
  tmp7 = np.fromiter((map(hash, tmp1)), dtype=np.int64)
  print(time.monotonic() - t)
  
  t=time.monotonic()
  tmp8 = np.fromiter((map(hash, tmp2)), dtype=np.int64)
  print(time.monotonic()-t)
  
  t = time.monotonic()
  tmp9 = np.fromiter((map(hash, tmp3)), dtype=np.int64)
  print(time.monotonic()-t)
  
  print(&quot;MAP + FROMITER COMBINATION TOOK : &quot; + str(time.monotonic()-t2))
  
  # NUMPY FUNCTION VECTORIZATION
  vfunc = np.vectorize(hash)
  
  print(&quot;--------&quot;)
  t2 = time.monotonic()
  
  t = time.monotonic()
  tmp10 = vfunc(tmp1)
  print(time.monotonic() - t)
  
  t = time.monotonic()
  tmp11 = vfunc(tmp2)
  print(time.monotonic() - t)
  
  t = time.monotonic()
  tmp12 = vfunc(tmp3)
  print(time.monotonic() - t)
  
  print(&quot;NUMPY FUNCTION VECTORIZATION TOOK: &quot; + str(time.monotonic()-t2))
  
  # SANITY CHECKS
  print(&quot;--------&quot;)
  print(not (tmp4 - tmp7).any() and not (tmp7 - tmp10).any(), end=&quot; &quot;)
  print(not (tmp5 - tmp8).any() and not (tmp8 - tmp11).any(), end=&quot; &quot;)
  print(not (tmp6 - tmp9).any() and not (tmp9 - tmp12).any())
  print(&quot;--------&quot;)
  
  breakpoint()
  </code></pre>
  <p>This code will be run a significant number of times, so it is important that it is as fast as possible.</p>
  <pre><code>--------
  6.9208437809720635
  0.08914285799255595
  0.09502507897559553
  NUMBA ITERATION TOOK: 7.1051117710303515
  --------
  0.009926816972438246
  0.02683716599131003
  0.026946138008497655
  MAP + FROMITER COMBINATION TOOK : 0.06381386297289282
  --------
  0.011753249040339142
  0.02864329604199156
  0.029279633017722517
  NUMPY FUNCTION VECTORIZATION TOOK: 0.06976548000238836
  --------
  True True True
  --------
  </code></pre>
  <p>Thanks for the help in advance!</p>