title: sklearn FeatureHasher output has lots of collisions and unused columns
body: <p>I am using <a href="https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.FeatureHasher.html" rel="nofollow noreferrer"><code>sklearn.feature_extraction.FeatureHasher</code></a> to encode categorical variables for machine learning. My categorical variables are numeric IDs, like 1, 2, 3, 6, 18, 19, 20, ... . In total I have 18,000 unique IDs and the highest ID is 28,000.</p>
  <p>I want to hash them to encode them as categories since One-hot encoding is out of the picture as it would create 18,000 columns and my dataset already has 4,000,000 rows and that would be painful.</p>
  <p>I can't show my dataframe so I illustrate on a sample dataframe.</p>
  <pre class="lang-py prettyprint-override"><code>from sklearn.feature_extraction import FeatureHasher
  import numpy as np
  import pandas as pd
  import matplotlib.pyplot as plt
  </code></pre>
  <pre class="lang-py prettyprint-override"><code>tmpdf = pd.DataFrame(np.arange(10000), columns=[&quot;test&quot;])
  </code></pre>
  <p>produces</p>
  <pre><code>      test
  0        0
  1        1
  2        2
  3        3
  4        4
  ...    ...
  9995  9995
  9996  9996
  9997  9997
  9998  9998
  9999  9999
  
  [10000 rows x 1 columns]
  </code></pre>
  <p>Now if I hash with a dictionary:</p>
  <pre class="lang-py prettyprint-override"><code>H = FeatureHasher(n_features=50, input_type=&quot;dict&quot;)
  
  cdict = [{str(i) : 1} for i in range(10000)]
  arr = H.transform(cdict)
  
  plt.matshow(arr.toarray()[::100], cmap=&quot;tab10&quot;, vmin=-5, vmax=4)
  plt.colorbar(shrink=0.4)
  plt.show()
  </code></pre>
  <p>this produces:</p>
  <p><a href="https://i.stack.imgur.com/7HEBYl.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/7HEBYl.png" alt="enter image description here" /></a></p>
  <p>If instead I hash with strings:</p>
  <pre class="lang-py prettyprint-override"><code>H = FeatureHasher(n_features=50, input_type=&quot;string&quot;)
  
  arr = H.transform(tmpdf[&quot;test&quot;].astype(str))
  
  plt.matshow(arr.toarray()[::100], cmap=&quot;tab10&quot;, vmin=-5, vmax=4)
  plt.colorbar(shrink=0.4)
  plt.show()
  </code></pre>
  <p>I get:</p>
  <p><a href="https://i.stack.imgur.com/k7Hfjl.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/k7Hfjl.png" alt="enter image description here" /></a></p>
  <h2>Question:</h2>
  <p>Why does the output of the string based hashing look so weird? looks like a lot of collisions are happening as a <strong>lot of columns remain completely unused</strong>... Am I using it wrong? For my task of encoding user IDs, is there a better way to do it?</p>