title: Python - Feature hashing on list of dictionaries with lists of strings
body: <p>I want to be able to take a list of dictionaries (records) in which some of its columns have a list of values as the value for the cell. Here is an example</p>
  
  <pre><code>[{'fruit': 'apple', 'age': 27}, {'fruit':['apple', 'banana'], 'age': 32}]
  </code></pre>
  
  <p>How can I take this input and perform a feature hash on it (in my data set I have thousands of columns). Currently I am using one hot encoding, but this seems to consume a lot of ram (more than what I have on my system).</p>
  
  <p>I tried taking my dataset as above and got an error:</p>
  
  <pre><code>x__ = h.transform(data)
  
  Traceback (most recent call last):
  
    File "&lt;ipython-input-14-db4adc5ec623&gt;", line 1, in &lt;module&gt;
      x__ = h.transform(data)
  
    File "/usr/local/lib/python2.7/dist-packages/sklearn/feature_extraction/hashing.py", line 142, in transform
      _hashing.transform(raw_X, self.n_features, self.dtype)
  
    File "sklearn/feature_extraction/_hashing.pyx", line 52, in sklearn.feature_extraction._hashing.transform (sklearn/feature_extraction/_hashing.c:2103)
  
  TypeError: a float is required
  </code></pre>
  
  <p>I also tried to turn it into a dataframe and pass it to the hasher:</p>
  
  <pre><code>x__ = h.transform(x_y_dataframe)
  
  Traceback (most recent call last):
  
    File "&lt;ipython-input-15-109e7f8018f3&gt;", line 1, in &lt;module&gt;
      x__ = h.transform(x_y_dataframe)
  
    File "/usr/local/lib/python2.7/dist-packages/sklearn/feature_extraction/hashing.py", line 142, in transform
      _hashing.transform(raw_X, self.n_features, self.dtype)
  
    File "sklearn/feature_extraction/_hashing.pyx", line 46, in sklearn.feature_extraction._hashing.transform (sklearn/feature_extraction/_hashing.c:1928)
  
    File "/usr/local/lib/python2.7/dist-packages/sklearn/feature_extraction/hashing.py", line 138, in &lt;genexpr&gt;
      raw_X = (_iteritems(d) for d in raw_X)
  
    File "/usr/local/lib/python2.7/dist-packages/sklearn/feature_extraction/hashing.py", line 15, in _iteritems
      return d.iteritems() if hasattr(d, "iteritems") else d.items()
  
  AttributeError: 'unicode' object has no attribute 'items'
  </code></pre>
  
  <p>Any Idea how I can implement this either with pandas or sklearn? Or maybe I can build my dummy variables a few thousand rows at a time?</p>
  
  <p>Here is How I am getting my dummy variables using pandas:</p>
  
  <pre><code>def one_hot_encode(categorical_labels):
      res = []
      tmp = None
      for col in categorical_labels:
          v = x[col].astype(str).str.strip('[]').str.get_dummies(', ')#cant set a prefix
          if len(res) == 2:
              tmp = pandas.concat(res, axis=1)
              del res
              res = []
              res.append(tmp)
              del tmp
              tmp = None
          else:
              res.append(v)
      result = pandas.concat(res, axis=1)
      return result
  </code></pre>