title: Efficient feature reduction in a pandas data frame
body: <p>I have a big data set of about 200K samples. each sample has a set of features (about 10) from a variety of about 100K possible features, and a few float-type measurements with it.</p>
  
  <p>For example, for the given data set</p>
  
  <pre><code>Features                  trial             observations
  {1, 40020, 8222}          4                 2
  {1, 40020, 22, 16000}     14                8
  {1, 20, 22, 1000}         1                 0
  {42, 22, 16000}           2                 1
  </code></pre>
  
  <p>So I need a function f such that:</p>
  
  <pre><code>f(data, {1, 40020})=
  
  Features                  trial             observations
  {1, 40020}                18                10
  {1}                       1                 0
  {}                        2                 1
  
  f(data, {22, 40020})=
  
  Features                  trial             observations
  {40020}                   4                 2
  {40020, 22}               14                8
  {22}                      3                 1
  </code></pre>
  
  <p>So, the function f groups the data by intersecting the feature column with a give set and summing the aggregated columns.</p>
  
  <p>Consider that I need to call "f" for the same dataset with many different feature sets as the second argument, so any pre-processing that can be done once to speed up each call will probably be beneficial.</p>
  
  <p>The fastest way I found is </p>
  
  <p><code>
  pandas.DataFrame([sample.data for sample in samples], index = [sample.features for sample in samples]).groupby(lambda x: x &amp; test_features, sort = False).sum()
  </code></p>
  
  <p>But the performance was not good enough. I am guessing that this is because I am using a function for the groupby. is there a way to optimize this?</p>