title: Pandas Dataframe selecting groups with minimal cardinality
body: <p>I have a problem where I need to take groups of rows from a data frame where the number of items in a group exceeds a certain number (cutoff). For those groups, I need to take some head rows and the tail row.</p>
  
  <p>I am using the code below</p>
  
  <pre><code>train = train[train.groupby('id').id.transform(len) &gt; headRows]
  groups = pd.concat([train.groupby('id').head(headRows),train.groupby('id').tail(1)]).sort_index()
  </code></pre>
  
  <p>This works. But the first line, it is very slow :(. 30 minutes or more.</p>
  
  <p>Is there any way to make the first line faster ? If I do not use the first line, there are duplicate indices from the result of the second line, which messes up things.</p>
  
  <p>Thanks in advance
  Regards</p>
  
  <p>Note: 
  My train data frame has around 70,000 groups of varying group size over around 700,000 rows . It actually follows from my other question as can be seen here <a href="https://stackoverflow.com/questions/23580009/data-processing-with-adding-columns-dynamically-in-python-pandas-dataframe">Data processing with adding columns dynamically in Python Pandas Dataframe</a>. 
  Jeff gave a great answer there, but it fails if the group size is less or equal to parameter I pass in head(parameter) when concatenating my rows as in Jeffs answer : In [31]: groups = concat.....</p>