title: Replace less frequent labels by categories
body: <p>I have a very long dataframe with numeric and categorical columns.</p>
  <pre><code>    id       dur    proto       state   att_cat
  0   1   0.121460    tcp         FIN     Normal
  1   2   0.649902    tcp         FIN     Normal
  2   3   1.623047    tcp         FIN     Normal
  3   4   1.681641    tcp         FIN     Normal
  4   5   0.449463    tcp         CON     Normal
  5   6   0.000009    udp         INT     Generic
  6   7   0.505859    udp         CON     Normal
  7   8   0.000009    udp         INT     Generic
  8   9   0.000009    udp         INT     Generic
  9   10  0.000009    tcp         INT     Generic
  10  11  0.222761    unas        ECO     Normal
  11  12  1.461278    tcp         CON     Normal
  12  13  1.065289    arp         FIN     Normal
  13  14  2.782646    udp         CON     Normal
  14  15  1.457923    tcp         FIN     Normal
  15  16  0.000009    udp         INT     Generic
  17  18  0.125550    arp         INT     Generic
  18  19  0.000009    tcp         INT     Generic
  19  20  0.000009    tcp         CON     Generic
  </code></pre>
  <p>In the column att_cat I have different labels I would like to group. When I do so, then I observe that for example, the column <em>proto</em> have some labels that appear more frequent than others, for example, for att_cat=Normal, tcp is more frequent than udp and others (not show in the dataframe); for Generic, the most frequent is udp.</p>
  <p>I would like, for each label in att_cat, to keep the <em>n</em> more frequent labels in proto (or any of the categorical columns), and the rest, substitute them by 'att_cat_proto'.</p>
  <p>For example, for the feature <em>proto</em>:</p>
  <pre><code>Normal = {tcp: 7, udp: 2, arp: 1, unas: 1}
  Generic = {udp: 4, tcp: 3, arp: 1, unas: 0}
  </code></pre>
  <p>For the feature 'state'</p>
  <pre><code>Normal = {FIN: 5, CON: 4, INT: 1, ECO: 1}
  Generic = {INT: 6, CON: 1, FIN: 1, ECO: 0}
  </code></pre>
  <p>If I fix <em>n=2</em> for both Normal and Generic for the feature <em>proto</em>, and <em>n=2</em> for Normal and <em>n=1</em> for Generic for <em>state</em>, then the desired result should look like:</p>
  <pre><code>    id       dur    proto          state            att_cat
  0   1   0.121460    tcp            FIN              Normal
  1   2   0.649902    tcp            FIN              Normal
  2   3   1.623047    tcp            FIN              Normal
  3   4   1.681641    tcp            Normal_State     Normal
  4   5   0.449463    tcp            CON              Normal
  5   6   0.000009    udp            INT              Generic
  6   7   0.505859    udp            CON              Normal
  7   8   0.000009    udp            INT              Generic
  8   9   0.000009    udp            INT              Generic
  9   10  0.000009    tcp            Generic_State    Generic
  10  11  0.222761    Normal_Proto   Normal_State     Normal
  11  12  1.461278    tcp            CON              Normal
  12  13  1.065289    Normal_Proto   FIN              Normal
  13  14  2.782646    udp            CON              Normal
  14  15  1.457923    tcp            FIN              Normal
  15  16  0.000009    udp            INT              Generic
  17  18  0.125550    Generic_Proto  INT              Generic
  18  19  0.000009    tcp            INT              Generic
  19  20  0.000009    tcp            Generic_State    Generic
  </code></pre>
  <p>So far I have tried doing something like:</p>
  <pre><code>def rare_labels(df,target,cat_var,n):
      df = df.copy()
      # Selects the low frequency labels
      cat_frame = df[cat_var]
      most_freq = cat_frame.value_counts().index[:n].to_list()
      less_freq = np.setdiff1d(cat_frame.unique(),most_freq)
      # Substitute the low frequency labels by a common label
      df[cat_var] = df.groupby(target).******* 
      # Returns the dataframe
      return df
  </code></pre>
  <p>but I'm stucked in the assigning part.</p>