title: How to replace values in a subset of data, based on values of other columns, within a Pandas method chaining expression
body: <p>The basic task in my question is rather trivial and there are certainly many ways to do it. But the key part of the question is how to do it using a Pandas method chaining expression. What would be the most idiomatic way to do it?</p>
  
  <p>Consider this reproducible example:</p>
  
  <pre><code>df = pd.DataFrame({'class_size':[1,2,3,4,5,6,7,8,9,10],
                     'num_classes':[100,80,50,40,38,2,1,1,0,0]})
  </code></pre>
  
  <p>now you want to group together all rows where the <code>class_size</code> is greater than 5, because beyond that point the <code>num_classes</code> is very small.</p>
  
  <p>You can quickly create a new column based on the class size and then replace the values greater than 5 for something else.</p>
  
  <pre><code>df['grp'] = df.class_size
  df.loc[df.class_size &gt; 5, 'grp'] = '&gt; 5'
  df
  
  class_size  num_classes grp
  0   1   100 1
  1   2   80  2
  2   3   50  3
  3   4   20  4
  4   5   10  5
  5   6   2   &gt; 5
  6   7   1   &gt; 5
  7   8   1   &gt; 5
  8   9   0   &gt; 5
  9   10  0   &gt; 5
  </code></pre>
  
  <p>Good, that's the desired result (trivial, as I said). And then later you can play with the new df, for example, to produce a summary table.</p>
  
  <pre><code>df.groupby('grp').agg(
      class_sizes_nunique = ('class_size', 'nunique'), 
      num_classes = ('num_classes', 'sum'),
  )
  
  class_sizes_nunique num_classes
  grp     
  1   1   100
  2   1   80
  3   1   50
  4   1   20
  5   1   10
  &gt; 5 5   4
  </code></pre>
  
  <p>Now, what is the best way to accomplish this, but within a Pandas method chaining expression?
  Something like </p>
  
  <pre><code>(
      df.
      replace_values_in_subset().
      to_other_stuff().
      …
  )
  </code></pre>
  
  <p>This constraint, I think, rules out the use of <code>.loc[]</code>, because you want the whole dataframe later down the line (with a few values replaced, or alternatively, with a new column with specific values in a subset given by other columns).</p>
  
  <p>So I gave <code>assign()</code> a try.</p>
  
  <p>My first attempt did not work, due to the error "The truth value of a Series is ambiguous".</p>
  
  <pre><code>(
      df.
      assign(grp = lambda x: x.class_size if x.class_size &gt; 5 else x.class_size).
      groupby('grp').agg(
          class_sizes_nunique = ('class_size', 'nunique'), 
          num_classes = ('num_classes', 'sum'),
      )
  )
  </code></pre>
  
  <p>Then I managed to get the result I want, combining <code>assign()</code> with <code>apply()</code>.</p>
  
  <pre><code>(
      df.
      assign(grp = lambda x: x.class_size.apply(lambda y: '&gt; 5' if y &gt; 5 else y)).
      groupby('grp').agg(
          class_sizes_nunique = ('class_size', 'nunique'), 
          num_classes = ('num_classes', 'sum'),
      )
  )
  </code></pre>
  
  <p>This gives me the desired output, …, but it seems weird and convoluted that way of nesting lambdas. And after googling and looking here in SO, I cant find a better way to do it, again, within the constraint of a method chain.</p>
  
  <p>Any ideas?</p>