title: Pandas DataFrame.agg produces multiindex after selection with missing categories
body: <p>I'm still working on a reproducible example, but unfortunately, I'm an absolute beginner at pandas, and I'm lost as to what I'm doing wrong. The behavior seems like a bug in Pandas 1.2.4.</p>
  <p>I have two dataframes, <code>files_df</code> which gives the type and complexity of source code files in several repos, and <code>proj_df</code> which gives the projects and the last time they were updated.</p>
  <p>I am trying to pick the top two projects for each language by complexity, and it looks like this:</p>
  <pre class="lang-py prettyprint-override"><code>def largest(x):
      return (
          # Pick the top two rows in each language (from the groupby) by scc_complexity
          x.nlargest(2, 'scc_complexity')
          # Combine the proj_repo values (we don't actually care about scc_complexity at this point)
          .agg({'proj_repo': lambda y: '&lt;br&gt;'.join(y.values)})
      )
  
  max_proj = (
      # Pick the language and complexity columns
      files_df[['scc_lang', 'scc_complexity', 'proj_repo']]
      # Sum scc_complexity by language and repo
      .groupby(['scc_lang', 'proj_repo'], observed=True).sum()
      .reset_index()
  )
  
  max_proj = (
      max_proj
      # Select only repos updated in the last two years
      .loc[max_proj['proj_repo'].isin(proj_df.loc[proj_df['last_author_time'] &gt; '2019-05-01', 'proj_repo'])]
      # Now pick the top repos by language complexity
      .groupby('scc_lang').apply(largest)
  )
  </code></pre>
  <p>This should produce a frame with <code>scc_lang</code> as its index and <code>proj_repo</code> as its only column, but instead it produces this shape:</p>
  <pre><code>&lt;class 'pandas.core.frame.DataFrame'&gt;
  MultiIndex: 151 entries, ('ASP', 'proj_repo') to ('XML Schema (min)', 'proj_repo')
  Data columns (total 2 columns):
   #   Column     Non-Null Count  Dtype 
  ---  ------     --------------  ----- 
   0   0          151 non-null    object
   1   proj_repo  0 non-null      object
  dtypes: object(2)
  memory usage: 8.7+ KB
  MultiIndex([(                        'ASP', 'proj_repo'),
              (                   'Autoconf', 'proj_repo'),
              (                       'BASH', 'proj_repo'),
              (                      'Batch', 'proj_repo'),
  ...
  </code></pre>
  <p>That is, there's now a multi-index, and the <code>proj_info</code> column is now a column named <code>0</code>, and the <code>proj_info</code> column is now empty.</p>
  <p>Now for the really strange part. This code produces the correct shape if you remove the <code>.loc[]</code> selection from <code>max_proj</code>. For the life of me, I can't figure out why making a selection changes the shape of the results. I haven't been able to reproduce the results with a small complete example.</p>
  <p>I will keep experimenting, and if I learn anything interesting, I'll report here.</p>
  <p><em>Slight update</em>: I've discovered that removing <code>observed=True</code> also seems to produce the correct shape—so maybe this is related to missing keys?</p>
  <p><strong>Bigger update</strong>: I managed to reproduce it, and yes, it does somehow relate to missing keys, though I'm not sure how:</p>
  <pre class="lang-py prettyprint-override"><code>tdf = pd.DataFrame([['lang1', 1, 'ab'],
                      ['lang2', 2, 'cd'],
                      ['lang2', 4, 'cd'],
                      ['lang2', 3, 'ef'],
                      ['lang2', 4, 'gh'],
                      ['lang3', 2, 'ef']
                     ],
                      columns=['lang', 'size', 'name']).astype({'lang': 'category'}, copy=False).set_index('lang')
  
  d = tdf.groupby(['lang', 'name'], observed=True).sum().reset_index()
  d = d.loc[d['name'].isin(pd.Series(['ab','cd','gh']))].groupby('lang').apply(lambda x: x.agg({'name': lambda y: ' '.join(y.values)}))
  display(d)
  d.info()
  </code></pre>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th></th>
  <th></th>
  <th><strong>0</strong></th>
  <th><strong>name</strong></th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td><strong>lang</strong></td>
  <td></td>
  <td></td>
  <td></td>
  </tr>
  <tr>
  <td><strong>lang1</strong></td>
  <td><strong>name</strong></td>
  <td>ab</td>
  <td>NaN</td>
  </tr>
  <tr>
  <td><strong>lang2</strong></td>
  <td><strong>name</strong></td>
  <td>cd gh</td>
  <td>NaN</td>
  </tr>
  </tbody>
  </table>
  </div>
  <pre><code>&lt;class 'pandas.core.frame.DataFrame'&gt;
  MultiIndex: 2 entries, ('lang1', 'name') to ('lang2', 'name')
  Data columns (total 2 columns):
   #   Column  Non-Null Count  Dtype 
  ---  ------  --------------  ----- 
   0   0       2 non-null      object
   1   name    0 non-null      object
  dtypes: object(2)
  memory usage: 356.0+ bytes
  </code></pre>
  <p>Removing either the 'ab' or 'ef' names triggers this, meaning it has to do with a missing category. Indeed, changing the column type back to <code>object</code> <em>also</em> fixes this issue. So it seems I've misunderstood categories...</p>
  <p><strong>Another update</strong>: I've managed to reduce it down to these two lines:</p>
  <pre class="lang-py prettyprint-override"><code>tdf2 = pd.DataFrame([['a', 'boof']], columns=['lang', 'name']).astype({'lang': pd.CategoricalDtype(['a', 'b'])})
  display(tdf2.groupby('lang').apply(lambda x: x.agg({'name': lambda y: y.values})))
  </code></pre>
  <p>We're pretty minimal now. Removing almost any bit of the above code causes the resulting data frame to snap to the correct shape.</p>
  <p>What we have here is an agg being applied to an empty dataframe produced by <code>.groupby('lang')</code>, which resurrects a category that has been removed from the data. There seems to be strangeness around the result being rejoined inside <code>apply()</code>.</p>
  <p>One new finding here is that returning <code>y.values</code> from the <code>apply()</code> lambda is part of it: if I return <code>y</code> (the Series) instead, the strange structure doesn't happen.</p>