title: Pandas: Can assign 1-column DataFrame to Series but not to DataFrame of same shape?
body: <p>Python version 3.7.13, pandas version 1.3.5</p>
  <p>I have two categorical DataFrames:</p>
  <pre class="lang-py prettyprint-override"><code>df1 = pd.DataFrame(
      data={
          &quot;col1&quot;: pd.Categorical([&quot;A&quot;, &quot;B&quot;, &quot;C&quot;]),
      },
      index=[0, 1, 2],
  )
  # and df2 defined exactly the same way,
  # both of which look like the following
     col1
  0   A
  1   B
  2   C
  </code></pre>
  <p>Now, if I want to assign <code>col1</code> of <code>df2</code> to <code>col1</code> of <code>df1</code>, I do this:</p>
  <pre class="lang-py prettyprint-override"><code>type(df1.loc[:, &quot;col1&quot;])
  # pandas.core.series.Series
  df1.loc[:, &quot;col1&quot;] = df2.loc[:, &quot;col1&quot;]
  </code></pre>
  <p>This way both slices are actually Series, and it works as intended just fine.
  If I sliced df2 to keep it a DataFrame while df1 is sliced into a Series, I still get the same result.</p>
  <pre class="lang-py prettyprint-override"><code>type(df2.loc[:, [&quot;col1&quot;]])
  # pandas.core.frame.DataFrame
  df1.loc[:, &quot;col1&quot;] = df2.loc[:, [&quot;col1&quot;]]
  </code></pre>
  <p>However, if I sliced both sides to be DataFrames, which have the same shape, it no longer works for some reason.</p>
  <pre class="lang-py prettyprint-override"><code>df1.loc[:, [&quot;col1&quot;]].shape == df2.loc[:, [&quot;col1&quot;]].shape
  # True
  df1.loc[:, [&quot;col1&quot;]] = df2.loc[:, [&quot;col1&quot;]]
  
  ValueError                                Traceback (most recent call last)
  &lt;ipython-input-69-571ef77d04b6&gt; in &lt;module&gt;()
  ----&gt; 1 df1.loc[:, [&quot;col1&quot;]] = df2.loc[:, [&quot;col1&quot;]]
  
  /usr/local/lib/python3.7/dist-packages/pandas/core/indexing.py in __setitem__(self, key, value)
      721 
      722         iloc = self if self.name == &quot;iloc&quot; else self.obj.iloc
  --&gt; 723         iloc._setitem_with_indexer(indexer, value, self.name)
      724 
      725     def _validate_key(self, key, axis: int):
  
  /usr/local/lib/python3.7/dist-packages/pandas/core/indexing.py in _setitem_with_indexer(self, indexer, value, name)
     1730             self._setitem_with_indexer_split_path(indexer, value, name)
     1731         else:
  -&gt; 1732             self._setitem_single_block(indexer, value, name)
     1733 
     1734     def _setitem_with_indexer_split_path(self, indexer, value, name: str):
  
  /usr/local/lib/python3.7/dist-packages/pandas/core/indexing.py in _setitem_single_block(self, indexer, value, name)
     1966 
     1967         # actually do the set
  -&gt; 1968         self.obj._mgr = self.obj._mgr.setitem(indexer=indexer, value=value)
     1969         self.obj._maybe_update_cacher(clear=True)
     1970 
  
  /usr/local/lib/python3.7/dist-packages/pandas/core/internals/managers.py in setitem(self, indexer, value)
      353 
      354     def setitem(self: T, indexer, value) -&gt; T:
  --&gt; 355         return self.apply(&quot;setitem&quot;, indexer=indexer, value=value)
      356 
      357     def putmask(self, mask, new, align: bool = True):
  
  /usr/local/lib/python3.7/dist-packages/pandas/core/internals/managers.py in apply(self, f, align_keys, ignore_failures, **kwargs)
      325                     applied = b.apply(f, **kwargs)
      326                 else:
  --&gt; 327                     applied = getattr(b, f)(**kwargs)
      328             except (TypeError, NotImplementedError):
      329                 if not ignore_failures:
  
  /usr/local/lib/python3.7/dist-packages/pandas/core/internals/blocks.py in setitem(self, indexer, value)
     1489 
     1490         check_setitem_lengths(indexer, value, self.values)
  -&gt; 1491         self.values[indexer] = value
     1492         return self
     1493 
  
  /usr/local/lib/python3.7/dist-packages/pandas/core/arrays/_mixins.py in __setitem__(self, key, value)
      180     def __setitem__(self, key, value):
      181         key = check_array_indexer(self, key)
  --&gt; 182         value = self._validate_setitem_value(value)
      183         self._ndarray[key] = value
      184 
  
  /usr/local/lib/python3.7/dist-packages/pandas/core/arrays/categorical.py in _validate_setitem_value(self, value)
     2043         if len(to_add) and not isna(to_add).all():
     2044             raise ValueError(
  -&gt; 2045                 &quot;Cannot setitem on a Categorical with a new &quot;
     2046                 &quot;category, set the categories first&quot;
     2047             )
  
  ValueError: Cannot setitem on a Categorical with a new category, set the categories first
  </code></pre>
  <p>I find this very confusing, because naively I would have expected the second case, assigning a DataFrame to a Series, to not work, while assigning two objects of the same type to work.</p>
  <p>This clearly has something to do with the pd.Categorical dtype, because if both DataFrames had object or numerical dtypes, no such exception ever occurs in any way of slicing. But then again, both DataFrames have exactly identical categories and values, so I don't see why it should complain about new categories.</p>
  <p>Also, if both DataFrames had more than 1 column, this exception never occurs either, i.e. the following works as intended:</p>
  <pre class="lang-py prettyprint-override"><code>df1 = pd.DataFrame(
      data={
          &quot;col1&quot;: pd.Categorical([&quot;A&quot;, &quot;B&quot;, &quot;C&quot;]),
          &quot;col2&quot;: pd.Categorical([&quot;C&quot;, &quot;B&quot;, &quot;A&quot;]),
      },
      index=[0, 1, 2],
  )
  # df2 defined exactly the same way
  df1.loc[:, [&quot;col1&quot;, &quot;col2&quot;]] = df2.loc[:, [&quot;col1&quot;, &quot;col2&quot;]]
  </code></pre>
  <p>Can someone explain what is special about assigning a 1-column categorical DataFrame to another please?</p>
  <p><strong>Edit</strong>:</p>
  <p>It seems this issue is not consistently reproduced on different python and pandas version (None of the cases above fails in python 3.9.12 and pd 1.3.2, plus yet another scenario in comment). Then perhaps let me go one step further. Originially I ran into this exception when I wanted to do:</p>
  <pre class="lang-py prettyprint-override"><code>df1.loc[:, [&quot;col1&quot;]] = df2.loc[:, [&quot;col1&quot;]].notnull()
  </code></pre>
  <p>which fails in both of my environments. However, again, if df1 was sliced to be a Series, it somehow works:</p>
  <pre class="lang-py prettyprint-override"><code>df1.loc[:, &quot;col1&quot;] = df2.loc[:, [&quot;col1&quot;]].notnull()
  </code></pre>
  <p>I know there are workarounds to accomplish this, which is why I didn't bring up this part at first and thought there was something more fundamental to the issue regarding DataFrame and Series types.</p>