BUG: Fix MultiIndex column stacking with dupe names
- [x] closes #36353
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

thanks @dsaxton yeah this must have been some leftover code from when the warning was removed. @meeseeksdev backport 1.1.x