BUG: Fix regression in pandas merge
- [x] closes #38616
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

Since this is a regression on master I think a whatsnew is not necessary?

thanks @phofl  @meeseeksdev backport 1.2.x