Bug in DataFrame.replace casts columns to ``object`` dtype if items in ``to_replace`` not in values
- [ ] closes #32988
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

I am ok with reverting things for the 1.0.4 branch but not for master. thanks @simonjayhawkins 