PERF: isin casting integers to object
- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

xref https://github.com/pandas-dev/pandas/pull/38279#issuecomment-739973166 cc @jorisvandenbossche 
lgtm. do we have / need asv's? do we need a whatsnew note (not sure if this was working in 1.1 at all) No, this was catched by an asv we already have, and only a regression on master