PERF: Fix performance regression in isin for empty values
- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Restores previous performance on my machine and gives us a speedup in other places
```
     [9f687cbc]       [17a3be81]
-     1.75±0.02μs        297±0.6ns     0.17  libs.InferDtype.time_infer_dtype_skipna('empty')
-      1.89±0.2μs        296±0.8ns     0.16  libs.InferDtype.time_infer_dtype('empty')
```

#49162
Cc @rhshadrach  Thanks @phofl  Oops, something went wrong applying the patch ... Please have a look at my logs. @meeseeksdev backport 1.5.x