CI: Test numpy 1.24 on 3.11 build
- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Why don't we try unpinning numpy on 3.11 instead? That build has no numba IIRC. Hm did not consider this, thanks for pointing it out. Makes sense to me cc @mroeschke this is ready for a review now, not sure if we want to leave all the ignores in... > not sure if we want to leave all the ignores in

If there are any deprecation or future warnings those should be fixed We are only filtering RuntimeWarnings now. Were any of the RuntimeWarnings addressable, or were they specific to the data being passed? All data dependent. Thanks @phofl  I think these tests are failing due to numpy being unpinned here

```
 FAILED pandas/tests/plotting/test_series.py::TestSeriesPlots::test_bar_log - AssertionError: numpy array are different

numpy array values are different (16.66667 %)
[left]:  [0.09999999999999999, 1.0, 10.0, 100.0, 1000.0, 10000.0]
[right]: [0.1, 1.0, 10.0, 100.0, 1000.0, 10000.0]
FAILED pandas/tests/plotting/frame/test_frame_subplots.py::TestDataFramePlotsSubplots::test_bar_log_no_subplots - AssertionError: numpy array are different

numpy array values are different (25.0 %)
[left]:  [0.09999999999999999, 1.0, 10.0, 100.0]
[right]: [0.1, 1.0, 10.0, 100.0]
FAILED pandas/tests/plotting/frame/test_frame_subplots.py::TestDataFramePlotsSubplots::test_bar_log_subplots - AssertionError: numpy array are different

numpy array values are different (16.66667 %)
[left]:  [0.09999999999999999, 1.0, 10.0, 100.0, 1000.0, 10000.0]
[right]: [0.1, 1.0, 10.0, 100.0, 1000.0, 10000.0]
``` Yikes this is weird, I xfailed them but this build started failing because the tests were passing again, so I removed it…