BLD: Add pyarrow extra for pip installation
Partial feedback from #54466
thx @mroeschke  Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 2.2.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 734501e595d1e771774949232ed5f382e238db8d
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #57551: BLD: Add pyarrow extra for pip installation'
```

4. Push to a named branch:

```
git push YOURFORK 2.2.x:auto-backport-of-pr-57551-on-2.2.x
```

5. Create a PR against branch 2.2.x, I would have named this PR:

> "Backport PR #57551 on branch 2.2.x (BLD: Add pyarrow extra for pip installation)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                