[WIP] Improve CI/CD pipeline performance
- [X] closes #24028 
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Out of curiosity, how big is the perf difference?  > Out of curiosity, how big is the perf difference?

It is WIP. WIll know once I make some progress. does this change perf? can you merge master as well @WillAyd @jreback yes, it is not reflecting on the performance improvement. I was off a couple of weeks now I'm back on. Will check it @bhavaniravi if you want to continue on this, pls merge master and post a comparison of before / after. I think this is stale or not really giving the results we are after, so closing. @bhavaniravi if you disagree ping and can reopen for sure

Thanks as always!