REGR: Make DateOffset immutable
- [x] closes #36940
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

I'm assuming this is a regression and not behavior we want?
LGTM thanks @dsaxton 

any ideas about the windows failures ? (obviously not related here) > any ideas about the windows failures ? (obviously not related here)

Hmm, I'm not sure to be honest