BUG: groupby.agg/transform casts UDF results
- [x] closes #21240
- [x] closes #17035
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

This PR stops casting when the func is a callable since we can't tell when casting will be more harmful than helpful. I think this results in a more expected/consistent behavior for users. 
@jreback -  change made and green. TODOs are added in regards to mean/median/var, opened #41137 and will be taken care of by #41139

> may also need to update groupby.rst, we might have this documented

I looked through, there is no mention of return dtype for agg/transform/apply. right i think we should try to start documenting return types for the udfs in the main docs with some examples (and ideally add to the doc strings too)
 @jreback - docs added and green. @jreback - changes made and green. I added versionchanged to the Notes section as well as the Examples sections. thanks @rhshadrach 