xfail test_scikit_learn
xref https://github.com/pandas-dev/pandas/issues/30309, this is a bit tricky. Will do a proper fix in https://github.com/pandas-dev/pandas/pull/30312, but xfailing for now.
Passed on travis. Will merge, and remove this as part of #30312.