PERF: call np.errstate less
These add a surprising amount of overhead.
> So now float point errors from numpy will warn correct?

This only removes np.errstate in places where doing so didn't produce any new warnings in the tests. Thanks @jbrockmendel 