BUG: to_datetime when called with a unit and coerce is buggy
closes #11758

@jorisvandenbossche straightforward but lots of cases. So actually what I need is a bunch more tests. 

Stuff like mixed float/int/string (that are integer/float-like) with some NaT's thrown in (these should pass).
