API: to_datetime, required unit with numerical (#15836)
* add test_to_datetime_numerical_input
* check arg for numerical type

 - [x] closes #15836 
 - [x] tests added / passed
 - [x] passes ``git diff upstream/master --name-only -- '*.py' | flake8 --diff``
 - [x] whatsnew entry

I took into account what you said, but I still have issues with two tests `test_dataframe_dtypes` and `test_dataframe` which do not need unit to work.
The units are directly in the keys of the structures.

What behaviour to adopt in those cases ? Given the changes that we have to make in our own test suite, we should maybe consider to make this a deprecation first before actually changing? (raising a warning you should provide `unit`, and in future not providing will raise?) I think its better to actually make this an error now. If you are doing this would almost always don't mean ``ns``, so that is the only case that is really affected. Otherwise its an error (not so obvious though), so better to fail this. IOW, this is a bug, not an API change. can you rebase # [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15896?src=pr&el=h1) Report
> Merging [#15896](https://codecov.io/gh/pandas-dev/pandas/pull/15896?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/a9421af1aac906cc38d025ed5db4a2b55cb8b9bc?src=pr&el=desc) will **increase** coverage by `0.13%`.
> The diff coverage is `90%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/15896/graphs/tree.svg?token=eZ4WkYLtcO&src=pr&height=150&width=650)](https://codecov.io/gh/pandas-dev/pandas/pull/15896?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #15896      +/-   ##
==========================================
+ Coverage   90.96%    91.1%   +0.13%     
==========================================
  Files         161      161              
  Lines       49301    49307       +6     
==========================================
+ Hits        44849    44923      +74     
+ Misses       4452     4384      -68
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `88.88% <90%> (+0.15%)` | :arrow_up: |
| #single | `40.2% <60%> (-0.06%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/15896?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/indexes/datetimes.py](https://codecov.io/gh/pandas-dev/pandas/pull/15896?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9kYXRldGltZXMucHk=) | `95.33% <100%> (ø)` | :arrow_up: |
| [pandas/core/dtypes/cast.py](https://codecov.io/gh/pandas-dev/pandas/pull/15896?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZHR5cGVzL2Nhc3QucHk=) | `86.89% <100%> (ø)` | :arrow_up: |
| [pandas/core/tools/datetimes.py](https://codecov.io/gh/pandas-dev/pandas/pull/15896?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvdG9vbHMvZGF0ZXRpbWVzLnB5) | `85.2% <83.33%> (+18.26%)` | :arrow_up: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/15896?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/15896?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.71% <0%> (-0.1%)` | :arrow_down: |
| [pandas/plotting/\_converter.py](https://codecov.io/gh/pandas-dev/pandas/pull/15896?src=pr&el=tree#diff-cGFuZGFzL3Bsb3R0aW5nL19jb252ZXJ0ZXIucHk=) | `65.05% <0%> (+1.81%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15896?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15896?src=pr&el=footer). Last update [a9421af...904e45c](https://codecov.io/gh/pandas-dev/pandas/pull/15896?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
 # [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15896?src=pr&el=h1) Report
> Merging [#15896](https://codecov.io/gh/pandas-dev/pandas/pull/15896?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/77bfe21c7229e724d01721bb84861283baf7e9d3?src=pr&el=desc) will **increase** coverage by `0.09%`.
> The diff coverage is `90%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/15896/graphs/tree.svg?width=650&height=150&src=pr&token=eZ4WkYLtcO)](https://codecov.io/gh/pandas-dev/pandas/pull/15896?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #15896      +/-   ##
==========================================
+ Coverage   91.01%    91.1%   +0.09%     
==========================================
  Files         163      161       -2     
  Lines       49570    49307     -263     
==========================================
- Hits        45116    44923     -193     
+ Misses       4454     4384      -70
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `88.88% <90%> (+0.1%)` | :arrow_up: |
| #single | `40.2% <60%> (-0.12%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/15896?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/indexes/datetimes.py](https://codecov.io/gh/pandas-dev/pandas/pull/15896?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9kYXRldGltZXMucHk=) | `95.33% <100%> (ø)` | :arrow_up: |
| [pandas/core/dtypes/cast.py](https://codecov.io/gh/pandas-dev/pandas/pull/15896?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZHR5cGVzL2Nhc3QucHk=) | `86.89% <100%> (-0.93%)` | :arrow_down: |
| [pandas/core/tools/datetimes.py](https://codecov.io/gh/pandas-dev/pandas/pull/15896?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvdG9vbHMvZGF0ZXRpbWVzLnB5) | `85.2% <83.33%> (+18.17%)` | :arrow_up: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/15896?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |
| [pandas/core/indexes/accessors.py](https://codecov.io/gh/pandas-dev/pandas/pull/15896?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9hY2Nlc3NvcnMucHk=) | `88.09% <0%> (-1.16%)` | :arrow_down: |
| [pandas/core/internals.py](https://codecov.io/gh/pandas-dev/pandas/pull/15896?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW50ZXJuYWxzLnB5) | `93.43% <0%> (-0.87%)` | :arrow_down: |
| [pandas/util/\_decorators.py](https://codecov.io/gh/pandas-dev/pandas/pull/15896?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvX2RlY29yYXRvcnMucHk=) | `65.3% <0%> (-0.7%)` | :arrow_down: |
| [pandas/plotting/\_timeseries.py](https://codecov.io/gh/pandas-dev/pandas/pull/15896?src=pr&el=tree#diff-cGFuZGFzL3Bsb3R0aW5nL190aW1lc2VyaWVzLnB5) | `60.2% <0%> (-0.53%)` | :arrow_down: |
| [pandas/core/sparse/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/15896?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc3BhcnNlL2ZyYW1lLnB5) | `94.26% <0%> (-0.45%)` | :arrow_down: |
| ... and [66 more](https://codecov.io/gh/pandas-dev/pandas/pull/15896?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15896?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15896?src=pr&el=footer). Last update [77bfe21...3a7b1c0](https://codecov.io/gh/pandas-dev/pandas/pull/15896?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
 can you rebase and update to comments. can you rebase and update closing as stale. if you'd like to continue pls ping.