ENH/PERF: Remove frequency inference from .dt accessor
can you run the asv's with --bench freq and see what improves. also pls add a whatsnew note (perf section) @cpcloud can you give a bit more context? 
What is the consequence of removing it? Why what the inference added in the first place?

I suppose the only reason to have the `freq='infer'` when constructing the DatetimeIndex for the `dt` accessor is for the `dt.freq` attribute? (I don't think the other methods available in `dt.` need the freq) 
But removing the `freq='infer'` makes that `dt.freq` will always be None and thus rendered effectively useless. And it *is* also a backwards incompatible change (so needs a note about that in the whatsnew). If we do it like this, I think we should remove (or deprecate) `dt.freq`

But maybe we can also do the infer step only when people ask for `dt.freq` ? (so there is no backwards incompatible change) Not sure how complicated it would be, because then it doesn't map directly to get the `freq` from the underlying datetimeindex object that was constructed 

 this was removed for perf, IOW we *don't* need to infer by default (we don't normally do this on construction). so the ``.freq`` attribute in the accessor could simply return ``inferred_freq`` instead (which does the inference if needed). @jorisvandenbossche the inference on attribute access is suboptimal for two reasons. 1. It makes attribute access scale with the size of the series. 2. Any order N operation on the dt values is now 2*N which is noticeable for large series. We shouldn't make users pay for something they don't necessarily need and we should try to avoid leaving 2x on the table if we don't have to.

I don't think this needs to be backwards incompatible, I'll fix it up.  # [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17210?src=pr&el=h1) Report
> Merging [#17210](https://codecov.io/gh/pandas-dev/pandas/pull/17210?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/a2c454373b8cd2334b93a68d52104e9cbcdd3721?src=pr&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `75%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/17210/graphs/tree.svg?token=eZ4WkYLtcO&width=650&src=pr&height=150)](https://codecov.io/gh/pandas-dev/pandas/pull/17210?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #17210      +/-   ##
==========================================
- Coverage   91.02%   91.01%   -0.02%     
==========================================
  Files         162      162              
  Lines       49517    49511       -6     
==========================================
- Hits        45075    45061      -14     
- Misses       4442     4450       +8
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `88.79% <75%> (ø)` | :arrow_up: |
| #single | `40.24% <25%> (-0.08%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/17210?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/indexes/accessors.py](https://codecov.io/gh/pandas-dev/pandas/pull/17210?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9hY2Nlc3NvcnMucHk=) | `88.76% <75%> (ø)` | :arrow_up: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/17210?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/17210?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.71% <0%> (-0.11%)` | :arrow_down: |
| [pandas/core/computation/eval.py](https://codecov.io/gh/pandas-dev/pandas/pull/17210?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29tcHV0YXRpb24vZXZhbC5weQ==) | `96.93% <0%> (-0.04%)` | :arrow_down: |
| [pandas/core/computation/pytables.py](https://codecov.io/gh/pandas-dev/pandas/pull/17210?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29tcHV0YXRpb24vcHl0YWJsZXMucHk=) | `92.35% <0%> (-0.03%)` | :arrow_down: |
| [pandas/core/computation/align.py](https://codecov.io/gh/pandas-dev/pandas/pull/17210?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29tcHV0YXRpb24vYWxpZ24ucHk=) | `97.91% <0%> (-0.03%)` | :arrow_down: |
| [pandas/core/computation/engines.py](https://codecov.io/gh/pandas-dev/pandas/pull/17210?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29tcHV0YXRpb24vZW5naW5lcy5weQ==) | `88.88% <0%> (ø)` | :arrow_up: |
| [pandas/core/computation/expr.py](https://codecov.io/gh/pandas-dev/pandas/pull/17210?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29tcHV0YXRpb24vZXhwci5weQ==) | `88.48% <0%> (ø)` | :arrow_up: |
| [pandas/core/computation/scope.py](https://codecov.io/gh/pandas-dev/pandas/pull/17210?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29tcHV0YXRpb24vc2NvcGUucHk=) | `94.05% <0%> (+0.92%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17210?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17210?src=pr&el=footer). Last update [a2c4543...91d81d7](https://codecov.io/gh/pandas-dev/pandas/pull/17210?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
 # [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17210?src=pr&el=h1) Report
> Merging [#17210](https://codecov.io/gh/pandas-dev/pandas/pull/17210?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/06850a148ad880eb2fd2564cc0ad7cae8606dd90?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `87.5%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/17210/graphs/tree.svg?height=150&width=650&token=eZ4WkYLtcO&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/17210?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #17210      +/-   ##
==========================================
+ Coverage      91%   91.01%   +<.01%     
==========================================
  Files         162      162              
  Lines       49531    49535       +4     
==========================================
+ Hits        45077    45084       +7     
+ Misses       4454     4451       -3
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `88.79% <87.5%> (+0.02%)` | :arrow_up: |
| #single | `40.25% <37.5%> (-0.08%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/17210?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/indexes/accessors.py](https://codecov.io/gh/pandas-dev/pandas/pull/17210?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9hY2Nlc3NvcnMucHk=) | `89.24% <87.5%> (+0.48%)` | :arrow_up: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/17210?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/17210?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.72% <0%> (-0.1%)` | :arrow_down: |
| [pandas/plotting/\_converter.py](https://codecov.io/gh/pandas-dev/pandas/pull/17210?src=pr&el=tree#diff-cGFuZGFzL3Bsb3R0aW5nL19jb252ZXJ0ZXIucHk=) | `65.05% <0%> (+1.81%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17210?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17210?src=pr&el=footer). Last update [06850a1...7361670](https://codecov.io/gh/pandas-dev/pandas/pull/17210?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
 lgtm. can you add a perf note thanks @cpcloud 